But once that’s sorted out “you just forget about that”.

Unfortunately I don’t have any experience with programming Windows Controls in unicode or utf8… last time I did any Windows GUI thing was with MFC and I can hardly remember anything! Sorry :-/

That however means to talk to all Windows Controls in Unicode (via the …W calls) whereas the main application code remains 8-bit based. Has anybody attempted this approach before us, or would you deem this in the end more troublesome than just fixing all “unclean” places in the code, e.g. where literals are used without T(” …. “)?

And it’s definitely right. I’ve had virtually zero problems when working with mac and linux together, whereas with windows there’s always the problem with encodings and line feeds (although line feeds do not show up as weird characters in the middle of the page).

Another reason to use UTF-8 in all code and databases is that it greatly facilitates compatibility between team members. Enforcing it requires a bit of buy-in from the team, but it’s worth it.

On my current project at Yahoo, some of us use Macs, and others use Windows. A few developers write their code on RHEL or FreeBSD, either in Vim or Eclipse. The servers either FreeBSD or RHEL.

Once you set up your editor and databases to use UTF-8, it’s immediately apparent when someone is saving in ISO-8859-1. The bad characters stick out like a sore thumb. The site looks wrong if they’re in the markup. Our policy is to stop everything, immediately change the bad characters, check the file back into CVS, and then look at the CVS logs and figure out who’s not using UTF-8. It’s very effective, because no one wants to be “that guy.”

The default encoding on most *nix systems (including Mac) is UTF-8, but the default in Windows is ISO-8859-1, and some *nix programs try to be “convenient” by silently supporting the Windows encoding. Unless we are all vigilant, it will cause problems.

@Remco

people in Asia aren’t very happy with it – western characters take 1 or 2 bytes, but many Asian ones take 3, 4 or even 5.

Forgive the insensitivity, but they need to get over it. At least for the foreseeable future, UTF-8 is the most widely supported Unicode worldwide. There are editors for every Asian language that can save in UTF-8. As long as a request is all in one language, the extra bytes are mostly taken care of by serving gzip-encoded pages, anyhow. (If you’re not telling your web server to gzip textual files, why not? It’s not 1990. Browsers actually support gzip these days!)

Hardware increases in power and capacity with unbelievable speed, and software is necessarily fraught with irreducible complexity. So, whenever possible, it seems that it is generally best to make the hardware do some extra work (storing and encoding 5 bytes per glyph instead of 2), if doing so will make the software simpler and easier to understand (by eliminating the character encoding layer.)