Comments on: UTF-8 checklist

By: winden

winden — Fri, 14 Dec 2007 00:42:12 +0000

My app was a japanese+english dictionary and stored the lines in utf8 in memory.

Searching a dictionary is nothing more complex than doing a lot of substring compares using strstr, and that’s safe to do between two utf8 strings due to the binary encoding.

All other per-char stuff was done with wide chars which is easy due to fixed bitsize chars.

bugfix for 4th point above:

wchar_s *s = L”whatever”;

By: sole

sole — Tue, 11 Dec 2007 23:16:09 +0000

I understood that if you just use chars when dealing with utf characters things can easily be broken – for example if uppercasing a string.
Although I haven’t checked the wide character string functions (I don’t use C for handling strings ;-)) …

By: winden

winden — Tue, 11 Dec 2007 10:06:36 +0000

Important steps if you are coding in C using POSIX:

1. Add setlocale(LC_CTYPE,”"); at the start of your main function.

2. Use char where you are managing UTF-8, and internally pack/unpack to wchar_t which is a flat 32bits-wide character when doing internal operations. CPU+cache is fast and memory is slow, so take advantage and pack your strings even while in memory.

3. A literal string with UTF-8 enconding:

char *s = “whatever”;

4. A literal string with UTF-32 encoding:

wchar_t *s = “whatever”;

By: Nahuel

Nahuel — Tue, 11 Dec 2007 07:01:02 +0000

I’ve worked with django for about 6 month now, I’ve seen it getting decent unicode support.
First make sure you’re using the SVN version, and also make sure you’re writing something like http://www.python.org/dev/peps/pep-0263/ says.
Even though django prints objects right with the __str__ method (something like java’s toString), don’t forget to implement __unicode__ for every model class.
Also keep in mind current python strings get the system encoding, so I don’t feel very confident when I write something like “%s %s” (var1, var2). I tend to write u’%s %s’ % (var1, var2).
If you are using some extra python package like reportlab for PDF output, make sure you use the unicode(instance), or u’something %s’ % obj
That’s all I have to do to get proper unicode handling, it’s easy though I miss java a little in this particular subject.