soledad penadés

repeat 4[fd 100 rt 90]

Following the discussion in the previous post (Reasons for using UTF-8) I thought it could be interesting to gather a series of steps needed to get a UTF-8 friendly environment.

I'm going to focus on php and mysql, because using mysql and ruby/rails and utf8 tends to be kind of easier (specially since newer Rails versions suggest mysql which charset to use when connecting automagically for you), but the advice can be applied to both platforms in any case.

In your preferred editor

Make sure your editor is set to use UTF-8, specially when editing templates and any other file which is used for building output content. If you include any non-ASCII content (for example, accented words) and they are mixed with more UTF-8 content (from other templates or sources), things will get messed up.

In your html/xml code

Make sure the document's charset is specified.

In HTML documents you would accomplish this with the content/type meta tag, which you should place in the header:

In XML documents this is done in the XML declaration, which needs to be placed at the immediate beginning of the document:

I have heard recommendations for placing the page title immediately after the Content-type has been declared, so that browsers can switch to the right charset immediately if the page title includes any non-ASCII code, but it sounds a little urban myth to me.

I have also seen people recommending to specify the encoding in every form you have in your site, but I haven't found any difference between doing it or not.

In Apache

Make sure the content is being served as UTF-8. A good AddDefaultCharset utf-8 should do. You don't need to change all your hosted content into UTF-8 if you don't want to, charsets can be configured per virtual hosts as well.

In the database

Make sure mysql is configured to use UTF-8 at server level.

Before issuing any other query, and right after you've connected to the server, execute

It seems otherwise mysql will not recognize properly the character set that the client is using (!!) and will return bad data.

Of course, when you create the tables, make sure that every column which can include text data (this includes TEXT, VARCHAR, etc) is using utf8 as well.

In php

I rarely need to send a header specifying the content-type, but if things are not working totally fine you could also try to manually specify the content type with the header:

Or application/xml, whatever you need!

In Flash (or Flex)

Do not use systemCodepage. Flash assumes XML is UTF-8 so if everything else has been properly set up, Flash will be served UTF-8 content, which is what it expects, and we'll be happy :-)

One more thing

Although generally it's a good idea to run a validator, it's even a better idea if you're doing utf-8 stuff and want to make sure you're not outputting bad stuff inadvertently. Validators are picky (even more if we speak about XML validators) and will cheekily reveal your inappropriate characters.

Missing something? Anything wrong?

I'm not an expert and I may easily have forgotten something so if you think something is missing or plainly wrong you know what to do ;-)

And before you ask - I don't have any experience with Pylons, Django, TurboGear, Zope or whatever your favourite Python framework is, so feel free to share with us your experience.

There are lots of urban legends around the intrincate complexities of setting up a web server in order to allow more than just one person to work with it. In fact, there are lots of bad habits, unfortunately very well established, like the "make your files writable == chmod 777 rule" which are just plainly Very Bad Ideas, and are also consequences of not understanding how things work.

So how do things work?

The main goal is to be able to add and modify files in the web server, regardless of which user does it. And the solution is just a combination of appropiate permissions and groups:

1. Identify which user is executing apache in your server. For example, in an ubuntu machine it is usually www-data.
2. Identify which users need to be able to read, write and delete files and directories inside the web root. For example, there could be a couple of users called foo and bar. Both need to be able to change files in the server, whether they were created first by them or not.
3. Add foo and bar to the www-data group.
4. For fixing possible glitches with permissions, do this:

1. sudo chgrp -R www-data /var/www # change all the files' group to www-data
2. sudo chmod g+w -R /var/www # allow people in the group to write to the files
3. sudo chmod g+s -R /var/www # set the group id bit, so newly created files and folders inside the folders will inherit the group id of their container

Now, with this set up, everybody in the www-data group can modify stuff in /var/www, and newly created files will belong to the www-data group. And, of course, apache will be able to read and modify those files, so all should work nicely.

The only problem you can find is that when new files are created or copied they don't get the write permission for its group. A solution could just be to do a chmod g+w right after creating it (that's what I'm doing currently); advanced solutions will involve modifying the mask used to deal with files in the user profile.

Conclussion

As you see, this is a very easy process - it just requires a bit of attention to detail when setting up the server, and saves lots of headaches later. It is mostly useful for people with a dedicated server, or with an in-house machine, like the ubuntu box we have, where we can modify permissions, ownerships, groups, etc, at our will.
If you are in a shared hosting solution, I hope your hosting sets up things properly, so you will be able to avoid most of the cases where you need to do a chmod 777 in order to have apache writing stuff to folders that you, as a user, created. It is not that difficult, as you see!

Unfortunately most of the times they set things like this:

1. apache is in its own group: apache
2. user accounts are in their own group: foo
3. stuff created by users, and the very folder for the user files, do not have the GID bit

So apache can read stuff which users upload, but it can't modify or write to those folders because apache and the user are in different groups. And the traditional solution is to give permissions to everybody. Which, if you have attacks like this one, may allow the attacker to modify the whole hosted tree, which proves that using chmod 777 systematically is a Very Bad Idea.

I have been finding LOTS of hacking attempts on several sites I manage. Each time I look at the server error log I can find entries like these ones:

[Tue Nov 29 00:47:32 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/xmlrpc
[Tue Nov 29 00:47:37 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/xmlsrv
[Tue Nov 29 00:47:43 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/services
[Tue Nov 29 00:47:48 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/blog
[Tue Nov 29 00:47:53 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/drupal
[Tue Nov 29 00:47:58 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/community
[Tue Nov 29 00:48:03 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/blogs
[Tue Nov 29 00:48:08 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/blogs
[Tue Nov 29 00:48:14 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/blog
[Tue Nov 29 00:48:19 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/blogtest
[Tue Nov 29 00:48:24 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/b2
[Tue Nov 29 00:48:29 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/b2evo
[Tue Nov 29 00:48:34 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/wordpress
[Tue Nov 29 00:48:39 2005] [error] [client 209.128.104.183] File does not exist: C:/Webroot/phpgroupware

I presume that the purpose of these little lamer script kiddies-hackers wannabe is to be able to enter a site through one of the xml-rpc interfaces that many web applications make available to be able to interact with other applications (via pings and so on), and then put an stupid message like "7H15 5173 |-|/\5 b€€|/| D3F/\C3D" (aka "This site has been defaced").

Well, guys, that's simply silly. Why don't you try and learn how to code something useful instead of using pre-made scripts to hack a site, or even better, stick your finger where you know and never put it out again!

And by the way, can somebody explain me if it corresponds to a new zombie-machines-exploiter virus? or is it just a brigade of bored teenagers?

Last time I did some tests with rubyonrails, I made it over a pc box. This time I decided to give another try on installing everything on my laptop, as it's easier to start and stop ruby applications from my own console and not by running a VNC viewer and then accessing the windows command line, etc.

I must recognize that my previous attempts on installing rubyonrails, apache2 or php5 on this computer left the web thingie completely unusable here. Even the apache server which came installed with tiger by default was not working anymore. Not to mention my ridiculous achievement: install all of rubyonrails, see it worked and do some simple scaffoldings and then destroy it again by trying to not to use webrick and use apache2 instead (which is what led to not to have even normal apache running).

But maybe today was my day and for some strange stars and planets alignment I tried again and finally got everything working. So, just in case it's useful for someone:

I started here, on How To Install on OS X Tiger. Then on point 3 I switched and followed the instructions at Darwin Ports, untill I got rails installed.

Then I came back to How To Install on OS X Tiger, and then to mysql website to get latest installable version of mysql for mac Os. They have a very nice installation program which will do everything for you.

Once that was done, I created the ruby mysql binding with sudo gem install mysql — –with-mysql-dir=/usr/local/mysql, so ruby can communicate with the mysql database. (Some tutorials say you need to put the whole /usr/local/mysql 5.5.0… directory… but that's not exactly true as the mysql installation procedure creates a symbolic link called /usr/local/mysql)

After that I needed a way of launching the applications and the ports and so on. On Rails On Osx I found there was an application framework called Locomotive which could help me in doing that. In fact, I think it has everything that I neded and if I had find it before I shouldn't have had to do the points above, but hey that's a way of learning.

Installed that one, launched it (Applications -> Locomotive folder -> Locomotive) and then created a simple application just for testing (click the + sign, obvious!) and started it. I went to the browser and typed in http://localhost:3000/ and yeeeeeha! it was working: Congratulations, you've put Ruby on Rails!

But I hadn't had enough… now was time for … apache2 and php5! Luckily I found a very nice article that explained everything very clearly, at phpmac: Building and Installing Apache 2 and PHP 5.0.5 on Mac OS X 10.4 Tiger. The only thing that was a bit tricky was the download locations for the source code files, both for apache and php. as sometimes the mirrors weren't working at all. I presume one can simply download the files manually and uncompress them where needed, without having to use curl for getting them.

In any case it worked like a charm. After several long minutes compiling and making configurations and installs (still don't get how people loves to rebuild and recompile the kernel just for the sake of doing it), seems like I have managed to get both running!

Needless to say, I have no words for this. I'm deeply impressed :D

UPDATE 7 december 2005: this tutorial seems to be interesting too: how to set apache2 to be the default mac os X httpd server.