I had heard quite nice successful experiences with Mechanize before, so I decided I would give it a try, and it turned out to vastly exceed what I expected from it! :-)

Mechanize is a port of the original Perl Mechanize library; there is a Python port too, and there might be ports for other languages too but I wasn’t interested in that.

For some reason my mind was in the ruby-mood yesterday, so I decided to go for the ruby port. Everything was flowing nicely between ruby and me. I even dared prototyping what I wanted to do in a terminal, using irb, before writing the final script I needed to write. My ruby muscle was getting toned again, so to speak.

I got some false starts, though. I first invoked just ‘ruby’ in the terminal, only to be greeted with a waiting pipe, i.e. nothing happened as –I guess– ruby was expecting me to provide him with something to run. I remembered that the interactive ruby executable was called irb. A Control+C and four other keystrokes later, I was in immediate-mode ruby.

Another gotcha: not having to

when running in irb, but having to when writing a script. Thankfully that I remembered and was able to fix quickly.

But I’m digressing. Back to Mechanize: it simulates a real browser interacting with a website. You can even fake the user agent. But unlike the most basic screen-scrappers, which simply download a page and then do something with it and then download another one and do something else, without keeping any sort of continuity between downloads and connections, Mechanize is stateful. Which means that it keeps the state between ‘visited’ pages. To all effects, and for the visited websites, there’s a normal browser at the other end of the line. Unless you go crazy and begin hammering a server with tons of requests, crawling websites this way might be almost invisible to servers.

Which is exactly what I wanted!

The syntax is quite nice and intuitive. Borrowing shamelessly from the manual:

Something that made me even more happier is that Mechanize also uses Nokogiri internally for parsing HTML–which means we can do the same style of nice DOM tree traversing that I used to do with Hpricot, only even better! (Nokogiri is the successor to Hpricot).

I didn’t need that last feature for this particular case, but it left me wondering what I could use Mechanize for–in order to use this! I will have a look at my TODO list and see where can I use my newly acquired Mechanize skills!

Now that I’m using Arch I couldn’t find any equivalent package but I thought that it was OK–that way I would only install what I really needed to install. In the process I’ve found a couple more plug-ins that I didn’t know about, so it’s been quite productive.

What I’ve done is I’ve written a script that takes care of downloading the scripts from the specified URLs, placing them in the appropriate gimp folder (~/.gimp2-6/scripts in my case), and the best of all is that it also patches them so that all are under the Filters/Photo menu entry. Yay! No more wandering around menus and submenus! My brain is now happy, I can just focus on playing with the pictures!

If you want to use this script just download it (it’s on my snippets repository). You might want to modify it as you wish. There are a couple of TO DO’s but the file is fully functional (as long as you’ve got Ruby installed).

Oh, and pardon my rudimentary Ruby. It’s been ages since I last wrote anything with Ruby. I still don’t know what I want to use. Ideally I’d like something as easy as php but with a syntax that is a mix of Python and Ruby, and lots of easy functional and metaprogramming.

I guess my problem is that I know too much, but still so little, not enough. Maybe if I just knew one language, I would be happy with that one. In any case, I’ll keep on experimenting and try to decide what I like best. Maybe my Japanese side will win over the European one? Who knows!

Meanwhile, be sure to enjoy the filters! Wooo!

So there I went after mentally memorising the map from Liverpool Street Station to LBi‘s offices in Brick Lane. Funny because I have done the Brick Lane-Liverpool St. route several times already, but since it was dark and I got an excellent guide, I never paid much attention. Anyway, I arrived, gave my name to the security guy which was kind of confused with my Spanish name and then went downstairs where the people with laptops on the couches were.

I took a look and didn’t know anyone — so I decided the direct approach would be better: waved and said “hi” to whoever was looking at me at that point, and got a friendly response. Yay!

In a minute I was happily talking with Francesca, a nice front end web developer, then with someone else whose name I didn’t find out, then I got to associate Julian’s name with his face. A little while after, Paul Carvill came downstairs with lots of pizza boxes –all vegetarian, and yummy!– and as we were discussing about Rails 3.0, Joanna Geary made a little speech, more or less on the lines of this:

‘Journalists, raise your hands!’

They do.

‘Journalists, find a developer and explain them what do you have in mind that could be done with code, and let’s see if it’s doable or not!’

And then a girl approached Francesca and me. She looked strangely familiar but I couldn’t really tell why. We sat in a table, took the laptops out of their sleeves and set out to try and find if the data we needed was available. But the wireless didn’t quite work. So we chatted about something fascinating and kind of related to the topic: Volcanoes! It really helped that she had a degree in Geology so she could really offer interesting insight about how volcanoes work. Also, amazing as it sounds, she could pronounce Eyjafjallajökull in one go without hesitation!

We ate some more pizza, and then the wireless started working. So we checked and saw that the data wasn’t available — but we agreed she would contact the organisation and ask them for it. Maybe they already have it but it’s not publicly advertised.

Then it’s when the real coding part began; she said she would love to learn programming, but didn’t know where to start, or the guides she had tried were boring. I immediately proposed Processing; I know there’s Hackety Hack by _why, which is ruby based, but I don’t have any experience with it, so I went for the safer option.

I actually hadn’t done much Processing recently and as such I always forget the names of the basic functions (was it setup, or was it load? was it loop or was it play?) but it was even better, so that way they could follow along. She downloaded Processing and was typing in stuff in the sketchbook immediately… which was amazing! No setup, no preparation, no rebooting the computer after the installation; just download and experiment!

We’ve done just extremely simple things such as opening a window, setting its background colour, changing its size and… drawing a line! Still, I find it really enlightening to observe how ‘normal people’ (i.e. not hardcore developers) react to all these concepts when exposed to them.

Interesting things I’ve noticed (teachers, take note!):

• people love to see results immediately – no theory to start with!
• people love to have some starting snippets, which they can modify and experiment with, learning how they work meanwhile

So that makes Processing an excellent programming environment for beginners, because it’s got that beatiful reference written in pretty much plain language and with example snippets that can be copied and pasted for each function/class.

In a few minutes the word had spread: there were people doing stuff with Processing in the room! And so a little audience gathered, another guy (a journalist) came with his laptop, downloaded Processing and was doing little simple things as the aforementioned ones, copying code from her wife’s screen –isn’t it sweet?

On the other side of the table I was showing Paul and Francesca my word processing and statistical experiments with Tolkien works and we also spoke about Processing versus other environments such as Javascript or Flash.

In between, the girl had told us her name, and if the face looked familiar, the name was insanely familiar! She humbly told me she was in lots of events and etc, but didn’t make a big thing of it. I asked her about events I had been in (that I could recall at that point) to no avail –we didn’t seem to have been in the same events. But then I came home and looked for “Suw Charman” and found that not only had I seen her speak at resfest’05 but she had also posted a few messages to a mailing list I follow. I think that’s why the name sounded sooo familiar!

Of course, there couldn’t be a nice event without Rob McKinnon in it! I just love how I always find him everywhere I go, whether it’s a full fledged event or just a coffee shop anywhere in London, it’s just so funny. Couldn’t exchange more than a few “hey hey hi!” waves because we both were busy with our new friends, so I don’t know which sorts of interesting stuff did he show this time (the last time he spoke about Hpricot).

The event ended at around 22h, but mainly because people had to leave and take trains and all those things, but I’m pretty sure otherwise we could have stayed and playing with Processing for a long while :-)

And that’s how ruby in the pub, “with hardly any ruby and certainly no pub”, was. See you in the next one?

As you can see, I didn’t download any source file, instead I was happy with using apt-get and the hpricot version from ubuntu repositories, although they are relatively old (for example rubygems is more than a year old). If I find any problem and need to update to newer versions I’ll report that here ;-)

It’s a pity they don’t have a metapackage for ruby’s development files (the ruby1.8-dev package), the same way there’s a ruby metapackage which depends on the ruby1.8 package, so whenever ruby is updated it will update the ruby version as well, without the user having to worry about the version number.

Even more, I instinctively tried a naive sudo apt-get install rubydev and was greeted with a sad “Couldn’t find package rubydev”. It somehow proves that a metapackage called rubydev would be quite useful… at least for instinctive users.

Enjoy your screen scrapping!

But how do you detect if there’s a description? It seems the answer was rather simple: use term.next and if the next element’s name is dd, we’re lucky and have a description. The only problem was that I didn’t know how to access the name of an element, until I just thought: what if I simply use name? and guess what… it worked! So term.next.name was exactly what I looked for :-)

Source at supersnippets.

I also extended this a bit to save the results into a database, using ActiveRecord, but since each db schema is a different world, I didn’t post that version here. If anybody thinks it might be useful just let me know.

Also, this code is not very rubyesque yet, suggestions in order to improve it will be really appreciated. I’m specially thinking about the if … else parts, I’m pretty sure there’s a way to shorten those lines :-)