I had heard quite nice successful experiences with Mechanize before, so I decided I would give it a try, and it turned out to vastly exceed what I expected from it! :-)

Mechanize is a port of the original Perl Mechanize library; there is a Python port too, and there might be ports for other languages too but I wasn’t interested in that.

For some reason my mind was in the ruby-mood yesterday, so I decided to go for the ruby port. Everything was flowing nicely between ruby and me. I even dared prototyping what I wanted to do in a terminal, using irb, before writing the final script I needed to write. My ruby muscle was getting toned again, so to speak.

I got some false starts, though. I first invoked just ‘ruby’ in the terminal, only to be greeted with a waiting pipe, i.e. nothing happened as –I guess– ruby was expecting me to provide him with something to run. I remembered that the interactive ruby executable was called irb. A Control+C and four other keystrokes later, I was in immediate-mode ruby.

Another gotcha: not having to

when running in irb, but having to when writing a script. Thankfully that I remembered and was able to fix quickly.

But I’m digressing. Back to Mechanize: it simulates a real browser interacting with a website. You can even fake the user agent. But unlike the most basic screen-scrappers, which simply download a page and then do something with it and then download another one and do something else, without keeping any sort of continuity between downloads and connections, Mechanize is stateful. Which means that it keeps the state between ‘visited’ pages. To all effects, and for the visited websites, there’s a normal browser at the other end of the line. Unless you go crazy and begin hammering a server with tons of requests, crawling websites this way might be almost invisible to servers.

Which is exactly what I wanted!

The syntax is quite nice and intuitive. Borrowing shamelessly from the manual:

Something that made me even more happier is that Mechanize also uses Nokogiri internally for parsing HTML–which means we can do the same style of nice DOM tree traversing that I used to do with Hpricot, only even better! (Nokogiri is the successor to Hpricot).

I didn’t need that last feature for this particular case, but it left me wondering what I could use Mechanize for–in order to use this! I will have a look at my TODO list and see where can I use my newly acquired Mechanize skills!

As you can see, I didn’t download any source file, instead I was happy with using apt-get and the hpricot version from ubuntu repositories, although they are relatively old (for example rubygems is more than a year old). If I find any problem and need to update to newer versions I’ll report that here ;-)

It’s a pity they don’t have a metapackage for ruby’s development files (the ruby1.8-dev package), the same way there’s a ruby metapackage which depends on the ruby1.8 package, so whenever ruby is updated it will update the ruby version as well, without the user having to worry about the version number.

Even more, I instinctively tried a naive sudo apt-get install rubydev and was greeted with a sad “Couldn’t find package rubydev”. It somehow proves that a metapackage called rubydev would be quite useful… at least for instinctive users.

Enjoy your screen scrapping!

But how do you detect if there’s a description? It seems the answer was rather simple: use term.next and if the next element’s name is dd, we’re lucky and have a description. The only problem was that I didn’t know how to access the name of an element, until I just thought: what if I simply use name? and guess what… it worked! So term.next.name was exactly what I looked for :-)

Source at supersnippets.

I also extended this a bit to save the results into a database, using ActiveRecord, but since each db schema is a different world, I didn’t post that version here. If anybody thinks it might be useful just let me know.

Also, this code is not very rubyesque yet, suggestions in order to improve it will be really appreciated. I’m specially thinking about the if … else parts, I’m pretty sure there’s a way to shorten those lines :-)

First I created a simple test page. It’s got some html elements, some have id’s, some contain certain text nodes. It looks like this:

The question was how to remove certain individual elements given certain conditions – more specifically, when the element attributes matched a condition. I don’t see why he had problems removing stuff with the remove method, since that’s what I have used. Since search returns a collection of elements, you just need to get a collection which contains only the element you want to remove, and then apply remove to that collection.

Here are three examples:

Removing the paragraph with id = bad_attribute

We find out the element using CSS selectors, where the hash means ‘id’.

Removing all the unordered lists (ul’s) which have an style attribute

Again, using CSS selectors:

There’s more info about CSS selectors in the Hpricot CSS search documentation. One can get very creative with this and allows for filtering almost everything!

Removing elements whose contents match certain conditions

When it’s not enough with CSS selectors, we can perfectly take advantage of ruby!

For example, if you want to remove list items (li’s) whose text doesn’t begin with E, you could do it with this:

which is the same as saying:

• Look for every list item in the document
• Take the results of that search (which is an Array of Hpricot Elements) and apply the collect! function to them
• collect! executes the code in the block for each element and stores the return value in an array
• But as it can return nils (when the inner_text doesn’t begin with ‘E’ and hence doesn’t match our little regular expression), we remove nil values from the array with compact, so that we don’t get errors when removing.
• And finally, remove the elements which are in the resulting array, with the classical Hpricot remove

Note how I used collect! instead of just collect, so that the changes are applied over the search results, and we don’t get a new array instead.

You should try using collect instead of collect!, and removing compact from the chain, to see what happens.

Final result

If one applies all these evil removals to the original code, the final result is this:

Pretty empty, isn’t it?!

Download these examples

I’ve uploaded the hpricot_remove_elements.rb and test.html together in a zip file: hpricot_remove_elements.zip. For running it, just unpack, and type ruby hpricot_remove_elements.rb

Or open with textmate and press Option+R ;-)

Since I saw one demonstration by Rob McKinnon at certain LRUG meeting, I have been willing to try Hpricot, but I hadn’t seen an application for it yet. No more! I found myself today wanting to extract data from a table in a web page and suddenly I thought: this is a job for Hpricot!. More specifically, I wanted to extract these EXIF tags, and I simply couldn’t accept the mere thinking of entering that data manually. It needed to be automated!

Getting it

Getting Hpricot is very easy:

(if you’re picky you can try more exotic ways of installing in its homepage).

if you’re in windows, of course.

Understanding it is easy as well, specially if you have used jquery before. It’s all about writing selectors for looking for things, so it helps a lot if the HTML document is well marked. Otherwise, you might have to end up doing lots of workarounds or extra code that could be avoided simply by having a class or id specified in the relevant elements.

Inspecting & traversing

So, once I got the library installed, I took a look at the page source code with Firebug. It is specially useful for this kind of jobs because it helps you to visualize the hierarchy of elements in the page, including classes and id’s, so you don’t have to traverse manually the HTML tree to gather the data you need.

What I was looking for was the table which contained the relevant data. In this case, we’re lucky and even if the table hasn’t got an id attribute which would make it uniquely identifiable in the whole document, it still has class=”inner”, which happens to be used only once in it, thus acting effectively as an element identifier.

Note how Firebug is showing the tree path for the selected table. If we didn’t have the class attribute, we would need to use a selector like “/html/body/blockquote/table/tbody/tr/td/table”, but it will be something as simple as “/table.inner”.

Hands on Ruby

Ok, so this is where we write a few lines of code which do a lot ;-)

First come the usual series of requires:

Rubygems is required in order to load hpricot, and open-uri is required in order to directly read data from a URI. open-uri comes with ruby, so we don’t need to install anything else.

Now we need to get the HTML file. It is as simple as

but since I was doing lots of tests and didn’t want to overload that guy’s server, I simply saved the document as EXIF.html and loaded it with this instead:

At this point we have the HTML document in the doc variable, so what are we waiting for?
We initialize a rows variable for holding the data that we’ll extract:

And now comes the real fun!

Ok, not that fast. I’ll elaborate a little more on the juicy bits.

This is the key for reaching the main data. It’s like saying I’m looking in doc for all the rows (the tr’s) which are contained in a table whose class equals ‘inner’. When we use a / it means we want an immediate child. // means a child below the element. As I said before, it’s all about selecting and traversing the tree.

With the last line of code, we get returned the content of each tr into the row variable. We can continue extracting data from within row, and that’s exactly what we do with

That one provides us with all the td elements immediately below the current row.

When we reach the td elements, all that is left is to extract the data for each cell and push it into the cells array, which will be pushed into the rows array. But we don’t just copy the cell data as it is; some cells contain notes, and some of those notes contain lists of values. I think we can all agree that those lists of values are commonly called Hashes, and they undoubtedly deserve an special treatment!

So that’s why I’m checking for the existance of an span with class == s inside each cell. If we find one, there’s a note in this row, and probably there’s one hash with values. I would say this is the funniest part of all:

I’m making use of the fact that each invoked function is returning another object, so that I can chain them consecutively instead of doing a series of assignments. And it reads like this: Take the html inside the span with class s, split it where you find a br, and for each of those split parts remove the surrounding whitespace and split it again where you find a =, so we get a pair of key-value values, remove the whitespace for those pairs as well and put them in a new Hash.

At the end we finish with an array of rows and cells, where certain cells occasionally contain a Hash with the constants used by the row EXIF tag.

It’s also interesting to note that the first row is unusable, because it corresponds to the th elements, so we’ll simply do a

and it’s gone. And to top it all, we could output the rows array to a yaml file, so that we do not need to run this each time we need the list of EXIF tags.

Arrays in ruby have a lovely method called to_yaml which dutifully generates a version of the array in yaml syntax. And it’s very easy to output that to a file:

And you’re done! I hope you liked this small Hpricot tutorial/introduction… and if you have any suggestion or improvement please let me know!

Of course, you can get the complete source code here: hexif.rb. It is a ridiculous 61 lines, including some commented lines and white spaces. Come on get it and do something cool!