Polyglot tracker module data decrunching, processing and crunching

I was sorting out a bunch of old digital files and folders and was quite amused at the realisation that it seems like I have written some sort of tracker module converter or reader in pretty much every programming language I’ve ever used.

NOTE: a tracker is a music creation program, and a module is how their files are called—they’re tightly optimised compressed binary files and use all sorts of tricks to save data.

Other people write To-Do apps to learn new languages, and I guess I write tracker module conversion tools? 🙃

So, in order of creation, as far as I can remember:

2005: C++: IT to events (in XML)

I wrote a utility in C++ to pre-process and parse pattern data from Impulse Tracker’s IT files, which would then be output as an XML file with event data, and then the C++ executable would use tinyxml to load the file. I am not entirely sure of why I didn’t just incorporate this event processing and reading directly into the executable instead of doing that at build time—I think I either didn’t have enough time to finish the exporter and I ended up generating the important events with Audacity or something like that, or my parser was very slow and I didn’t want the demo to take ages to load.

I can’t find this code, but I remember it was very buggy and I wasn’t progressing very fast. I wrote about it back then! There was a lot of confusion and non-written knowledge around the tracking format that I finally understood asking other people for help (e.g. how to calculate how much time do each row and each ‘tick’ take). Also, lots of segmentation faults and time spent thinking about memory and string allocations! And I was using the wrong tool for the job.

GitHub and Google Code didn’t even exist back then and SourceForge was terrible, so this is probably collecting digital dust in some folder.

2007: Ruby: XM to events

Apparently I also wrote a Fast Tracker’s XM to timed event utility in Ruby because I was going to work with a musician that used XM (hi JosSs!) and I was so into Ruby as server side and general utility language back then—the demo was going to be built in Flash with probably ActionScript 2.

I actually didn’t finish the demo, therefore the exporter is quite incomplete and I haven’t bothered publishing it, but it was interesting to use Ruby to process binary data from a file in a sort of stream like way.

In absence of a demo, here’s my favourite song from JosSs:

2008: PHP: IT to events (in a .h file)

Another iteration of this event list idea was to parse the data with PHP and generate a song.h file containing a big data array to be used in 64k intros, like this one.

I can’t find the code, but if I remember correctly, the nicest aspect of building this was that I could use some sort of text template to generate the .h file, and since string manipulations are way easier with PHP than with C, I got to the point where I wanted to be really quickly and without tons of segfaults.

2009: PHP: XRNS to MOD

I found another twist on my data conversion efforts: xrns2mod takes a Renoise song and converts it into an Amiga MOD file.

This is not a trivial conversion, as Renoise files are XML based and the player is significantly evolved compared to the MOD format: it can have a lot of channels, and more than one track per channel, plus arbitrary length patterns, plus a lot of instruments, and more than one sample per instrument, etc, whereas the most basic MOD files only have 4 tracks, use a limited note range, different base frequencies, less sample effects, and lesser quality samples, etc. So it’s not just moving numbers from one place to another.

I think I wanted to enter some MOD contest but I found all the MOD trackers quite uncomfortable compared to Renoise, and I wanted to have the best of both worlds: ease of use and also entering the contest. It’s incomplete and quite hardcoded in places, but it was already converting files quite decently the last time I ran it, so I put it on GitHub, just in case it maybe helps someone. Maybe.

Of course, minutes after I published it I found XRNS2XMOD, a similar tool which seems more complete as it also exports to XM in addition of MOD, but it requires Mono and what not.

~2010: Python + Lua: XRNS to events (in a .lua file)

I wrote an exporter in Python for generating a list of timed XRNS events and use them in a demo for ~~incredible synchronisation~~. The demo was run with Luisita which was my programmable graphics environment built with Lua and C++, and since the exporter generated a .lua file, it was very easy to load it all into an array like object at runtime.

This was used in this demo — as you can see the visuals are tightly related to the music:

I later tried to port this demo to WebGL, but I never finished it… perhaps I should just share the code for whoever was interested. It was a bit challenging to migrate because I was using OpenGL immediate mode in the Luisita version, so I needed to change my mental paradigms to WebGL/three.js’s and also write a shader or two to make it look as I wanted it to look… also those were a ton of events that I wanted to send to JavaScript, so my naive initial attempts were making the Garbage Collector work a lot more than it should have worked.

Funnily enough I remember being frustrated with not having tweening in the Lua version, but it could be ‘easy’ to add it to the web version using tween.js. But it’s not going to happen.

2013: node.js: XRNS to JSON

I also wrote a node.js module for extracting XRNS data into something the browser could use more easily (i.e. JSON). I guess not many people are using it or are interested in it, because 3 years later I haven’t heard of anyone improving or using it, but if you want you can install it with npm:

npm install renoise

I used this to generate the events and note data for the slides for my JSConf.EU 2013 talk in which I programmed virtual instruments to play a tracked song in the browser, built their interfaces with Web Components and also used hardware controlled/read with Open Sound Control and node.js! Yay to mixing everything together! 😎

Here’s the video starting with the actual demo:

Well, I hope you have enjoyed this tour, and remember to always use the best tool for the job 😉


ladieswhocode 20130508: adventures in paranoia with sinatra+sequel, and networking

ladieswhocode is an informal meeting that takes place every month in London (and I understand other cities too, but I haven’t been there so I can’t tell!). At the beginning I was a bit skeptical as I’m not a big proponent of gender division–I’d rather prefer everyone, no matter the gender, to be able to sit in the same room and listen to speakers of any gender as well. But after attending a few meetings I can understand why some women rather not go to the usual, 99% male, technical meetings. There’s quite a bit of fear about not being good enough, and it’s quite interesting to listen to more female developers share experiences that I thought were unique and isolated to my case… but apparently are not.
Continue reading “ladieswhocode 20130508: adventures in paranoia with sinatra+sequel, and networking”

A first impression on Ruby’s Mechanize

I had to do some screen scrapping yesterday and, while I previously have used hpricot for these kinds of things (or maybe even just plain cURL or similar), this time the page required me to be logged in, and not using any HTTP “standard” way of Auth that cURL could deal with, but a custom CMS html-based login form. So I understood that I needed something more powerful; something that could pass as a “proper” browser and not just a simple crawler.

I had heard quite nice successful experiences with Mechanize before, so I decided I would give it a try, and it turned out to vastly exceed what I expected from it! 🙂

Mechanize is a port of the original Perl Mechanize library; there is a Python port too, and there might be ports for other languages too but I wasn’t interested in that.

For some reason my mind was in the ruby-mood yesterday, so I decided to go for the ruby port. Everything was flowing nicely between ruby and me. I even dared prototyping what I wanted to do in a terminal, using irb, before writing the final script I needed to write. My ruby muscle was getting toned again, so to speak.

I got some false starts, though. I first invoked just ‘ruby’ in the terminal, only to be greeted with a waiting pipe, i.e. nothing happened as –I guess– ruby was expecting me to provide him with something to run. I remembered that the interactive ruby executable was called irb. A Control+C and four other keystrokes later, I was in immediate-mode ruby.

Another gotcha: not having to

require 'rubygems'

when running in irb, but having to when writing a script. Thankfully that I remembered and was able to fix quickly.

But I’m digressing. Back to Mechanize: it simulates a real browser interacting with a website. You can even fake the user agent. But unlike the most basic screen-scrappers, which simply download a page and then do something with it and then download another one and do something else, without keeping any sort of continuity between downloads and connections, Mechanize is stateful. Which means that it keeps the state between ‘visited’ pages. To all effects, and for the visited websites, there’s a normal browser at the other end of the line. Unless you go crazy and begin hammering a server with tons of requests, crawling websites this way might be almost invisible to servers.

Which is exactly what I wanted!

The syntax is quite nice and intuitive. Borrowing shamelessly from the manual:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://google.com/')

# List links on the page
page.links.each do |link|
  puts link.text, link.href

# Click on the first link with text 'News'
page = agent.page.link_with(:text => 'News').click

# Do a Google search, using the search form
google_form = page.form('f')
google_form.q = 'ruby mechanize'
page = agent.submit(google_form, google_form.buttons.first)

Something that made me even more happier is that Mechanize also uses Nokogiri internally for parsing HTML–which means we can do the same style of nice DOM tree traversing that I used to do with Hpricot, only even better! (Nokogiri is the successor to Hpricot).

I didn’t need that last feature for this particular case, but it left me wondering what I could use Mechanize for–in order to use this! I will have a look at my TODO list and see where can I use my newly acquired Mechanize skills!

My favourite GIMP plug-ins

When I used Ubuntu I normally installed a fancy package called gimp-plugin-registry that came with most of the plug-ins I liked, and then some more stuff that I never used. I had learnt to ignore that, but there was still the pain of not knowing where things were. Some filters went to “Colors”, others to “Light and shadow”, etc. It was extremely hard to remember where each filter was meant to be.

Now that I’m using Arch I couldn’t find any equivalent package but I thought that it was OK–that way I would only install what I really needed to install. In the process I’ve found a couple more plug-ins that I didn’t know about, so it’s been quite productive.

What I’ve done is I’ve written a script that takes care of downloading the scripts from the specified URLs, placing them in the appropriate gimp folder (~/.gimp2-6/scripts in my case), and the best of all is that it also patches them so that all are under the Filters/Photo menu entry. Yay! No more wandering around menus and submenus! My brain is now happy, I can just focus on playing with the pictures!

If you want to use this script just download it (it’s on my snippets repository). You might want to modify it as you wish. There are a couple of TO DO’s but the file is fully functional (as long as you’ve got Ruby installed).

Oh, and pardon my rudimentary Ruby. It’s been ages since I last wrote anything with Ruby. I still don’t know what I want to use. Ideally I’d like something as easy as php but with a syntax that is a mix of Python and Ruby, and lots of easy functional and metaprogramming.

I guess my problem is that I know too much, but still so little, not enough. Maybe if I just knew one language, I would be happy with that one. In any case, I’ll keep on experimenting and try to decide what I like best. Maybe my Japanese side will win over the European one? Who knows!

Meanwhile, be sure to enjoy the filters! Wooo!