Category Archives: Projects

Polyglot tracker module data decrunching, processing and crunching

I was sorting out a bunch of old digital files and folders and was quite amused at the realisation that it seems like I have written some sort of tracker module converter or reader in pretty much every programming language I’ve ever used.

NOTE: a tracker is a music creation program, and a module is how their files are called—they’re tightly optimised compressed binary files and use all sorts of tricks to save data.

Other people write To-Do apps to learn new languages, and I guess I write tracker module conversion tools? 🙃

So, in order of creation, as far as I can remember:

2005: C++: IT to events (in XML)

I wrote a utility in C++ to pre-process and parse pattern data from Impulse Tracker’s IT files, which would then be output as an XML file with event data, and then the C++ executable would use tinyxml to load the file. I am not entirely sure of why I didn’t just incorporate this event processing and reading directly into the executable instead of doing that at build time—I think I either didn’t have enough time to finish the exporter and I ended up generating the important events with Audacity or something like that, or my parser was very slow and I didn’t want the demo to take ages to load.

I can’t find this code, but I remember it was very buggy and I wasn’t progressing very fast. I wrote about it back then! There was a lot of confusion and non-written knowledge around the tracking format that I finally understood asking other people for help (e.g. how to calculate how much time do each row and each ‘tick’ take). Also, lots of segmentation faults and time spent thinking about memory and string allocations! And I was using the wrong tool for the job.

GitHub and Google Code didn’t even exist back then and SourceForge was terrible, so this is probably collecting digital dust in some folder.

2007: Ruby: XM to events

Apparently I also wrote a Fast Tracker’s XM to timed event utility in Ruby because I was going to work with a musician that used XM (hi JosSs!) and I was so into Ruby as server side and general utility language back then—the demo was going to be built in Flash with probably ActionScript 2.

I actually didn’t finish the demo, therefore the exporter is quite incomplete and I haven’t bothered publishing it, but it was interesting to use Ruby to process binary data from a file in a sort of stream like way.

In absence of a demo, here’s my favourite song from JosSs:

2008: PHP: IT to events (in a .h file)

Another iteration of this event list idea was to parse the data with PHP and generate a song.h file containing a big data array to be used in 64k intros, like this one.

I can’t find the code, but if I remember correctly, the nicest aspect of building this was that I could use some sort of text template to generate the .h file, and since string manipulations are way easier with PHP than with C, I got to the point where I wanted to be really quickly and without tons of segfaults.

2009: PHP: XRNS to MOD

I found another twist on my data conversion efforts: xrns2mod takes a Renoise song and converts it into an Amiga MOD file.

This is not a trivial conversion, as Renoise files are XML based and the player is significantly evolved compared to the MOD format: it can have a lot of channels, and more than one track per channel, plus arbitrary length patterns, plus a lot of instruments, and more than one sample per instrument, etc, whereas the most basic MOD files only have 4 tracks, use a limited note range, different base frequencies, less sample effects, and lesser quality samples, etc. So it’s not just moving numbers from one place to another.

I think I wanted to enter some MOD contest but I found all the MOD trackers quite uncomfortable compared to Renoise, and I wanted to have the best of both worlds: ease of use and also entering the contest. It’s incomplete and quite hardcoded in places, but it was already converting files quite decently the last time I ran it, so I put it on GitHub, just in case it maybe helps someone. Maybe.

Of course, minutes after I published it I found XRNS2XMOD, a similar tool which seems more complete as it also exports to XM in addition of MOD, but it requires Mono and what not.

~2010: Python + Lua: XRNS to events (in a .lua file)

I wrote an exporter in Python for generating a list of timed XRNS events and use them in a demo for ~~incredible synchronisation~~. The demo was run with Luisita which was my programmable graphics environment built with Lua and C++, and since the exporter generated a .lua file, it was very easy to load it all into an array like object at runtime.

This was used in this demo — as you can see the visuals are tightly related to the music:

I later tried to port this demo to WebGL, but I never finished it… perhaps I should just share the code for whoever was interested. It was a bit challenging to migrate because I was using OpenGL immediate mode in the Luisita version, so I needed to change my mental paradigms to WebGL/three.js’s and also write a shader or two to make it look as I wanted it to look… also those were a ton of events that I wanted to send to JavaScript, so my naive initial attempts were making the Garbage Collector work a lot more than it should have worked.

Funnily enough I remember being frustrated with not having tweening in the Lua version, but it could be ‘easy’ to add it to the web version using tween.js. But it’s not going to happen.

2013: node.js: XRNS to JSON

I also wrote a node.js module for extracting XRNS data into something the browser could use more easily (i.e. JSON). I guess not many people are using it or are interested in it, because 3 years later I haven’t heard of anyone improving or using it, but if you want you can install it with npm:

npm install renoise

I used this to generate the events and note data for the slides for my JSConf.EU 2013 talk in which I programmed virtual instruments to play a tracked song in the browser, built their interfaces with Web Components and also used hardware controlled/read with Open Sound Control and node.js! Yay to mixing everything together! 😎

Here’s the video starting with the actual demo:

Well, I hope you have enjoyed this tour, and remember to always use the best tool for the job 😉

 

Making things, and a very annoying thing

Last holidays I consolidated and “inventoried” all the various kits of electronics / making and mobile phones that I had amassed over the last years. And I decided that instead of gathering dust or just disposing of them, these things must be turned into something.

They don’t need to be something useful, but they need to be something that does something, even if it’s useless.

That’s the perfect excuse to finally learn some hardware and maybe contribute to the Internet of Things. Ha!

Today I brought back to life an old Arduino Duemilanove board. I think it’s from 2009 but it could also be from 2008. It was gathering dust on a box, but not anymore. It took me a while to figure out what the method to upload my sketches to the board should be.

In theory I should just be able to open the Arduino IDE, select the board (Duemilanove or Diecimila), the chip (ATMega 168) and the port (/dev/tty.usbserial-blabla). And then I could push sketches to the board as many times as I wanted.

In practice I can only upload once and then the next time I have to disconnect and reconnect or else I get an error message about unresponsive programmers:

avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 1 of 10: not in sync: resp=0x00

Some people say that they found the same issue and wrote a Python script to reset the board before uploading stuff. The official documentation says that these boards are able to be reset before uploading, but maybe they aren’t. I tried using the script, but I of course stumbled upon Python lacking the Serial library on my system and I gave up because installing things in Python is not only like opening another can of worms, but also being forced to chew on a few of them: DISGUSTING.

I found out that I can brute force reset the board by… disconnecting it every time.

This is horrible, but works.

And without further ado, allow me to introduce a very annoying thing I just made:

I was a bit tired of always using LEDs as the “Hello World” of Arduinos. I had a couple of buzzers around and it turns out that you can replace the LEDs with a buzzer, and it will beep instead of blink. And all using the same sketch code!

void setup() {
  pinMode(13, OUTPUT);
}

void loop() {
  digitalWrite(13, HIGH);
  delay(100);
  digitalWrite(13, LOW);
  delay(1000);
}

This sounds as annoying as a smoke alarm. Perhaps my next creation will be a Prank Smoke Alarm!

tween.js mega changes

Yesterday I had my every-two-months lucky day in which I could sit down and work on tween.js, and DID I GET THINGS DONE!!!

The first thing I did was to get rid of the minified version. Since the build process wasn’t fully automated, I often forgot to produce and check-in that minified version, and people who used it would get all sorts of weird errors that I didn’t see (specially in Safari and iOS, as the polyfills we added would be in the uncompressed version, but not on the minified, sigh!).

Then I started using semantic-release as my invaluable helper for producing releases. Each time a push to the git repository happens, another service (Travis) runs a battery of tests to make sure nothing is broken. If the tests pass, semantic-release will get in action and (probably not in this order):

  • determine what’s the next “semver” for the package. This is a function of the type of commit you made (a bug fix, a new feature, docs, chore…). Important / breaking commits will cause bumps in the first digit, etc. (I suggest you read more on semver if you’re interested). The type of commit is specified by having the commit message follow a certain syntax. E.g. a feature will be feat: implement feature A
  • tag the commit with the version. E.g. v16.0.1. I believe this is what bower people use and desperately need, and I never provided because I don’t use bower and so didn’t notice.
  • create a github release changelog thingie in github. These go to the releases page in github
  • publish the new version of the package to npm

I think semantic-release can do so much more than this, but just having all these steps performed for me is A W E S O M E. So once I established this “infrastructure” I could go on and fix many other long-standing issues and also merge PRs and address questions.

Since we don’t have to produce a minified version, I got rid of gulp which is what I was using it for. Installing tween.js with npm is now very very lean because I also added an .npmignore and so it just essentially installs the code of the library only. Your trees will not include the examples anymore. Not that it was incredibly big but every byte counts to some people, it seems 😛

I also added jshint (for code correctness) and jscs (for code style) verification as part of the test suite. This was something that would put me off reviewing PRs… and specially explaining to people that it was not OK to change the whole whitespace in the file, or that they should respect the existing guidelines (even if it’s in the contributing file that very few read). So the rules are now there, and everyone has to abide to them, or the tests don’t pass and so the PRs are not accepted.

Interestingly, I added these steps using the advice in Kate Hudson’s talk from Nordic.js front-end automation with npm scripts, where she showed how you don’t actually need a task runner–I recommend you to watch it! Or check out her reading list on the topic.

Next up was dealing with a ton of sorta old and sometimes outdated PRs that had been lying in the guts of github for months. As I explained in my previous tween.js post, something had happened and I hadn’t even seen the notifications for these.

I prioritised FIXES first. Many people are coming up with some novel ideas and features, and I’m grateful for that, but I decided to focus on accuracy and robustness for now. Some aspects of the code are a bit obscure and I am not sure I understand them well, mostly because I just merged them in when someone proposed them, and now I’m paying the price when strange edge case errors are reported.

Of course, I’m still not done by any means, because there was a massive backlog and the day only has so many hours, and I’d like to do human things such as sleeping, etc.

This is brings us to this interesting paradox: many people use tween.js, including big agencies who charge a bunch for their projects, but only a very few submit code or respond to my pleas for help. Maintaining a JS library has become way more demanding over the years. Back in 2011 people didn’t care about npm, bower, tags and releases and what not, so working on tween.js was way more simple and less time consuming. I could have just put a zip file on a website, and people would be happy with that, for all that is worth.

But since I changed roles at Mozilla I am travelling a lot more, and it literally devours your time. I am not complaining about that, but I need quiet time to sit and get heads down on code, and I’m not having much of that lately. Mozilla supports me working on tween.js but I do have my own work duties which take priority. All my coding time is happening during working hours, and when I’m off work I like to enjoy my free time doing things such as being outdoors or just talking to people face to face, not via a bug tracker. Or despair about the list of issues and PRs growing and me not having time to even acknowledge them ?

But before totally blaming open source for its toxicness, I decided to own this a little bit myself, and totally revamped the README file to make it a bit more welcoming and clear (I took heavy inspiration from Express). I have also filed some bugs and tagged them as help needed or good first bug respectively. If you enjoyed tween.js and want to give back by contributing to these bugs, you’d gain extra points of awesomeness.

People like roadmaps, so this is what I’d like to see next:

  • Review all pending bugs and PRs and resolve or close them.
  • Fix the things that have to be fixed and ensure all code is tested and clear before adding new features, because it is getting to a point where it is unwieldy and scary to even look at a diff (what even does this thing do!?). Hopefully the new automation will help here, and we can focus on logic and not on chores!
  • Divert all new feature ideas to the future ES6/ES2015/ESWHATEVER version of tween where everything will be super modular and you should be able to use parts of it as you need and hack other types of tweening engines as you see fit.

This is it for now. Thanks for reading, and happy tweening!

A three.js npm template

rotating demo cube

I can’t imagine dropping a series of <script> tags on a page to build a minimally complex website anymore. I’ve got used to browserify and so I can’t go back to building my fancy 3D thingies like it was 2012 all over again.

So I made this minimal template that sets the bare minimum you might need to build a three.js powered site using browserify via gulp. That lets you use any other npm-based modules and libraries you want! So your code should not be messy and/or a tremendously big file with everything tucked on it.

There’s also some extra help like a file watcher… and not much more, really.

Do let me know if you use it for something cool… or if you can/want to help on this issue! 😀

safeRegisterElement

For the project I’m working on I’m building web components that sometimes require other components to build their UI. For example, the theremin-ui uses the slider. So they will try to load and register the component before creating instances of it. It worked fine when there was only one level of dependencies, but then I put all of the components on the same document, to let the user choose an instrument from the collection of Web Audio instruments, and I got an error from the browser complaining about… something:

NotSupportedError: Operation is not supported

I traced it down to the register() call, and I figured that I was registering an element twice. So I made safeRegisterElement, a one-function module I’m using now instead of directly calling document.registerElement:

var safeRegisterElement = require('safe-register-element');

// safeRegisterElement(type, prototype);
// e.g.:
safeRegisterElement('custom-name', customElementPrototype);

All my custom element modules have a register method that uses safeRegisterElement internally (this lets you register the element with whatever name you want to use).

In npm: https://www.npmjs.com/package/safe-register-element
And sources: https://github.com/sole/safe-register-element