Don’t force users to install node modules globally when you can avoid that

How often have you seen instructions for installing a project that are something like this?

git clone https://wherever/the/project/is
cd cloned_folder
npm install -g gulp # <-- 😱

(or, instead of gulp, any other utility that the project requires for development)

This is not a good idea. You’re installing the same version of a utility globally, and perhaps the project has not been tested with that version. Things might break or even worse, work not-so-well in a strange, undebuggable way!

Don’t do it.

A better way is to use npm scripts, which belong to the project, and specify development dependencies in package.json, so when you run npm install, it also installs devDependencies locally to the projects’ node_modules folder. This keeps everything under control and does not pollute other people’s computers global space.

So, for example, if a project was asking users to run npm install -g gulp, they should do this instead:

  1. Install gulp locally as a development dependency, saving it to package.json:
    npm install --save-dev gulp
  2. Add a new entry to the scripts array in package.json:
    "build": "gulp"
  3. Ask users to run
    npm install

    the first time. And

    npm run build

    each time they need to build the project

The reason this works is because when npm runs a script, it modifies the current PATH variable to add node_modules/.bin, which contains any binaries that you installed as part of your modules. So your computer will search for binaries starting in the .bin folder, before searching in any other default location for binaries in your computer (e.g. /usr/bin)

You can find more documentation about this behaviour in the run-script page of the npm manual. I really encourage you to read it as it has lots of good stuff and Things You Wished Had Known Before 🙂

And in case it makes things clearer, this is how a hypothetical package.json would look before:

{
  devDependencies: {}
}

And after:

{
  devDependencies: {
    "gulp": "12.34.56"
  },
  scripts: {
    "build": "gulp"
  }
}

With this, the devDependency is clearly stated on the package.json file, the project developer will be developing with that version, and will not get surprises! (and the users of the module won’t either).

Another advantage of this approach is that you are not tying the project to a specific utility.

If you decide to use grunt instead of gulp later on, you just need to do whatever changes you need (Gruntfile instead of Gulpfile, edit package.json etc), and yet users of the project will still start the build process with npm run build.

It’s invisible to them (except perhaps running npm install again)!

You’re hiding internals that they do not necessarily need to know about, so it’s a good abstraction. Seriously—you won’t even need to update your getting started guide! Good stuff.

Hope that helps!

If you want to learn even more cool npm scripts tricks, I recommend Kate’s talk and accompanying post. Actually, I just realised she explained the same thing I did, so here’s her take, haha!

Extreme decoupling or all-as-a-module

I opened my laptop in the morning and found one of my open tabs in Nightly was for Vue.js. I don’t even remember how I ended up there. Was I reading about frameworks? Did anyone send me the link? Who knows!

But I was curious. I am not a megafan of frameworks, but I like looking at them. One, because their idioms are often adopted by other developers, so it’s good to be aware of where are things going. And two, because frameworks do lots of “magic” in the background, and I always want to know how they implement their “magic”—maybe I’ll want to adopt some of it!

So instead of closing the tab, I perused the page. It has a virtual DOM as React does, but they seem to take great pride on their overall minimalism (small file size, little intrusiveness). The examples are amongst the most readable I’ve found for frameworks when it comes to the JavaScript API; the HTML directives are as alien-feeling as most frameworks.

Later I was discussing this strange incident with friends (“I found an open tab in my browser—do you think this is a signal from the universe that I should get into Vue.js?”) and Irina also highlighted the fact that Vue.js “components” might be simpler to build than the equivalent in React, and also be less coupled.

This derived into talking about The Dream:

You know what the dream is? Have everything be an npm package that I can plug in to any framework I like. And everything is packages packages packages

📦.js

Oprah giving free packages away to everyone
You get a package! And you get a package! And you get a package! And you get a package! And you get a package… everyone gets a package!

(Irina demanded an Oprah themed meme)

And of course this reminded me to earlier conversations with chameleonic Jen about modularising everything and maximising reuse. She would propose, for example, abstracting a card game into separate modules; one for handling the rendering, other for handling card games in an abstract way, another one for handling a specific type of game. This way you could build multiple games by just providing an implementation for the specific game. (Games are notoriously often not built this way).

Likewise, Aria talked about radical modularity at Web Rebels and the notion that if your modules are small enough, they are done. Finished. You rarely need to ever touch them again (unless there’s a bug). Watch the talk: it’s very inspiring.

I really like this “pure” idea, and can work very nicely as long as you keep your data and logic separate from your view.

Unfortunately, the issue is that UI code often intermingles both data and view, so you end up declaring your data as instances of whatever base component the UI is using, which is not very maintainable on the long run. If you want to change the UI you will need to take the ‘data’ bits out of the UI code, or write some sort of adapter between “UI code” and “data”, to have to only change “adapter” when you decide you don’t like your current view layer. This could be a performance hit, so you might want to sacrifice flexibility for performance.

But hey… everything in computing is always a trade-off!

 

Volumio: a Raspberry Pi jukebox

Volumio playing aerodynamisk neger

Last Sunday, I spent some time tinkering with my Raspberry Pi 3, trying to get some sort of jukebox software up and running.

I finally settled on something called Volumio. It is a complete distribution that you install on the Pi and then it becomes a machine that plays many audio formats and that you can control from other devices, using a web interface.

And thanks to the Raspberry Pi 3’s WiFi, I just need to connect the audio jack and the power cable, and then the little machine can hide under the table with the rest of cables and other devices I don’t want to look at generally (NAS, modem, etc).

Volumio is installed by flashing a complete distribution onto a mini SD card. I had a small 4GB one roaming around so I used that. It’s cool because I can just swap SD cards and boom! I have a completely new environment without destroying the previous one.

The “official stable” builds that you’re directed to from their website won’t work with the Pi 3. They get stuck at the colourful loading pattern. I found someone had had the same problem, and the solution was to flash another image for Volumio 2 RC2. I flashed this new version, which is very hard to locate unless you know that such a thing exists or read the forum, and the initial boot didn’t show up the device on Bonjour, but after a reboot I could see it and start using it.

I configured it using the web interface, accessible at volumio.local. I told it to use WiFi (it worked!) and also to use my NAS music library (which also worked!). Then I shut it down again, moved everything out of my desk, and turned it on again to enjoy some music.

Apparently the initial Volumio front-end was built with PHP. Now they are rebuilding everything from scratch as Volumio 2, using Node.js, Socket.IO and Angular. I was excited about the node.js bit! But although there’s plenty of documentation around, I felt like I couldn’t understand where things were in the source, in the distribution, how to get a developer environment running, etc. Maybe it’s because I came back from holidays on Saturday and I’m still pretty jetlagged and my brain is not very functional yet, but maybe it’s also for the best: I surely don’t need to contribute to yet another project.

Anyway, I have a working jukebox. It can even play modules! It’s a bit basic at points (e.g. you cannot navigate by ‘genre’ or other MP3 metatags) but it is getting the job done so I’m happy.

Polyglot tracker module data decrunching, processing and crunching

I was sorting out a bunch of old digital files and folders and was quite amused at the realisation that it seems like I have written some sort of tracker module converter or reader in pretty much every programming language I’ve ever used.

NOTE: a tracker is a music creation program, and a module is how their files are called—they’re tightly optimised compressed binary files and use all sorts of tricks to save data.

Other people write To-Do apps to learn new languages, and I guess I write tracker module conversion tools? 🙃

So, in order of creation, as far as I can remember:

2005: C++: IT to events (in XML)

I wrote a utility in C++ to pre-process and parse pattern data from Impulse Tracker’s IT files, which would then be output as an XML file with event data, and then the C++ executable would use tinyxml to load the file. I am not entirely sure of why I didn’t just incorporate this event processing and reading directly into the executable instead of doing that at build time—I think I either didn’t have enough time to finish the exporter and I ended up generating the important events with Audacity or something like that, or my parser was very slow and I didn’t want the demo to take ages to load.

I can’t find this code, but I remember it was very buggy and I wasn’t progressing very fast. I wrote about it back then! There was a lot of confusion and non-written knowledge around the tracking format that I finally understood asking other people for help (e.g. how to calculate how much time do each row and each ‘tick’ take). Also, lots of segmentation faults and time spent thinking about memory and string allocations! And I was using the wrong tool for the job.

GitHub and Google Code didn’t even exist back then and SourceForge was terrible, so this is probably collecting digital dust in some folder.

2007: Ruby: XM to events

Apparently I also wrote a Fast Tracker’s XM to timed event utility in Ruby because I was going to work with a musician that used XM (hi JosSs!) and I was so into Ruby as server side and general utility language back then—the demo was going to be built in Flash with probably ActionScript 2.

I actually didn’t finish the demo, therefore the exporter is quite incomplete and I haven’t bothered publishing it, but it was interesting to use Ruby to process binary data from a file in a sort of stream like way.

In absence of a demo, here’s my favourite song from JosSs:

2008: PHP: IT to events (in a .h file)

Another iteration of this event list idea was to parse the data with PHP and generate a song.h file containing a big data array to be used in 64k intros, like this one.

I can’t find the code, but if I remember correctly, the nicest aspect of building this was that I could use some sort of text template to generate the .h file, and since string manipulations are way easier with PHP than with C, I got to the point where I wanted to be really quickly and without tons of segfaults.

2009: PHP: XRNS to MOD

I found another twist on my data conversion efforts: xrns2mod takes a Renoise song and converts it into an Amiga MOD file.

This is not a trivial conversion, as Renoise files are XML based and the player is significantly evolved compared to the MOD format: it can have a lot of channels, and more than one track per channel, plus arbitrary length patterns, plus a lot of instruments, and more than one sample per instrument, etc, whereas the most basic MOD files only have 4 tracks, use a limited note range, different base frequencies, less sample effects, and lesser quality samples, etc. So it’s not just moving numbers from one place to another.

I think I wanted to enter some MOD contest but I found all the MOD trackers quite uncomfortable compared to Renoise, and I wanted to have the best of both worlds: ease of use and also entering the contest. It’s incomplete and quite hardcoded in places, but it was already converting files quite decently the last time I ran it, so I put it on GitHub, just in case it maybe helps someone. Maybe.

Of course, minutes after I published it I found XRNS2XMOD, a similar tool which seems more complete as it also exports to XM in addition of MOD, but it requires Mono and what not.

~2010: Python + Lua: XRNS to events (in a .lua file)

I wrote an exporter in Python for generating a list of timed XRNS events and use them in a demo for ~~incredible synchronisation~~. The demo was run with Luisita which was my programmable graphics environment built with Lua and C++, and since the exporter generated a .lua file, it was very easy to load it all into an array like object at runtime.

This was used in this demo — as you can see the visuals are tightly related to the music:

I later tried to port this demo to WebGL, but I never finished it… perhaps I should just share the code for whoever was interested. It was a bit challenging to migrate because I was using OpenGL immediate mode in the Luisita version, so I needed to change my mental paradigms to WebGL/three.js’s and also write a shader or two to make it look as I wanted it to look… also those were a ton of events that I wanted to send to JavaScript, so my naive initial attempts were making the Garbage Collector work a lot more than it should have worked.

Funnily enough I remember being frustrated with not having tweening in the Lua version, but it could be ‘easy’ to add it to the web version using tween.js. But it’s not going to happen.

2013: node.js: XRNS to JSON

I also wrote a node.js module for extracting XRNS data into something the browser could use more easily (i.e. JSON). I guess not many people are using it or are interested in it, because 3 years later I haven’t heard of anyone improving or using it, but if you want you can install it with npm:

npm install renoise

I used this to generate the events and note data for the slides for my JSConf.EU 2013 talk in which I programmed virtual instruments to play a tracked song in the browser, built their interfaces with Web Components and also used hardware controlled/read with Open Sound Control and node.js! Yay to mixing everything together! 😎

Here’s the video starting with the actual demo:

Well, I hope you have enjoyed this tour, and remember to always use the best tool for the job 😉

 

A first for everything: Greece, Denmark, Norway

Writing this with a very slow connection in Thessaloniki, Greece (ahhh, hotel internet).

I am very excited that my Greek studies are finally coming in handy and so I can read the street signs even if they’re written using the Greek alphabet. It sort of reminded me to last year when I was in Hong Kong and couldn’t read the signs… except now I can! Yay!

I arrived yesterday after a terribly early flight–but then it was well past noon because of the +2h timezone.

Kostas, one of the organisers of DEVit, helped me get a SIM card and he was OK with me telling the world that it took us an hour and a half to get it working with data, because all of the processes they require. I think we did three queues and had time to go for lunch and go back. We also had to enable the Greek keyboard on my phone in order to send a text to the operator containing the Greek ‘y’ (which is not the same as the English keyboard ‘y’), and then it would send me 5 texts in response, confirming whatever it is the ‘y’ meant (I mentally read it as “why?”) with a lot of ALL UPPERCASE Greek paragraphs.

I will just say that in London you can turn up at a shop and buy a SIM with pre-paid credit in 5 minutes, so if I didn’t come from Spain where these processes are similarly ridiculous, I would have been wildly amused.

I re-learnt the most important sentences:

  • παρακαλω = parakalo = please
  • ευχαριστωσ = eucharistos = thanks

and last but not least: χαχαχα = hahaha = jajaja 😃

This is also my first time in Greece so I don’t quite know what to expect, but so far it seems welcoming, nice and sunny, full of interesting things to look at, and also very busy. I did try a gyros and I’m looking forward to trying out the Greek coffee: “cooked!”. Will report.

Side note: for a Spanish speaker, hearing spoken Greek in the background is funny: your brain thinks it can understand it because the cadence is very similar to Spanish and many Spanish words are of Greek origin, but then it starts to pay attention and realises it actually can’t.

I’ll attend the NodeJS/Ruby synergy meetup this evening… be sure to say hi and bring all your synergies and thoughts out of the box!

Next week I’ll be in Denmark for AtTheFrontEnd and the following week will see me in Norway for WebRebels. I’ve not been in either country before, so those will be another two first times for me, but if they’re anything like Sweden or Finland I’ll be more than happy.

I’ll be talking about MediaRecorder and how you can now generate video in the browser, natively, without plug-ins. If you are interested in real time audio or video processing, my talk will be relevant to you 🙂