Notes on Web Audio Hackday Singapore 2016

The last installment of notes from my DevFest Asia 2016 trip! Here’s the official day website.

We did three brief presentations:

  • someone (I forgot his name, sorry 😳) introduced BandLab, which is an online service that lets you compose music and also collaborate on it, all together (a bit like “git but for music” except they didn’t use “fork” because it can sound “funny”). Very interesting project, it was enlightening to hear the features that are available depending on specific platforms and limitations, and also how they do use real instruments in their user testing because this is more of a tool to help musicians, not a tool to replace them
  • Matt McKegg introduced the Web Audio node school workshop, for anyone who wants to learn Web Audio but doesn’t know how / where to start
  • And I introduced the MediaRecorder API (here’s an article I wrote), and an experiment I had built for this hackday

This hackday was small, and I didn’t get to stay until the end to see the projects that people worked on, as I had to catch my flight to Australia, but it was good to catch up with Matt, Joshua and Chinmay and learn what they have been up to recently.

For example, Matt has been busy building Ferment, a sort of decentralised peer to peer website/app/service/whatever to share music from creator to creator. It’s built with Electron, Scuttlebot and also WebTorrent—I find it super interesting! I have watched a few talks on all these concepts before but I hadn’t really seen any application that “clicked” everything in place like this one. It’s super exciting to consider that each time you listen to a song you’re becoming another peer and can share it with more people, making the song way more accessible to more listeners. I am already using it, what about you?

Joshua has a folder full of Work-In-Progress projects so there’s no point in me listing all of them—but he’s recently been working on controlling the speed of videos using a bezier curve editor. He was super cool and tried to combine that with MediaRecorder but we found a bug, ooops! Here’s the bug report 😎

I also had the chance to talk to more local developers, and again I was terrible at catching names, but it was fun to see what other people are building with or using Web Audio for. Someone had built a sound board of silly sounds when she was bored with something else, someone else was using Pedalboard.js to experiment with applying live sound effects to the output of his electric ukulele without carrying a whole bunch of hardware with him.

Notes on Live.JS at JSConf.Asia 2016

I had sketched some ideas for the Web Audio Hackday before flying to Singapore, but I really had to finish them before the actual day came. So I spent Friday doing an intense labour of testing, testing and more testing with the latest versions of the browsers that implement MediaRecorder. Since I was on the business of talking and promoting this API, I was determined to get more people to try it out, but that meant that I needed to be prepared for the “surprises” that working with new and upcoming APIs entail. Namely, that what worked yesterday might not work anymore, and what didn’t might have started to work without prior warning.

That said, my infinitely curious self would not resist flying to such a beautiful city as Singapore and not exploring a little bit of it: I squeezed some time out of the morning to visit the Marina Bay gardens, which again I found baffling (botanical gardens which are cooled down instead of heated!), then did a lot of work, and then later in the afternoon I went to the CSS Conf/JS Conf venue, as Live.JS was going to happen.

And “What is Live.JS?”, you might be wondering. Basically it’s a collective that is dedicated to making audio visual shows using JavaScript. JavaScript!!! It’s a bunch of people, but that doesn’t mean that you will always get the same people in each “show”. In this case, it was

  • Matt McKegg, which would play music using his own Loop Drop instrument for live performances. He built it using Google Web Apps initially, then migrated to Electron when GWA was starting to stagnate. He uses two Novation launchpad controllers, and the sound generation itself is via Web Audio. And it is really impressive, if you ask me!
  • Ruth John, who was VJing. Not sure what the name of her software is, but she makes heavy use of CSS variables (to animate stuff), SVG to draw elements on screen, and of course Web Audio’s Analyser nodes to modify the values of variables, in response to the music being played. Also, Web MIDI so she can use external MIDI controllers to do things such as change layer opacities and switching between effects at the turn of a knob, etc.
  • Martin Schuhfuss, who was controlling the lights in the venue using the DMX512 protocol (which happens to be a close MIDI relative) and a Monumental Hack with CSS property interpolation to run light animations: colour, rotation, focus, etc. This was quite impressive to hear!

I personally I’m a big fan of Matt’s music since I saw him live in CampJS, so I was excited about this event. And of course, I was quite intrigued about the visual side of things. Hearing about the techniques underneath made me giggle and be in awe at the same time: “can’t this be done in a better, proper way, or are these hacks just used for the sheer pleasure of subverting the original purpose of the technology?”, I wondered.

In the meantime, “special cocktails” were served—although they were actually called “codetails” in the leaflet:

  • ES4: Old Fashioned
  • jQuery: Rum Punch
  • Outsourced: Curry and tonic
  • Unicode: Unicorn tears (non-alcoholic)

I had “a jQuery” as it seemed the sweetest of the punch, and chatted around with a bunch of people, then went and admired Matt’s total LoopDrop skills, and wondered again if I’d ever understand how it works for reals.

Then I focused my attention on what the lights were doing: many things. I guess my issue was exactly that: they were doing many things, and I was kind of expecting some sort of carefully built progression which would not happen, because the LiveJS people had just met a day or two ago, and Martin had only had access to the light equipment the same day, because obviously you don’t travel from Germany with a rack of professional light equipment. So there’s only so much you can prepare, and given that Matt’s music is so improvisational, I should probably not expect a heavily synchronised audiovisual show. Still, I couldn’t but admire two facts:

  • a browser was interpolating between CSS values and ultimately this was generating MIDI commands that caused spotlights to move and change colours.
  • that the DMX512 standard did indeed work well enough that he could just show up and control DMX512 lights with his software, without further changes

That was super cool!

People often fixate in the business value of JavaScript, but it’s important to also consider JavaScript as a creative medium which has lots of potential expressivity and also reach by virtue of being online. I’m glad that Live.JS exists to inspire and support this, and I’d love to see even more exploration in this field. Bring it on! 🤘🏼

Here’s the video of this event:

Notes on CSSConf Asia 2016

Previous notes on this conference were decidedly minimalist due to my massive jetlag and lack of rest. Let’s fix it now that I’m back to London–and also the videos are out!

I arrived to Singapore the day before the conference started, i.e. I was to be on a stage less than 24 hours after setting my feet on Singaporean land. This, you imagine well, is nothing less than a recipe for disaster. I slept little on the plane, and when I arrived to the hotel I just had a shower (ah that first shower after a long haul flight…!), went to the mall downstairs to get some food and came back to keep obsessing over my slides until I couldn’t resist it anymore, and went to finally sleep in a proper bed, although not enough, but I would not have been able to sleep more because I was agitated about the talk anyway.

So I was not in my best shape. I was cold, tired, unable to really articulate complicated thoughts. The only thing that was getting me going was adrenaline at that point, and copious amounts of it, judging by the crash afterwards. But let’s not anticipate events…

The venue

It was very nice and warm outdoors, specially coming from London where it was dark and chilly to the bone and we were wearing not only scarves but also gloves! and even hats! One of my favourite things about hot weather is that it’s the perfect excuse to have iced lattes. And so cold drink in hand I went to the theatre where the conference was held.

The entrance hall was fine, but indoors… indoors was a totally different experience. You needed one or two additional layers of clothes to be comfortable—more so if you’re not feeling great, as I was. Still, I sat and listened to the speakers before me.

Much as I love JS, I tend to have a sweet spot for CSS conf talks. They gravitate more towards design, UX and (would you believe it) CSS, which are topics which don’t usually attract the same level of attention, but affect everyone the same or more. So I was genuinely excited for the talks!

The talks

Rachel Andrews: Laying Out The Future With Grid And Flexbox

I had seen an earlier version of this talk last year at Fronteers in Amsterdam. Back then I sort of missed more examples, but this time there were plenty of them, and it was a very informative and useful talk. Plus many of the techniques will be available in stable browsers some time this year—do watch it and start learning so you can start using Grid this year! (you can already use flexbox today).

Kenneth Ormandy: Syntax Highlight Everything

When Kenneth started speaking, super fast and super charged, my first thought was “oh wow I don’t think I can handle this in my current state”… but I pushed myself to listen and the more I listened the more enthralled I became. There was so much good material, although the title seemed a bit contrived and slightly disconnected from the actual talk content: started talking about syntax highlighting but ended up going through new CSS font features…? But maybe if I rewatch it again I’ll think otherwise:

Chris Lienert: Access All Areas

A poignant talk on accessibility and best practices. Definitely got me thinking.

Sarah Drasner: Functional Animation

Sadly I didn’t finish watching this one, but what I saw was really good. Lots of info about do’s and don’ts about animation on the web, user flow, UX… all very good stuff! Happy to finally see the whole talk now!

And my talk: make websites, not apps

Normally, I find it this “hot outdoors-freezing indoors” concept baffling. But in this context it was debilitating.

Half way during my talk I noticed my hands were shaking because of the cold. The only other time I’ve felt like that was during my talk at CampJS 2015, but we were outdoors and it was almost winter, and at least I had a hot cup of coffee in my hand back then! Here I was just trying to squeeze some sense out of my tired, jetlagged brain while working really hard to vocalise and not shiver uncontrollably. Ahhh!

Still, I think it wasn’t too bad: I got the message across. Many people reached out to me afterwards, saying they felt very empowered by my advice and excited to keep practising their craft –CSS– and becoming even more proficient at it, without feeling demeaned or less of a developer because of they not using or developing JavaScript, or not developing apps.

Others were also excited by the prospect of rendering engines such as Servo‘s WebRender making their life easier so they can write readable CSS without worrying about implementation details or using obscure hacks.

Even more, someone got inspired by the key concept in my talk (building along the Web’s direction) and gave a lightning talk:

Thanks, Souvik! (slides are here, but unfortunately I don’t think the lightning talks were recorded).

So I will call it a success. I am not done with “banging this drum” yet, and a proper write up is coming, but you can check out my talk in the meantime:

(gosh, I sound so tired)

In which Sole eventually becomes really debilitated and leaves the venue to rest

My talk was the last one before lunch. Joshua Koo (mighty three.js contributor which wrote the text class amongst other cool particle stuff) suddenly became our restaurant guide, as a local. So we went to this indian vegetarian place, and it was lovely.

The issue is that we sat in a long table, and more and more people kept adding themselves to the group. What was initially 4 people grew to 6, 8, 10… I lost track. And people started to wait for each other’s food before starting to eat, which is fine in a table of four, but not when you have lost track of how many people are in the table. Cold bread is bad bread.

I was so hungry. I think I had had a muffin or something ridiculous like that, very early in the morning, and didn’t snack on anything on the first break. So I was really suffering as my food was in front of me and I was not able to eat it due to following the ‘social construct’, and meanwhile it was getting cold, and because I hadn’t slept or eaten much, I was also getting colder and colder. Eventually I gave up and started grabbing bits of the bread, and someone else kind of started doing the same, and the people whose food hadn’t come yet gave us the ‘approval’ to eat, so we could stop pretending to just be content with some morsels, and eat for reals.

We talked about various experiments we were working on. Joshua has always so many ideas going on! Sounds familiar? I showed him my early 3d presentation framework prototype, which heavily relies on his text to 3D code!

Unfortunately after about 15 minutes my adrenaline seemed to be gone. My brain refused to listen much more—I guess it preferred to focus on the food I had just given it. I started to zone out, and become more of a listener than an active person at the table. Soon after we went back to the conference venue for the next talks. Sarah was speaking, and it sounded so great and informative, but 2 minutes after I sat down I started to not only feel really cold beyond anything I had felt before, but also see things flying past my eyes. At which time I decided it was stupid to keep torturing myself, and I went back to the hotel for “a 15 minutes nap”.

It was way more than that.

I woke up with the usual disturbed sleep weirdness: dry mouth, a terrible headache, hungry again, and specially: not cold, and not seeing flying things past my eyes! Wonderful!

The conference was almost over—and with that and my massive headache, there was no point in going back. And I had already done my talk! It was OK. I walked around the neighbourhood. I found a strange shopping mall full of used text book shops. Shop after shop of text books, and other books, and office supplies and office furniture. Also a massive art and crafts store. A most peculiar shopping mall. Colonial houses mingling with practical high density colourful blocks. Clean streets. I walked through all that in a sort of dream state—am I really here? How is this so warm and nice, it was very cold yesterday? Or was it the day before yesterday already?

I had dinner, and went to bed again, happy to have met so many nice people and have inspired at least a handful of them. The community was very open and respectful, and it was really wonderful to talk to them, hear what they are working on, what they’re concerned about and learn about how the tech scene is right now in Singapore. I am still genuinely sorry that I was in such a terrible state and couldn’t make the most out of the conversation 😭

Thanks to Thomas, Aysha and Zell for having me! It was a pleasure 😃

CSS/JSConf.AU:after

I was going to write this yesterday after dinner, but I happened to find Nadieh (or more appropriately, she found me!) and we had dinner and interesting conversations until late. So the only thing I had time for was packing my bag in preparation for my long trip home.

It was exciting to be back in Melbourne when the weather is warmer. Last year it was so cold just as London was getting warm! The coffees were good, and same for the food.

The audience at CSS/JSConf was really nice to interact with and I think they enjoyed my talk, which was recorded, but I am not sure when it’ll be published, and I want to check it out. Apparently the stenographer was having a field day thanks to my audio glitching experiments:

so I’m curious to relive that.

It was a bit accidented–my browser hung half way when encoding video, which it hadn’t in MONTHS, then the occasional bug that only happened one every twenty times? it also manifested itself. At least the projector played nicely with my laptop. It could have been worse, I say to myself.

Next day I indulged myself and stayed in bed for waayyy longer than was immediately necessary as I am going to spend so many hours in planes and airports this week-end… first an eight hours flight to Hong Kong, then a lay-over, and back to London in a twelve hours flight. It’s upcoming realities like these that make you truly appreciate the beauty of a simple bed on a quiet environment. Where quiet means literally quiet, not artificially induced quietness (via ear plugs etc).

I’ve been away from home for the last 12 days but it feels way longer due to travelling east, and also further down south. Everything is pretty much shifted for me right now. It’s “summer” here where it’s “winter” in London, and I’m wishing good morning to my people as I get ready to have dinner. It’s all a bit confusing.

I’m looking forward to being back and not having to do things such as putting a ‘please do not disturb’ sign on my door every day, or having to take food decisions every day (versus just buying familiar stuff and cooking it with my familiar kitchen accessories in my kitchen), or taking pills to go to sleep, every day (because I am still not used to the +11 hours shift).

I am glad I brought my own pillow with me this time. I used to think that was such a silly thing to do: I was wrong. There’s nothing like your own pillow to sleep on. Except for one night, I have slept really well during this trip, albeit not enough, as there was always something to prepare, somewhere to be, someone to meet.

And as much as I enjoy exploring new places and trying out new stuff, I am yearning to go back to normality. I found myself daydreaming with tea and toast and listening to BBC radio on a Sunday morning. If that’s not homesickness, I don’t know what that is!

One demo: two new bugs!

I thought that since I was going to the Web Audio Hack Day here in Singapore, it would be great to have a more Web Audio focused Media Recorder demo, so I built a little demo that essentially asks for audio permission, then records a short clip of audio, decodes it as an audio buffer and uses it to loop a BufferSourceNode in Web Audio.

You wouldn’t believe what happened next… or would you?

Two Things That Happened When Sole Ran This Demo

1. Chrome Canary Just Totally Gave Up

Aw, snap!

I filed a bug, since there wasn’t really much I could do…

2. Nightly Sometimes Insisted In Returning Some Mysterious 596 Samples Length Which Was Totally Not What I Expected, And Did So Without Following Any Meaningful Reproducible Pattern

596 samples for some reason

I tried to dump the contents of the returned blob once read as a buffer, using String.fromCharCode–perhaps looking at binary data while having my breakfast coffee would enlighten me?

I eat hex dumps for breakfast

As it turns out, no. So I filed another bug.

Try this yourself maybe?

instalooper

The demo is here: instalooper (sources) … just in case you want to play with the code–maybe it will be fixed next month, who knows!!? Or maybe it is just a bug in my machine and it works in everyone else’s!

Disclaimer: I have not tested this anywhere else than Nightly and Canary. So I’ve no idea of how it works (or not) in mobile.

I have a more complicated version that not only loops the sample but also randomly changes its playback frequency. Sometimes the generated sounds are strangely fascinating. It still needs a bit of interface work, but hey… watch out, Steve Reich! 😎

And now… time to prepare to go to the airport and fly to Melbourne for JSConf.AU! More jetlag, YAY!