tween.audio

tween.audio

Got this funny idea to mix two things I like together: Tweens and audio! Thus--tween.audio!

Or: how to abuse two things I like: tween.js and the <audio> tag. I'll explain:

Abusing tween.js

Normally tween.js is used for tweening graphic properties. Vector positions, colors, values. I'm surprised no one thought of changing audio values with tween.js yet!

But you can't use tween.js the normal way via TWEEN.update() with audio stuff because even an slight hiccup on audio output will have lethal consequences. OK, maybe not that serious, but you'll notice easily. Whereas when frame rate goes slightly down it's more or less passable. Unless you're a purist, of course, but most normal people don't notice. Heck, they don't even notice pixelated pictures!

So we need to ensure that the interpolated sound values are provided to the browser with the highest timing accuracy, which automatically discards waiting for requestAnimationFrame to be called.

The solution is to skip TWEEN.update and TWEEN.Tween itself, and just use the equations for modulating the pitch in order to visualise how the equations sound.

Abusing the <audio> tag

Once that was established I thought about what would be the best solution to actually make things sound. I could try to use some of the Buffer objects in the Web Audio API, but that would exclude Firefox. And I actually didn't need any reverb effect in this case. So? Ah, if I could use the <audio> tag? But I needed to specify a pre-rendered source, which would get me in trouble as soon as I started having to provide sources for mp3, ogg, or maybe if I wanted to change something (e.g. the volume or the length). Wouldn't it be great to dynamically generate the source?

Maybe I could just use some sort of base64 trickery in the src attribute, just as it's done with images...? Well, to cut a long story short: yes, that was the solution!

I had bookmarked several projects which used HTML5's audio capabilities, I looked into their sources and I found a function that could help me, in the source for the jsmod player from possan.

With a couple of modifications (mostly made it parameterisable) it was fit for my test:

function getWaveData(data, sampleRate) {
    var n = data.length;
    var integer = 0, i = 0;
    var header = "RIFF<##>WAVEfmt \x10\x00\x00\x00\x01\x00\x01\x00<##><##>\x01\x00\x08\x00data<##>";
    function insertLong(value) {
        var bytes = "";
        for (i = 0; i < 4; i++) {
            bytes += String.fromCharCode(value % 256);
            value = value >> 8;
        }
        header = header.replace('<##>', bytes);
    }
    insertLong(36 + n);
    insertLong(sampleRate);
    insertLong(sampleRate);
    insertLong(n);
    for ( var i = 0; i < n; ++i) {
        header += String.fromCharCode(Math.round(Math.min(1, Math.max(-1, data[i])) * 126 + 128));
    }
    return 'data:audio/wav;base64,' + btoa(header);
}

Would I get any audio output? I built an array with 88200 Math.random values (= 44100 sampling rate * 2 seconds), got a WAV with the function and set it as source of a dynamically created <audio> tag. And it worked! \o/ Even more: it worked on Firefox too! And Safari, of all browsers*!

So the plan was to make the script build the src attributes for each audio corresponding to graphs, and that was it! But as doing that requires quite a bit of calculations, the browser didn't show anything more than a blank page for the first seconds. Not nice!

I thought about changing the calculations and run them in Web Workers land, but it was a bit overkill for this case. At the end I just spaced them out with some liberal use of setTimeout so they would allow the browser to render something in between.

TCHK!

Now I needed to solve another problem: the "wave is not centered at 0 at the end" problem. That produced a very audible TCHK! when a sample finished playing. Quite annoying! I consulted my favourite pro-producer and demoscene mate echolevel and he suggested what I had thought of already (but dismissed as it seemed too simple): just fade it out. Which I did!

Now go and play some tweens. Which one is your favourite? I'm undecided between Elastic.InOut and Bounce.Out. Both sound so 50s sci-fi movies!


<small>* Sadly it didn't work on any of the mobile devices I tried--booh</small>