Hands On Web Audio

An interactive slide-deck showcasing the Web Audio API

Using WebGL + Web Audio. Takes a bit to load. Sorry about that.

hey there

Hands-On Web Audio

Soledad Penades

@supersole

Web Audio is like a tiny new language

small vocabulary

few grammar rules

and a couple of idioms

AudioContext

where we start

where everything happens

var ac = new AudioContext();

AudioContext

As a toolbox

"instance factory"

var oscillator = ac.createOscillator();
var gain = ac.createGain();
// etc...

Nodes taxonomy

AudioContext

As audio graph container

introducing the most important method in nodes

connect()

Modular routing

Simplest graph

var oscillator = ac.createOscillator();
oscillator.connect(ac.destination);

Simplest graph

Not so simple graph

var osc1 = ac.createOscillator();
osc1.connect(ac.destination);
var osc2 = ac.createOscillator();
osc2.connect(ac.destination);
var osc3 = ac.createOscillator();
osc3.connect(ac.destination);
// ...

Not so simple graph

More complex graph

var gain = ac.createGain();
var osc1 = ac.createOscillator();
var filter1 = ac.createBiquadFilter();
osc1.connect(filter1);
filter1.connect(gain);
var osc2 = ac.createOscillator();
var filter2 = ac.createBiquadFilter();
osc2.connect(filter2);
filter2.connect(gain);
// ...
gain.connect(ac.destination);

More complex graph

Sound modulation

or...

audio as input data

instead of connecting node to node,

connect() node to node properties

adds node output to node param

but sound sources only output [-1, 1]

a [-1, 1] fluctuation

is barely audible in most contexts!

solution:

Gain nodes

multiply input by gain value

var gain = ac.createGain();
var osc = ac.createOscillator();
var gain = ac.createGain();
osc.connect(gain);
gain.connect(ac.destination);
gain.gain.value = 100;
// osc out [-100, 100] now
window.addEventListener('mousemove', function(e) {
	var v = e.clientY / window.innerHeight;
    gain.gain.setValueAtTime(v, ac.currentTime);
});

gain value as f(mouseY)

var lfOsc = ac.createOscillator();
var lfGain = ac.createGain();
lfOsc.connect(lfGain);
lfGain.gain.setValueAtTime(10, now);
// lfGain outputs [-10, 10]
var osc = ac.createOscillator();
lfGain.connect(osc.frequency);
osc.frequency.setValueAtTime(100, now);
// osc oscillates at [90, 110]

Modulation

Corollary

You can build rich and complex graphs...

+ adding node outputs to param inputs

* multiplying outputs with Gain nodes

Performance

the Web Audio promise:

Low latency +

Sample accurate playback

but how?

Web Audio has its own thread

Web Audio 'just'

processes scheduled events

schedule events via node instances

events = (mostly) value changes

osc.start(2.0);
osc.frequency.setValueAtTime(440, 3.5);
osc.frequency.linearRampToValueAtTime(880, 13.5);
gain.gain.exponentialRampToValueAtTime(0, 13.5);

the graph is super optimised

creating and throwing nodes away is cheap

keeping track of state is expensive

don't keep references to nodes

(to avoid memory leaks)

stop and disconnect sound sources

(so they're disposed of)

What else can we do?

BufferSource

Playing sound samples

Loading samples

Three steps process

Load binary sample data

Decode

Use it!

1/Load binary data

var req = new XMLHttpRequest();
req.open('get', samplePath, true);
req.responseType = 'arraybuffer';
req.addEventListener('load', function() {
	var response = req.response;
});

2/Decode into a buffer

context.decodeAudioData(response,
function(decodedBuffer) {
    // ...
}, function(error) {
    // urgh
});

3/Use it!

var bufferSource = ac.createBufferSource();
bufferSource.buffer = decodedBuffer;
bufferSource.connect(ac);
bufferSource.start();

the Amen Loop!

bufferSource.playbackRate

BufferSource

can do pitch bending!

var now = ac.currentTime;
var y = getMouseY();
bufferSource.playbackRate.setValueAtTime(y, now);

BufferSource

Pitch bending!

StereoPanner

Place sounds in 2D space

StereoPanner

var osc = ac.createOscillator();
var stereoPanner = ac.createStereoPanner();
osc.connect(stereoPanner);
// left
stereoPanner.pan.setValueAtTime(-1, now);
// or right
stereoPanner.pan.setValueAtTime(1, now);

((((( O_O )))))

Panner

Place sounds in 3D space

Understanding panning

there is a listener (us)

ac.listener

and one or more positioned sources of sound

var panned = ac.createPanner();
osc.connect(panned);

update positions for accurate placement

ac.listener.setPosition(100, 0, 0);
panned.setPosition(0, 0, 0);

Panner

3D flock

BiquadFilter

Lets certain frequencies pass

BiquadFilter

var filter = ac.createBiquadFilter();
filter.type = 'lowpass';
filter.connect(ac.destination);
bufferSource.connect(filter);
bufferSource.start();

It's a party!

... but where are the visuals?

Analyser

real-time frequency and time-domain information

Analyser

var analyser = ac.createAnalyser();
bufferSource.connect(analyser);
analyser.connect(ac.destination);
var results = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(results);

Analyser

Getting creative

Real time input

+ processing

Wrap up

Web Audio: minimal and powerful

many built-in modular nodes,

one graph, simple connect rules

sound generation, processing and analysis

Web Audio

great for games, VR and interactive applications

it's also

FUN!

Now you know the basics,

go explore!

Play with the built-in features and

make it your own

Also,

come to the Web Audio meet-ups!

(London/Berlin/NY/Philadelphia)

github.com/sole/howa

Thanks!

@supersole