Real time
front-end alchemy

Soledad PenadΓ©s

soledadpenades.com / @supersole

Warning: moving images and maybe awful feedback in this presentation

If video loops or sound glitches make you feel unwell, please leave now. Sorry πŸ˜”

I work at

Mozilla

Engineering manager at @FirefoxDevTools

Real time
front-end alchemy

or: capturing, playing, altering and encoding video and audio streams, without servers or plugins!

Streams!?!

Streams!?!

Not:

  • Node.js streams
  • Future JavaScript streams

Yes:

  • MediaStream instances

MediaStreams are
track containers

Tracks can be audio or video

There can be any number of tracks per stream

You might have used MediaStreams already...

getUserMedia()

getUserMedia()

					
						navigator.mediaDevices.getUserMedia({
							 audio: true,
							 video: true
						}).then(function (stream) {
							// do something with the stream
						})
					
				
					
						navigator.mediaDevices.getUserMedia({
							audio: true,
							video: true
						}).then(function (stream) {
							var video = document.createElement('video');
							document.body.appendChild(video);
							video.src = URL.createObjectURL(stream);
							video.play();
						}
					
				

BORING

this is old news 😴

You didn't come here
to hear boring stuff
πŸ™…πŸ»

What's πŸ†• and πŸ†’?
MediaRecorder

Native JavaScript API

Familiar syntax with methods + DOM events

MediaRecorder in a nutshell

					
						var recorder = new MediaRecorder(stream);
						recorder.addEventListener('dataavailable', e => {
							video.src = URL.createObjectURL(e.data);
							video.play();
						});
						recorder.start();
						setTimeout(() => {
							recorder.stop();
						}, 5000);
					
				

I just encoded audio and video natively in the browser, without plug-ins.

πŸ’«πŸŽ™πŸ’₯ MIC DROP

jk 😜

And we've just barely scratched the surface...

Alternative stream sources, 1:

getUserMedia screen and window sharing

					
						// about:config -> add domain to media.getusermedia.screensharing.allowed_domains
						navigator.mediaDevices.getUserMedia({
							video: {
								mediaSource: 'screen'
							}
						}).then(function (stream) {
							// do something with stream
							// ...
						});
					
				

Alternative stream sources, 2:

canvas.captureStream()

					
						var canvasStream = canvas.captureStream(30 /* frames per second */ );
						var recorder = new MediaRecorder(canvasStream);
						// Simplified drawing white noise on the canvas
						var pixels = ctx.getImageData(0, 0, width, height); var data = pixels.data; 
						for(var i = 0; i < numPixels; i++) {
							var grey = Math.round(Math.random() * 255);
							data[offset++] = grey; data[offset++] = grey; data[offset++] = grey; offset++;
						}
						ctx.putImageData(pixels, 0, 0);
						// plus usual recorder.start(), stop()...
					
				

We can also transform webcam images with canvas pixel manipulation

					
					context.drawImage(video, 0, 0, width, height); // draw video into canvas
					
					var imageData = context.getImageData(0, 0, width, height);
					var data = imageData.data;
					for (var i = 0; i < data.length; i+=4) {
						data[i] = data[i] >= cutOff ? 255 : 0; // red
						data[i + 1] = data[i + 1] >= cutOff ? 255 : 0; // green
						data[i + 2] = data[i + 2] >= cutOff ? 255 : 0; // blue
					}
					
				

And we can record the output of this canvas, using captureStream

					
					// some steps from before, combined
					// 1. init webcam
					// 2. make video stream
					// 3. copy to canvas and alter pixels
					// 4. get stream from canvas
					// 5. record!
					
				

You should not use canvas 2D pixel manipulation for this because it can be really slow. Use WebGL!

(but WebGL takes longer to setup and learn and explain, and won't fit on slides, so... πŸ’πŸ»)

Alternative stream sources, 3:

Web Audio's MediaStreamAudioDestinationNode

(... and we made fun of Java class names πŸ˜‚)

This enables us to modify and capture audio in real time, using an AudioContext

AudioContext is your door to the power of Web Audio

It lets you create input and output nodes and connect them together to form an audio graph to route and manipulate sounds in various ways.

I cannot explain Web Audio to you in two minutes, so this will have to do!

This is what we're going to do:

input stream -> audio context -> output stream

But we'll start with:

input stream -> audio context

					
					// we obtained a stream with an audio track, with getUserMedia
					var ac = new AudioContext();
					var inputNode = audioContext.createMediaStreamSource(stream);
					var effect = makeEffect(audioContext);
					// [...]
					
					inputNode.connect(effect.input);
					effect.output.connect(ac.destination);
					
				

Let's add a MediaStreamAudioDestinationNode to record this:

input stream -> audio context -> output stream

					
						var outputNode = audioContext.createMediaStreamDestination();
						effect.output.connect(outputNode);

						var recorder = new MediaRecorder(outputNode.stream);
						// and record as usual...
						recorder.start();
					
				

So far, we have looked at manipulating and recording video and audio separately

When we process audio and video in parallel, we get two streams, but MediaRecorder only takes one...

How can we join both streams together and capture it? πŸ€”

What is
✨the ultimate secret✨
to real time front-end alchemy?

MediaStream()

MediaStream

With this, we can recombine any number of streams into just one and keep on recording!

(as MediaRecorder doesn't care where did the stream come from)

					
					// Video =========================
					var videoTracks = inputStream.getVideoTracks();
					var videoStream = new MediaStream();
					videoTracks.forEach(function(track) {
						videoStream.addTrack(track);
					});
					// [...] manipulate videoStream into a canvas
					// then get result from canvas stream into videoOutputStream
					var videoOutputStream = videoCanvas.captureStream();
					
				
					
					// Audio =========================
					var audioTracks = inputStream.getAudioTracks();
					var audioStream = new MediaStream();
					audioTracks.forEach(function(track) {
						audioStream.addTrack(track);
					});
					// [...] manipulate audio with an audio context
					// then get result from audio destination node into audioOutputStream
					var audioOutputStream = streamDestination.stream;
					
				
					
					// Merging into one stream only ==
					var outputStream = new MediaStream();
					[audioOutputStream, videoOutputStream].forEach(function(s) {
						s.getTracks().forEach(function(t) {
							outputStream.addTrack(t);
						});
					});
					
				
					
					// And recording! ================
					var finalRecorder = new MediaRecorder(outputStream);
					
				

Why is this API important?

MediaRecorder + Web Audio + WebGL enable you to build really sophisticated apps without plugins or server side support so it's faster and respectful of your users' privacy

Also

Photography and video apps are ~kind of~ a big deal 😌...

Play Store: 4th and 5th top apps

App Store: 6th and 8th top apps

My point being...

MediaRecorder enables you to build Instagram and SnapChat on the browser

Although maybe we don't need yet another social network?

Get creative instead!

For example...

Some more ideas

More examples?

Example: Boo

A videobooth built with WebGL and Web Audio

Boo videobooth

Example: Greenscreening Cookie Monster

Greenscreening cookie monster

Wrap up

The MediaCapture API is very cool!

Wrap up

Support is so so πŸ˜”:

Wrap up

But it is getting great really quickly

... and we need you to play with it, contribute and help us make it better!

More resources

I WILL TWEET THE LINK!

Thanks!

soledadpenades.com / @supersole