Warning: moving images and maybe awful feedback in this presentation
If video loops or sound glitches make you feel unwell, please leave now. Sorry π
I work at
Engineering manager at @FirefoxDevTools
Real time front-end alchemy
or: capturing, playing, altering and encoding video and audio streams, without servers or plugins!
Streams!?!
Streams!?!
Not:
Node.js streams
Future JavaScript streams
Yes:
MediaStream instances
MediaStreams are track containers
Tracks can be audio or video
There can be any number of tracks per stream
You might have used MediaStreams already...
getUserMedia()
getUserMedia()
navigator.mediaDevices.getUserMedia({
audio: true,
video: true
}).then(function (stream) {
// do something with the stream
})
navigator.mediaDevices.getUserMedia({
audio: true,
video: true
}).then(function (stream) {
var video = document.createElement('video');
document.body.appendChild(video);
video.src = URL.createObjectURL(stream);
video.play();
}
BORING
this is old news π΄
You didn't come here to hear boring stuff π π»
What's π and π? MediaRecorder
Native JavaScript API
Familiar syntax with methods + DOM events
MediaRecorder in a nutshell
Constructor: new MediaRecorder(stream)
Methods: start(), stop(), pause(), resume()...
Events: dataavailable, start, stop...
var recorder = new MediaRecorder(stream);
recorder.addEventListener('dataavailable', e => {
video.src = URL.createObjectURL(e.data);
video.play();
});
recorder.start();
setTimeout(() => {
recorder.stop();
}, 5000);
I just encoded audio and video natively in the browser, without plug-ins.
π«ππ₯ MIC DROP
jk π
And we've just barely scratched the surface...
Alternative stream sources, 1:
getUserMediascreen and window sharing
// about:config -> add domain to media.getusermedia.screensharing.allowed_domains
navigator.mediaDevices.getUserMedia({
video: {
mediaSource: 'screen'
}
}).then(function (stream) {
// do something with stream
// ...
});
Alternative stream sources, 2:
canvas.captureStream()
var canvasStream = canvas.captureStream(30 /* frames per second */ );
var recorder = new MediaRecorder(canvasStream);
// Simplified drawing white noise on the canvas
var pixels = ctx.getImageData(0, 0, width, height); var data = pixels.data;
for(var i = 0; i < numPixels; i++) {
var grey = Math.round(Math.random() * 255);
data[offset++] = grey; data[offset++] = grey; data[offset++] = grey; offset++;
}
ctx.putImageData(pixels, 0, 0);
// plus usual recorder.start(), stop()...
We can also transform webcam images with canvas pixel manipulation
context.drawImage(video, 0, 0, width, height); // draw video into canvas
var imageData = context.getImageData(0, 0, width, height);
var data = imageData.data;
for (var i = 0; i < data.length; i+=4) {
data[i] = data[i] >= cutOff ? 255 : 0; // red
data[i + 1] = data[i + 1] >= cutOff ? 255 : 0; // green
data[i + 2] = data[i + 2] >= cutOff ? 255 : 0; // blue
}
And we can record the output of this canvas, using captureStream
// some steps from before, combined
// 1. init webcam
// 2. make video stream
// 3. copy to canvas and alter pixels
// 4. get stream from canvas
// 5. record!
You should not use canvas 2D pixel manipulation for this because it can be really slow. Use WebGL!
(but WebGL takes longer to setup and learn and explain, and won't fit on slides, so... ππ»)
Alternative stream sources, 3:
Web Audio's MediaStreamAudioDestinationNode
(... and we made fun of Java class names π)
This enables us to modify and capture audio in real time, using an AudioContext
AudioContext is your door to the power of Web Audio
It lets you create input and output nodes and connect them together to form an audio graph to route and manipulate sounds in various ways.
I cannot explain Web Audio to you in two minutes, so this will have to do!
This is what we're going to do:
input stream -> audio context -> output stream
But we'll start with:
input stream -> audio context
// we obtained a stream with an audio track, with getUserMedia
var ac = new AudioContext();
var inputNode = audioContext.createMediaStreamSource(stream);
var effect = makeEffect(audioContext);
// [...]
inputNode.connect(effect.input);
effect.output.connect(ac.destination);
Let's add a MediaStreamAudioDestinationNode to record this:
input stream -> audio context -> output stream
var outputNode = audioContext.createMediaStreamDestination();
effect.output.connect(outputNode);
var recorder = new MediaRecorder(outputNode.stream);
// and record as usual...
recorder.start();
So far, we have looked at manipulating and recording video and audio separately
When we process audio and video in parallel, we get two streams, but MediaRecorder only takes one...
How can we join both streams together and capture it? π€
What is β¨the ultimate secretβ¨ to real time front-end alchemy?
MediaStream()
MediaStream
we can create additional instances, as we have access to the constructor: new MediaStream()
and each instance has very useful methods
getAudioTracks()
getVideoTracks()
addTrack()
etc...
With this, we can recombine any number of streams into just one and keep on recording!
(as MediaRecorder doesn't care where did the stream come from)
// Video =========================
var videoTracks = inputStream.getVideoTracks();
var videoStream = new MediaStream();
videoTracks.forEach(function(track) {
videoStream.addTrack(track);
});
// [...] manipulate videoStream into a canvas
// then get result from canvas stream into videoOutputStream
var videoOutputStream = videoCanvas.captureStream();
// Audio =========================
var audioTracks = inputStream.getAudioTracks();
var audioStream = new MediaStream();
audioTracks.forEach(function(track) {
audioStream.addTrack(track);
});
// [...] manipulate audio with an audio context
// then get result from audio destination node into audioOutputStream
var audioOutputStream = streamDestination.stream;
// Merging into one stream only ==
var outputStream = new MediaStream();
[audioOutputStream, videoOutputStream].forEach(function(s) {
s.getTracks().forEach(function(t) {
outputStream.addTrack(t);
});
});
// And recording! ================
var finalRecorder = new MediaRecorder(outputStream);
Why is this API important?
MediaRecorder + Web Audio + WebGL enable you to build really sophisticated apps without plugins or server side support so it's faster and respectful of your users' privacy
Also
Photography and video apps are ~kind of~ a big deal π...
Play Store: 4th and 5th top apps
App Store: 6th and 8th top apps
My point being...
MediaRecorder enables you to build Instagram and SnapChat on the browser
Although maybe we don't need yet another social network?
Get creative instead!
For example...
Some more ideas
Beatbox
Sampling based music apps... combine with Web MIDI!
Dictaphone
Field recording... with Geolocation API
Vine app, but without the 'app' bit
Granular and glitch-based audio synthesis
Your take?
More examples?
Example: Boo
A videobooth built with WebGL and Web Audio
Example: Greenscreening Cookie Monster
Wrap up
The MediaCapture API is very cool!
Wrap up
Support is so so π:
Firefox supports everything I showed you (obviously)
Chrome 57+ can do pretty much everything I demoed inc. rec streams from Web Audio (also: bugs)
Safari lolsob ππ₯
Internet Explorer: not in your dreams, but MS Edge seems open to develop this
Firefox/Chrome on Android: bit hit and miss, and no hardware accelerated encoding yet
I've no idea about Windows Phone or Blackberry
I don't think Opera Mini will support this
Definitely not in Lynx
Wrap up
But it is getting great really quickly
... and we need you to play with it, contribute and help us make it better!