Category Archives: Software

Article about the MediaRecorder API in .net magazine

MediaRecorder article picture

I wrote an article on the MediaRecorder API for .net magazine issue 283.

This isn’t an exhaustive article, just a little advance on what the API can do, as part of the Standards section that tracks new and upcoming APIs. If you want to learn more about the MediaRecorder API, there’s the way more detailed Hacks article I wrote: Record almost everything in the browser with MediaRecorder, or maybe you can watch one of the talks I’ve given on the subject, like this one.

dogetest.com

This summer, Sam and me are exploring Servo’s capabilities and building cool demos to showcase them (and most specially, WebRender, its engine “that aims to draw web content like a modern game engine”).

We could say there are two ways of experimenting. One in which you go and try out things as you think of them, and another one in which you look at what works first, and build a catalogue of resources that you can use to experiment with. An imperfect metaphor would be a comparison between going into an arts store and pick a tool… try something… then try something else with another tool as you see fit, etc (if the store would allow this), versus establishing what tools you can use before you go to the store to buy them to build your experiment.

WebRender is very good at CSS, but we didn’t know for sure what worked. We kept having “ideas” but stumbled upon walls of lack of implementation, or we would forget that X or Y didn’t work, and then we’d try to use that feature again and found that it was still not working.

To avoid that, we built two demo/tests that repeatedly render the same type of element and a different feature applied to each one: CSS transformations and CSS filters.

CSS transformations test

This way we can quickly determine what works, and we can also compare the behaviour between browsers, which is really neat when things look “weird”. And each time we want to build a new demo, we can look at the demos and be “ah, this didn’t work, but maybe I could use that other thing”.

Our tests use two types of element for now: a DIV with the unofficial but de facto Servo logo (a doge inside a cog wheel), and an IFRAME with an image as well. We chose those elements for two reasons: because the DIV is a sort of minimum building block, and because IFRAMEs tend to be a little “difficult”, with them being their own document and etc… they raise the rendering bar, so to speak 😉

All together, the tests looked so poignantly funny to me, I couldn’t but share them with the rest of the world:

and Eddy Bruel said something that inspired us to build another test… how many doges can your computer render before it slows down?

I love challenges, so it didn’t take me much to build the dogemania test.

It can ridiculise my MacBook retina very quickly, while people using desktop computers were like “so what’s the fuss, I get 9000 doges with no sweat?”

It was also very funny when people sent me their screenshots of trying the test on their office screen systems:

Servo engineers were quite amused and excited! That was cool! We also seemed to have surfaced new bugs which is always exciting. And look at the title of the bug: Doges disappear unless they are large and perfectly upright. Beautiful!

To add to the excitement, this morning I found this message on my idling IRC window:

7:22 PM <jack> http://dogetest.com
7:22 PM <jack> we all love it so much i thought it needed it's own domain :)

So you can now send everyone to dogetest.com 🎉

Thanks, Jack!

Web Animations: why and when to use them, and some demos we wrote

There’s been much talk of Web Animations for a long time but it wasn’t quite clear what that meant exactly, or even when could we get that new thing working in our browsers, or whether we should actually care about it.

A few weeks ago, Belén, Sam and me met to brainstorm and build some demos to accompany the Firefox 48 release with early support for Web Animations. We had been asked with very little time, but we reckoned we could build at least a handful of them.

We tried to look at this from a different, pragmatic perspective.

Most graphic API demos tend to be very technical and abstract, which tends to excite graphics people but mean nothing to web developers. The problem web developers face is not making circles bounce randomly on the screen, it is to display and format text on screen, and they have to write CSS and work with designers who want to play a bit with the CSS code to give it as much ‘swooosh’ as they envision. Any help we can provide to these developers so they can evaluate new APIs quickly and decide if it’s going to make their work easier is going to be highly appreciated.

So our demos focused less on moving DIVs around and more on delivering value, either to developers by offering side by side comparisons (JS vs CSS code and JS vs CSS performance) or to end users with an example of an style guide where designers can experiment with the animation values in situ, and another example where the user can adjust the amount of animation to their liking—we see this as a way of using the API for improving accessibility.

You can also download all the demos from the repo:

git clone https://github.com/mozdevs/Animation-examples.git

And should you use CSS or should you use JS? The answer is: it depends.

Let’s look at the same animation defined with CSS and JS, taken straight from the CSS vs JS demo:

CSS:

.rainbow {
    animation: rainbow 2s alternate infinite;
}

@keyframes rainbow {
    0% { background: #ff004d; }
    20% { background: #ff77ab; }
    50% { background: #00e756; }
    80% { background: #29adff; }
    100% { background: #ff77ab;}
}

JavaScript:

let el = document.querySelector('.rainbow');
el.animate([
    { background: '#ff004d', offset: 0 },
    { background: '#ff77ab', offset: 0.20 },
    { background: '#00e756', offset: 0.5 },
    { background: '#29adff', offset: 0.80 },
    { background: '#ff77ab', offset: 1 }
], {
    duration: 2000,
    direction: 'alternate',
    iterations: Infinity
});

As you can see, this is fairly similar.

CSS’s syntax for animations has been with us for a while. We know how it works, there is good support across browsers and there is tooling to generate keyframes. But once the keyframes are defined, there’s no going back. Modifying them on the fly is difficult unless you start messing with strings to build styles.

This is where the JS API proves its worth. It can tap into live animations, and it can create new animations. And you can create animations that are very tedious to define in CSS, even using preprocessors, or just plain impossible to predict: imagine that you want custom colour animations starting from an unknown colour A and going to another unknown colour B. Trying to pregenerate all these unknown transitions from the beginning would result in an enormous CSS file which is hard to maintain, and would not even be as accurate as a newly created animation could be.

There’s another advantage of using the Web Animations API versus animating with a library such as tween.js—the API is hardware accelerated and also runs on the compositor thread, which means incredibly smooth performance and pretty much perfect timing. When you animate DOM elements using JavaScript you have to create lots of strings to set the style property of these DOM elements. This will eventually kick a pretty massive Garbage Collector, which will make your animations pause for a bit: the dreaded jank!

Finally another cool feature of these native animations is that you can inspect them using the browser developer tools. And you can scrub back and forth, change their speed to slow them down or make them faster, etc. You cannot tap into JS-library-defined animations (e.g. tween.js) unless there’s support from the library such an inspector, etc.

Essentially: Web Animations cheat by running at browser level. This is the best option if you want to animate DOM elements. For interpolating between plain values (e.g. a three.js object position vector x, y, z properties) you should use tween.js or similar.

If you want to know more here’s an article at Hacks talking about compositors, layers and other internals, and inspecting with DevTools. And with plenty of foxes!

The full API isn’t implemented in any browser yet, but both Firefox and Chrome stable have initial support already. There’s no excuse: start animating things already! I’m also excited that both browsers shipped this early support more or less simultaneously—makes things so much easier for developers.

With mega thanks to Belén for pushing this forward and to Sam for going out of his internship way to help us with this 🙌🏼

Volumio: a Raspberry Pi jukebox

Volumio playing aerodynamisk neger

Last Sunday, I spent some time tinkering with my Raspberry Pi 3, trying to get some sort of jukebox software up and running.

I finally settled on something called Volumio. It is a complete distribution that you install on the Pi and then it becomes a machine that plays many audio formats and that you can control from other devices, using a web interface.

And thanks to the Raspberry Pi 3’s WiFi, I just need to connect the audio jack and the power cable, and then the little machine can hide under the table with the rest of cables and other devices I don’t want to look at generally (NAS, modem, etc).

Volumio is installed by flashing a complete distribution onto a mini SD card. I had a small 4GB one roaming around so I used that. It’s cool because I can just swap SD cards and boom! I have a completely new environment without destroying the previous one.

The “official stable” builds that you’re directed to from their website won’t work with the Pi 3. They get stuck at the colourful loading pattern. I found someone had had the same problem, and the solution was to flash another image for Volumio 2 RC2. I flashed this new version, which is very hard to locate unless you know that such a thing exists or read the forum, and the initial boot didn’t show up the device on Bonjour, but after a reboot I could see it and start using it.

I configured it using the web interface, accessible at volumio.local. I told it to use WiFi (it worked!) and also to use my NAS music library (which also worked!). Then I shut it down again, moved everything out of my desk, and turned it on again to enjoy some music.

Apparently the initial Volumio front-end was built with PHP. Now they are rebuilding everything from scratch as Volumio 2, using Node.js, Socket.IO and Angular. I was excited about the node.js bit! But although there’s plenty of documentation around, I felt like I couldn’t understand where things were in the source, in the distribution, how to get a developer environment running, etc. Maybe it’s because I came back from holidays on Saturday and I’m still pretty jetlagged and my brain is not very functional yet, but maybe it’s also for the best: I surely don’t need to contribute to yet another project.

Anyway, I have a working jukebox. It can even play modules! It’s a bit basic at points (e.g. you cannot navigate by ‘genre’ or other MP3 metatags) but it is getting the job done so I’m happy.

Why is Instagram not a website (yet)?

Rem was wondering about this yesterday in Twitter:

A couple years ago I had to do some analysis of popular apps (basically peeking inside their guts, so to speak). What we found is that Instagram used a lot of native libraries for dealing with video and audio.

Rem suggests that Instagram could be almost 100% a website nowadays, and not the strange hybrid it is. I would love to see that, but the web platform needs a lot more features we don’t have yet, for example:

  • reliably being able to select which camera to use, and also being able to focus
  • encoding into a certain type of video format, in a performant non-draining-your-battery way
  • manipulating videos in real time, sounds included
  • accessing the ‘photoroll’ easily

You can access a camera stream with getUserMedia, but for privacy reasons you cannot see details about the devices so your code cannot know if they’re back or front cameras, and so the UI cannot respond appropriately. When running in phones, which use better camera hardware than webcams, we cannot access the camera settings from JavaScript (we can, with Firefox OS, but the APIs haven’t been standardised as no other vendor was interested in implementing them).

As usual, video is a huge issue in the web. Encoding is complex and expensive in battery terms, but thanks to MediaRecorder it is becoming more accessible. Still the formats MediaRecorder outputs are only viewable in browsers (vp8/webm); if you encode a video using the browser you have to convert it locally to something else like mp4 if you want to upload it to sites like Twitter, and unless you’re a tech-savvy person and have installed VLC or similar you cannot even watch the video you generated in your desktop if using the default video viewers in your system.

Similarly, handling video and audio post processing in real time is possible with WebGL + Web Audio but recording is not supported in anything other than Firefox. Chromieum/Opera won’t give you a stream from canvas or record a stream from a Web Audio context either.

And being able to access the photoroll in mobile or a directory in desktop would require that the APIs for exposing directory listings are available so the app could do things such as building thumbnails of the images, but they aren’t.

It’s not impossible—it’s “just” a matter of implementation and adoption.

If apps such as Instagram were 100% web based, and had a reliable chance of working reasonably well on every device, there would be no distinctions between device operating systems, and we could just focus on the device features (price, camera quality, battery life, storage, design…). This would be huge for the users and quite probably for the app makers which would need only one team to build the app, not one per operating system.

But would it be huge for Google and Apple?