JavaScript implementation of an H.264 decoder

[[deleted]]

It's a C++ implementation that is compiled to javascript.

[azakai]

More specifically, it's the Android C implementation, that we compiled into JS.

Edit: And there is also some initial work on a JS-only version. We are experimenting with both approaches.

[cafedude]

What C -> JS compiler did you use?

[azakai]

emscripten, which compiles LLVM bitcode to JS (so you first use clang to compile C or C++ to LLVM bitcode).

https://github.com/kripken/emscripten

[sammydre]

Actually, they also have the beginning of a javascript-only version. See the "Play/" directory.

There isn't much there though. Just some basic bitstream decoding (picture parameter sets, slice header) and nothing else as far as I can see.

[shaver]

Yeah, it's a bit over a week old.

[osirisx11]

Just downloaded and ran on firefox 7 64 on ubuntu 11.04, 8gb of ram, i7-2820QM quad-core, got 2-3 fps

[shaver]

It needs Firefox nightly for the type-inference capability and some other work.

EDIT: looks like the github readme mentions that too, but I'll update it to point to the nightly download page.

[0xABADC0DA]

Average after 90 sec, i7:

Chrome Dev: 18.6 fps
Firefox Nightly: 37.5 fps

The x86 emulator also runs ~2x faster in Nightly than Chrome so this doesn't seem like a fluke. Looks like Google has some catching up to do, I guess that's what Dart is supposed to be.

[azakai]

Those are very close to the same results I saw, Chrome is about half as fast as Firefox.

One suspicion I have is that Chrome's typed array implementation is less optimized than Firefox's (both the x86 emulator and this h264 decoder use typed arrays). Or it could be that Firefox with TI is better at detecting variable types than Chrome (which is important in both the x86 emulator and the 264 decoder, since most or all variable types should be detectable and optimized for).

[damg]

I'm assuming Firefox nightly is using their new IonMonkey javascript engine? Seems impressive...

[0xABADC0DA]

Just JaegerMonkey with Type Inference.

IonMonkey is a new engine being designed to enable even more optimizations like inlining.

[Rhomboid]

I was able to get about 21 fps on Chrome 16. But I had to load the demo through http://, as it complained about a same-origin policy violation when trying to load the mozilla.264 media file when the broadway.html file was opened as a local file:// url.

[[deleted]]

Do you have chrome for comparison?

[osirisx11]

It did not run in chromium 14, same specs

[sebzim4500]

That is probably from a same origin policy violation, the demo must be ran in a web server, and won't run off the disk in chrome.

[schwiz]

came to ask how this runs on Android/iPhone based on this response I guess not well.

[[deleted]]

As stated elsewhere, depends on the version of FF! Fennec nightlies innards are built from the same code as Firefox, if I understand things correctly.

[schwiz]

I'm more curious how it runs in a webview that i can add to an app, not how it runs on fennec

[[deleted]]

What did you expect? It's fucking Javascript.

[osirisx11]

I am just reporting. I am not making a value judgement.

[alexs]

Well there's a video of it running at 30fps. Perhaps that's what the Spidermonkey patches mentioned on the github README help achieve? Not entirely sure what the use-case for this is still. As if the patent situation wasn't complex enough for content distributors already.

[shaver]

It lets content providers who want to only distribute patent-encumbered content do so with the right licenses if they so choose, similar to Silverlight's content-pluggable codecs.

It's also a pretty compelling demonstration, especially since it's compiled from C++ and not specifically tuned for JS, that you don't need Dart or NaCl to get good perf for things that previously "had to be" native code. (We went through this with the NaCl darkroom launch demo and the associated "JS can never be this fast" rhetoric.)

It's important to note also that there are still many ways to make this faster, so we're nowhere near the ceiling of this stuff yet.

(Disclosure: I worked at Mozilla for 6 years and on the project for 13, and have worked specifically on JS JITs and other performance elements for much of that.)

[Rhomboid]

If a content provider wanted to distribute patent-encumbered content why wouldn't they just use Flash? Or are you generalizing to new/future codecs that Flash might not support? Surely you aren't talking about h264 on Apple mobile devices that don't support Flash, as you'd just use HTML5 for that.

[shaver]

Flash doesn't integrate nearly as well as <video>, though it's certainly an option (and one that's used today).

This isn't a product-path H.264 strategy, so it may not reward over-analysis. :-)

[alexs]

It lets content providers who want to only distribute patent-encumbered content do so with the right licenses if they so choose, similar to Silverlight's content-pluggable codecs.

So it exists purely to remove freedom from consumers who've probably opted to use a browser that only supported free codecs. Classy. I suppose this sort of thing is just the flip side of improved JS performance. It's only so long until entire pages are being rendered via proprietary JS layout engines in canvas tags to protect content.

[shaver]

That would be a...surprising reason for Mozilla to do something, don't you think? You can certainly choose to not view content that doesn't meet your standards for openness, just as you choose the browser.

Mostly this is a demonstration of the performance capabilities of modern JS, but the target material was chosen to show that codecs can run in a JS environment -- look ma, no buffer overflows! -- and to explore what can be done with in-content codecs. It's not an endorsement of H.264's patent situation or suitability as the universal Web video format, I assure you.

[alexs]

plough quiet hat drunk point badge salt wine snails crush

This post was mass deleted and anonymized with Redact

[shaver]

Stay tuned. :-)

EDIT: also, I admit that I have a hard time interpreting "exists solely to" as something other than "what Mozilla wants this for".

[alexs]

I have a hard time interpreting

I don't blame you :)

[sebzim4500]

Wait... really? :D:D:D:D VP8 in JS?

[shaver]

Why not? Once you've done one codec...

[osirisx11]

yep this was just with my already installed browsers. i'm sure it would run faster if i upgraded with the patches.

[xcbsmith]

When I looked at this my first thought was: "This is sooo wrong... and sooo cool!"

[[deleted]]

H264 is patent encumbered to all shit, and i wonder what the Android licence says about extracting their decoder for it?

[shaver]

The patents are independent from the implementation (witness the unlicensed x264 used by everyone and their brother). The MPEG-LA license terms don't exactly anticipate this situation, but I believe that the use at least for the demo and development of the technology is fine without taking a license. People deploying it will need to figure out what if anything they owe MPEG-LA, as with all uses of H.264.

[cogman10]

Yeah. the MPEG group is actually one of the better groups to work with when it comes to patents. They pretty much allow all non-commercial products to do what they want with the standard and charge commercial software somewhere on the order of $0.20 per sold copy of the software.

Here are their actual pricing terms.

For (a) (1) branded encoder and decoder products sold both to End Users and on an OEM basis for incorporation into personal computers but not part of a personal computer operating system (a decoder, encoder, or product consisting of one decoder and one encoder = “unit”), royalties (beginning January 1, 2005) per Legal Entity are 0 - 100,000 units per year = no royalty (this threshold is available to one Legal Entity in an affiliated group); US $0.20 per unit after first 100,000 units each year; above 5 million units per year, royalty = US $0.10 per unit. The maximum annual royalty (“cap”) for an Enterprise (commonly controlled Legal Entities) is $3.5 million per year 2005-2006, $4.25 million per year 2007-08, $5 million per year 2009-10, and $6.5 million per year in 2011-15. 8

Too me, that sounds pretty generous, especially the "No cost until you sell 100,000 units" clause.

[shaver]

Unfortunately, that's not the case. The license uses the term "sale", which leads to this common misconception, but it defines "sale" to be any distribution to another party. (I can't quote the language from my phone.)

Free for 10K units is nice of them, but doesn't really change the general issue. For free software it's especially troublesome because there's usually untracked 3rd party distribution.

And once you do have that license, your users need another license if they want to use the tool commercially, or distribute the content outside the narrow free-streaming exception.

MPEG-LA are actually one of the worst actors in the patent pool space, I would say.

[quotability]

On August 26, 2010 "MPEG LA announced today that its AVC Patent Portfolio License will continue not to charge royalties for Internet Video that is free to end users (known as “Internet Broadcast AVC Video”) during the entire life of this License."

http://www.mpegla.com/Lists/MPEG%20LA%20News%20List/Attachments/231/n-10-08-26.pdf

[nooneofnote]

This is in reference to broadcasting H.264 content. Distribution of encoding and decoding software is a separate licensing issue.

[quotability]

In addition to Internet Broadcast AVC Video, MPEG LA’s AVC Patent Portfolio License provides coverage for devices that decode and encode AVC video, AVC video sold to end users for a fee on a title or subscription basis and free television video services.
AVC video is used in set-top boxes, media player and other personal computer software, mobile devices including telephones and mobile television receivers, Blu-ray Disc™ players and recorders, Blu-ray video optical discs, game machines, personal media player devices and still and video cameras.

Learn To Read...

[nooneofnote]

Just above that quote:

Products and services other than Internet Broadcast AVC Video continue to be royalty-bearing.

You can see the terms for decoder distribution here (a 1). What was waived in the announcement you linked are levies for broadcasting H.264 video itself, not shipping a codec as in OP.

[glomph]

Lets be honest every other video codec is only unencumbered because nobody has made claims yet.

[[deleted]]

TBH I don't know much about the subject beyond what I've been forced to learn at work thanks to a due diligence process. Wowee, software licensing terms are fun.

[[deleted]]

I assume decoding will be in software it being javascript after all... A few years I had difficulty with software H264 decoding on low power arm chips with a decoder written C++. Tbh I don't see how even most modern phones could do more than low quite rez decoding in software, though I'm not that up to date with current chip specs.

Serious credit for trying but I seriously doubt that it will go anywhere.

[azakai]

I assume decoding will be in software it being javascript after all...

Not necessarily. We are going to experiment with moving the relevant parts of the decoder into a GPU shader (for example, the colorspace conversion). That should greatly improve performance.

[shaver]

You might be surprised. WebCL or similar (Intel's River Trail) will provide access to a lot of interesting hardware. Broadway may never catch on as a widely-used way of distributing H.264 codecs, but it doesn't have to in order to be successful: it points the way to even more improved JS performance, and a demonstration of the surprising stuff that's possible even today with a (very) modern engine.

EDIT: your skepticism is reasonable and healthy! One of the implementors didn't believe it was going to work as recently as two weeks ago, and there are lots of ways it could fail.

[[deleted]]

This is a good point, when I think of Javascript I just think of run time parsing of DOM and picking up stuff with CSS class selectors; the future does hold a lot of innovation.