On 6 mrt. 2014, at 14:26, Brion Vibber bvibber@wikimedia.org wrote:
Spent some time tonight on audio/video sync; it's now working both with native Web Audio API and with the Flash audio shim.
Check out a video with someone talking, and marvel at the synchronization! https://brionv.com/misc/ogv.js/demo/#file=Sneak_Preview_-_Wikipedia_VisualEd... (Hi Trevor!)
Wow, really impressive brion. !
I also snuck in a size selection override, so you can pick the larger 480p or original files (note that webm originals don't play as only a Theora decoder is included so far), or down to the tiny 160p, to see the quality and CPU usage differences.
If there's no objection to use of a Flash shim, I'll keep doing some experiments in that direction and see if I can rig up a fully-Flash version that works in old versions of Internet Explorer as well. (The main ogv.js needs no Flash as long as needed APIs are available, and even runs on iOS though it's a bit slow for videos.)
Well, even if there are objections, it might be useful to make sure that it becomes a more suitable solution for other parties as well. And more interested parties would mean those might want to invest in the codebase. Should be a win
DJ
Audio-only files run great on iOS 7 devices. The 160p video transcodes we experimentally enabled recently run *great* on a shiny 64-bit iPhone 5s, but are still slightly too slow on older models.
The Flash audio shim for IE is a very simple ActionScript3 program which accepts audio samples from the host page and outputs them -- no proprietary or patented codecs are in use. It builds to a .swf with the open-source Apache Flex SDK, so no proprietary software is needed to create or update it.
I'm also doing some preliminary research on a fully Flash version, using the Crossbridge compiler[3] for the C codec libraries. Assuming it performs about as well as the JS does on modern browsers, this should give us a fallback for old versions of IE to supplement or replace the Cortado Java player... Before I go too far down that rabbit hole though I'd like to get peoples' opinions on using Flash fallbacks to serve browsers with open formats.
As long as the scripts are open source and we're building them with an open source toolchain, and the entire purpose is to be a shim for missing browser feature support, does anyone have an objection?
[3] https://github.com/adobe-flash/crossbridge
-- brion
On Mon, Oct 7, 2013 at 9:01 AM, Brion Vibber bvibber@wikimedia.orgwrote:
TL;DR SUMMARY: check out this short, silent, black & white video: https://brionv.com/misc/ogv.js/demo/ -- anybody interested in a side project on in-browser audio/video decoding fallback?
One of my pet peeves is that we don't have audio/video playback on many systems, including default Windows and Mac desktops and non-Android mobile devices, which don't ship with Theora or WebM video decoding.
The technically simplest way to handle this is to transcode videos into H.264 (.mp4 files) which is well supported by the troublesome browsers. Unfortunately there are concerns about the patent licensing, which has held us up from deploying any H.264 output options though all the software is ready to go...
While I still hope we'll get that resolved eventually, there is an alternative -- client-side software decoding.
We have used the 'Cortado http://www.theora.org/cortado/' Java applet to do fallback software decoding in the browser for a few years, but Java applets are aggressively being deprecated on today's web:
- no Java applets at all on major mobile browsers
- Java usually requires a manual install on desktop
- Java applets disabled by default for security on major desktop browsers
Luckily, JavaScript engines have gotten *really fast* in the last few years, and performance is getting well in line with what Java applets can do.
As an experiment, I've built Xiph's ogg, vorbis, and theora C libraries cross-compiled to JavaScript using emscriptenhttps://github.com/kripken/emscriptenand written a wrapper that decodes Theora video from an .ogv stream and draws the frames into a <canvas> element:
- demo: https://brionv.com/misc/ogv.js/demo/
- code: https://github.com/brion/ogv.js
- blog & some details:
https://brionv.com/log/2013/10/06/ogv-js-proof-of-concept/
It's just a proof of concept -- the colorspace conversion is incomplete so it's grayscale, there's no audio or proper framerate sync, and it doesn't really stream data properly. But I'm pleased it works so far! (Currently it breaks in IE, but I think I can fix that at least for 10/11, possibly for 9. Probably not for 6/7/8.)
Performance on iOS devices isn't great, but is better with lower resolution files :) On desktop it's screaming fast for moderate resolutions, and could probably supplement or replace Cortado with further development.
Is anyone interested in helping out or picking up the project to move it towards proper playback? If not, it'll be one of my weekend "fun" projects I occasionally tinker with off the clock. :)
-- brion
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l