I've gotten some great review feedback from Gilles on the desktop-web integration of my ogv.js JavaScript & Flash compatibility layers for Ogg Theora/Vorbis media files -- thanks Gilles!
* libraries: https://gerrit.wikimedia.org/r/#/c/165477/ * desktop integration: https://gerrit.wikimedia.org/r/#/c/165478/
These are getting pretty close to ready to land, I think.
I would love to get some review on the mobile overlay I've whipped up as well. This supports both native WebM playback (Android Chrome, Android Firefox, Firefox OS) and ogv.js playback (iOS 7/8 Safari).
* mobile overlay: https://gerrit.wikimedia.org/r/#/c/165479/ * Live demo: https://ogvjs-testing.wmflabs.org/
A few open questions:
1) Is this the right way to do mobile overlay code? (It's basically a rip of the existing photo viewer overlay in MobileFrontend, but lives in TimedMediaHandler.) Is the overlay interface stable enough for other extensions to use it for mobile-specific features? (I had to make updates for object-model and template things that changed since this summer.)
2) Is the inline icon too huge/ugly here for audio files? Should it be arranged differently, or display the player inline instead of as an overlay for audio?
3) Should more controls be added to the overlay's bottom toolbar, such as manual resolution selection or an 'Open in VLC' link to support HD playback on iOS?
4) Should we autoplay when opening the overlay, or require a second tap?
5) How should we handle devices with no native playback that are either too slow (iOS 6 Safari) or lack necessary features needed for the player (Windows Phone)?
Current known bugs in the mobile overlay:
* CPU speed check not yet integrated to force to lowest resolution for old iPhones/iPads (this exists on the desktop integration, just needs to be moved to common code)
* autoplay doesn't seem to work with native playback right now
-- brion
This is awesome. Rather than MMV depending on MobileFrontend, if we want to make this work on mobile I imagine a better thing to do would be to make it dependent on Extension:Mantle [1] since Mantle currently provides the templates that this would need and is a much more lightweight extension to add as a dependency.
This could be a great opportunity to move Overlay.js from MobileFrontend to extension Mantle and try and begin consolidating MediaViewer, MobileFrontend, VE and Flow's Overlay/Modal JS and css code here. We already successfully began porting MobileFrontend's class system to OOJS so this would give us an opportunity to reduce some of MediaWiki's code debt relating to overlays and move closer to a unified OOUI modal/overlay thing. I'd be interested in looking at how OOUI's Modals and MobileFrontend's Overlay differ.
This is what I'd do personally, but I'd be interested in other opinions. I can help out with code review and moving this along as this sounds great! :-)
See also [Tracking] Consolidate code for MediaViewer across desktop and mobile [2]
[1] https://www.mediawiki.org/wiki/Extension:Mantle [2] https://bugzilla.wikimedia.org/show_bug.cgi?id=63504
On Mon, Oct 13, 2014 at 2:32 PM, Brion Vibber bvibber@wikimedia.org wrote:
I've gotten some great review feedback from Gilles on the desktop-web integration of my ogv.js JavaScript & Flash compatibility layers for Ogg Theora/Vorbis media files -- thanks Gilles!
- libraries: https://gerrit.wikimedia.org/r/#/c/165477/
- desktop integration: https://gerrit.wikimedia.org/r/#/c/165478/
These are getting pretty close to ready to land, I think.
I would love to get some review on the mobile overlay I've whipped up as well. This supports both native WebM playback (Android Chrome, Android Firefox, Firefox OS) and ogv.js playback (iOS 7/8 Safari).
- mobile overlay: https://gerrit.wikimedia.org/r/#/c/165479/
- Live demo: https://ogvjs-testing.wmflabs.org/
A few open questions:
- Is this the right way to do mobile overlay code? (It's basically a rip of
the existing photo viewer overlay in MobileFrontend, but lives in TimedMediaHandler.) Is the overlay interface stable enough for other extensions to use it for mobile-specific features? (I had to make updates for object-model and template things that changed since this summer.)
- Is the inline icon too huge/ugly here for audio files? Should it be
arranged differently, or display the player inline instead of as an overlay for audio?
- Should more controls be added to the overlay's bottom toolbar, such as
manual resolution selection or an 'Open in VLC' link to support HD playback on iOS?
Should we autoplay when opening the overlay, or require a second tap?
How should we handle devices with no native playback that are either too
slow (iOS 6 Safari) or lack necessary features needed for the player (Windows Phone)?
Current known bugs in the mobile overlay:
- CPU speed check not yet integrated to force to lowest resolution for old
iPhones/iPads (this exists on the desktop integration, just needs to be moved to common code)
- autoplay doesn't seem to work with native playback right now
-- brion
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Mon, Oct 13, 2014 at 7:08 PM, Jon Robson jrobson@wikimedia.org wrote:
This is awesome.
:D
Rather than MMV depending on MobileFrontend, if we want to make this work on mobile I imagine a better thing to do would be to make it dependent on Extension:Mantle [1] since Mantle currently provides the templates that this would need and is a much more lightweight extension to add as a dependency.
For the moment I'm actually inheriting directly from MobileFrontend's ImageOverlay class for the MediaOverlay. But this might be too fragile a coupling.
As for lightweight/heavyweight dependency; the mobile overlay is only used within MobileFrontend's interface right now. But I'm thinking long-term might want to rework the whole TimedMediaHandler MwEmbedPlayer front-end in a cleaner way that eventually gives us a universal zoomable, overlayable media player overlay.
Not quite sure where to work things in with MMV too; possibly MMV needs a hook point that TMH can extend with just the video player portion?
This could be a great opportunity to move Overlay.js from
MobileFrontend to extension Mantle and try and begin consolidating MediaViewer, MobileFrontend, VE and Flow's Overlay/Modal JS and css code here. We already successfully began porting MobileFrontend's class system to OOJS so this would give us an opportunity to reduce some of MediaWiki's code debt relating to overlays and move closer to a unified OOUI modal/overlay thing. I'd be interested in looking at how OOUI's Modals and MobileFrontend's Overlay differ.
That sounds good but lots of work. :D
Would there be some kind of hook so MobileFrontend can give a consistent 'mobile style' to the overlays versus desktop views, or would we try to make a single overlay UI that works well on both?
-- brion
cool.
comments inline:
On Oct 13, 2014, at 4:32 PM, Brion Vibber bvibber@wikimedia.org wrote:
I've gotten some great review feedback from Gilles on the desktop-web integration of my ogv.js JavaScript & Flash compatibility layers for Ogg Theora/Vorbis media files -- thanks Gilles!
- libraries: https://gerrit.wikimedia.org/r/#/c/165477/
- desktop integration: https://gerrit.wikimedia.org/r/#/c/165478/
These are getting pretty close to ready to land, I think.
I would love to get some review on the mobile overlay I've whipped up as well. This supports both native WebM playback (Android Chrome, Android Firefox, Firefox OS) and ogv.js playback (iOS 7/8 Safari).
- mobile overlay: https://gerrit.wikimedia.org/r/#/c/165479/
- Live demo: https://ogvjs-testing.wmflabs.org/
A few open questions:
Is this the right way to do mobile overlay code? (It's basically a rip of the existing photo viewer overlay in MobileFrontend, but lives in TimedMediaHandler.) Is the overlay interface stable enough for other extensions to use it for mobile-specific features? (I had to make updates for object-model and template things that changed since this summer.)
Is the inline icon too huge/ugly here for audio files? Should it be arranged differently, or display the player inline instead of as an overlay for audio?
yea the pop-up may not be needed for Audio, or otherwise change the display size of the player in the popup.
- Should more controls be added to the overlay's bottom toolbar, such as manual resolution selection or an 'Open in VLC' link to support HD playback on iOS?
sounds good. I don’t think its possible to know ahead of time if VLC is available, but you can detect a failure to switch the location to a particular application handler. So a bit of UI flow would probably have to accompany that.
- Should we autoplay when opening the overlay, or require a second tap?
Would make sense to auto play, do you need to capture click gestures to initiate audio API in iOS ?
- How should we handle devices with no native playback that are either too slow (iOS 6 Safari) or lack necessary features needed for the player (Windows Phone)?
I think we provide a download link today or at least intended to do so. Then its up to the host machine to have any registered mime handler in browser or otherwise have a handler to open the downloaded asset.
Current known bugs in the mobile overlay:
CPU speed check not yet integrated to force to lowest resolution for old iPhones/iPads (this exists on the desktop integration, just needs to be moved to common code)
autoplay doesn't seem to work with native playback right now
* seek support ... I guess that is a more generalized issue.
-- brion _______________________________________________ Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
On Wed, Oct 15, 2014 at 10:30 AM, Michael Dale mdale@wikimedia.org wrote:
On Oct 13, 2014, at 4:32 PM, Brion Vibber bvibber@wikimedia.org wrote:
- Is the inline icon too huge/ugly here for audio files? Should it be
arranged differently, or display the player inline instead of as an overlay for audio?
yea the pop-up may not be needed for Audio, or otherwise change the display size of the player in the popup.
*nod* I'm especially thinking it might be nice to start audio playing and keep scrolling/reading without being visually interrupted. But not sure offhand how to squeeze good mobile-y UI into a smaller space than the currently displayed icon.
- Should more controls be added to the overlay's bottom toolbar, such as
manual resolution selection or an 'Open in VLC' link to support HD playback on iOS?
sounds good. I don’t think its possible to know ahead of time if VLC is available, but you can detect a failure to switch the location to a particular application handler. So a bit of UI flow would probably have to accompany that.
Yeah, as I recall from our attempt to do this on TimedMediaHandler in the desktop view (before we switched iPads to show the mobile view by default) we basically had to try launching it and see if it fails, then if it failed send the user over to the App Store entry for VLC.
- Should we autoplay when opening the overlay, or require a second tap?
Would make sense to auto play, do you need to capture click gestures to initiate audio API in iOS ?
I've now got autoplay-after-tap working on iOS Safari (with ogv.js) and Android Chrome/Android Firefox/Firefox OS (native playback). iOS requires a little shuffling to initiate audio but it seems to work. :D
- How should we handle devices with no native playback that are either
too slow (iOS 6 Safari) or lack necessary features needed for the player (Windows Phone)?
I think we provide a download link today or at least intended to do so. Then its up to the host machine to have any registered mime handler in browser or otherwise have a handler to open the downloaded asset.
*nod* will have to devise a nice mobile-friendly download UI...
Current known bugs in the mobile overlay:
- CPU speed check not yet integrated to force to lowest resolution for old
iPhones/iPads (this exists on the desktop integration, just needs to be moved to common code)
fixed
- autoplay doesn't seem to work with native playback right now
fixed
- seek support ... I guess that is a more generalized issue.
Yeah, there's two main possibilities for seeking:
1) Buffer the whole file into memory, and allow seeking anywhere within the area that's been buffered.
This sounds awful, but we actually have to buffer the whole file into memory currently on Safari anyway -- I'm kind of banking on most videos being short-ish and loading relatively low-resolution files that aren't too huge. :(
Adding seeking on top of that shouldn't be super hard, I just haven't gotten to it yet.
2) Break up the files into a manifest and small chunks (as with WebM-DASH or Apple's HTTP Live Streaming) that can be individually loaded.
With chunked files, we could seek by jumping to a different chunk and picking up where needed in the stream. This would require more infrastructure work, including generation of files. Sounds hard.
Soo....... I'll probably do number 1) at some point. :)
-- brion
On Oct 19, 2014, at 5:16 AM, Brion Vibber bvibber@wikimedia.org wrote:
- Buffer the whole file into memory, and allow seeking anywhere within the area that's been buffered.
This sounds awful, but we actually have to buffer the whole file into memory currently on Safari anyway -- I'm kind of banking on most videos being short-ish and loading relatively low-resolution files that aren't too huge. :(
Adding seeking on top of that shouldn't be super hard, I just haven't gotten to it yet.
- Break up the files into a manifest and small chunks (as with WebM-DASH or Apple's HTTP Live Streaming) that can be individually loaded.
With chunked files, we could seek by jumping to a different chunk and picking up where needed in the stream. This would require more infrastructure work, including generation of files. Sounds hard.
Soo....... I'll probably do number 1) at some point. :)
We can’t do http range requests against the ogg index byte / time mappings ?
—michael
On Sun, Oct 19, 2014 at 9:33 AM, Michael Dale mdale@wikimedia.org wrote:
On Oct 19, 2014, at 5:16 AM, Brion Vibber bvibber@wikimedia.org wrote:
- Buffer the whole file into memory, and allow seeking anywhere within
the area that's been buffered. [snip]
- Break up the files into a manifest and small chunks (as with
WebM-DASH or Apple's HTTP Live Streaming) that can be individually loaded. [snip]
We can’t do http range requests against the ogg index byte / time mappings ?
As I recall, XHR won't let us send 'Range' header without some CORS opt-in we haven't figured out how to get working right. If we can resolve that, should be able to rig up seeking & chunked streaming based on Range headers (this would also let us run large files without eating up memory in Safari).
-- brion
On Sun, Oct 19, 2014 at 9:47 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Sun, Oct 19, 2014 at 9:33 AM, Michael Dale mdale@wikimedia.org wrote:
We can’t do http range requests against the ogg index byte / time mappings ?
As I recall, XHR won't let us send 'Range' header without some CORS opt-in we haven't figured out how to get working right. If we can resolve that, should be able to rig up seeking & chunked streaming based on Range headers (this would also let us run large files without eating up memory in Safari).
Woohoo, Faidon fixed the CORS bit with a Varnish tweak -- now allows OPTIONS requests through to the backend so the CORS preflight request can go and confirm that it's ok to send Range headers:
https://bugzilla.wikimedia.org/show_bug.cgi?id=55631 https://gerrit.wikimedia.org/r/#/c/167542/
I'll work on seeking this weekend if I don't have time before then. :D
-- brion
On Mon, Oct 20, 2014 at 10:58 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Sun, Oct 19, 2014 at 9:47 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Sun, Oct 19, 2014 at 9:33 AM, Michael Dale mdale@wikimedia.org wrote:
We can’t do http range requests against the ogg index byte / time mappings ?
As I recall, XHR won't let us send 'Range' header without some CORS opt-in we haven't figured out how to get working right. If we can resolve that, should be able to rig up seeking & chunked streaming based on Range headers (this would also let us run large files without eating up memory in Safari).
Woohoo, Faidon fixed the CORS bit with a Varnish tweak -- now allows OPTIONS requests through to the backend so the CORS preflight request can go and confirm that it's ok to send Range headers:
Spent some weekend time on this and got both chunked streaming for Safari and a bisection-based seek working[1]. Haven't merged it into the TMH version yet as I'm not satisfied with the seek performance, but I have a pretty good idea how to seek based on Ogg Skeleton metadata by using liboggz on the C side and giving it custom i/o functions that manipulate the JS XHR wrapper.
Of course, now I need to redesign the controls for the mobile overlay so there's a way to actually invoke the seeking. ;) (On desktop it should be able to use the existing controls in MwEmbedPlayer.)
[1] https://github.com/brion/ogv.js/commit/cad8ef8d067a509a2a18b4ed683e2c4a17b51...
-- brion
Wow, that's great news that the CORS stuff is finally working !!!
DJ
On Mon, Oct 27, 2014 at 3:15 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Mon, Oct 20, 2014 at 10:58 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Sun, Oct 19, 2014 at 9:47 AM, Brion Vibber bvibber@wikimedia.org wrote:
On Sun, Oct 19, 2014 at 9:33 AM, Michael Dale mdale@wikimedia.org wrote:
We can’t do http range requests against the ogg index byte / time mappings ?
As I recall, XHR won't let us send 'Range' header without some CORS opt-in we haven't figured out how to get working right. If we can resolve that, should be able to rig up seeking & chunked streaming based on Range headers (this would also let us run large files without eating up memory in Safari).
Woohoo, Faidon fixed the CORS bit with a Varnish tweak -- now allows OPTIONS requests through to the backend so the CORS preflight request can go and confirm that it's ok to send Range headers:
Spent some weekend time on this and got both chunked streaming for Safari and a bisection-based seek working[1]. Haven't merged it into the TMH version yet as I'm not satisfied with the seek performance, but I have a pretty good idea how to seek based on Ogg Skeleton metadata by using liboggz on the C side and giving it custom i/o functions that manipulate the JS XHR wrapper.
Of course, now I need to redesign the controls for the mobile overlay so there's a way to actually invoke the seeking. ;) (On desktop it should be able to use the existing controls in MwEmbedPlayer.)
[1] https://github.com/brion/ogv.js/commit/cad8ef8d067a509a2a18b4ed683e2c4a17b51...
-- brion
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
From: Brion Vibber bvibber@wikimedia.org Date: Sunday, October 26, 2014 at 8:15 PM To: Michael Dale mdale@wikimedia.org Cc: Wikimedia Foundation Multimedia Team multimedia@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org Subject: Re: [WikimediaMobile] [Multimedia] Review needed for mobile video overlay (native and ogv.js playback)
Spent some weekend time on this and got both chunked streaming for Safari and a bisection-based seek working[1]. Haven't merged it into the TMH version yet as I'm not satisfied with the seek performance, but I have a pretty good idea how to seek based on Ogg Skeleton metadata by using liboggz on the C side and giving it custom i/o functions that manipulate the JS XHR wrapper.
Of course, now I need to redesign the controls for the mobile overlay so there's a way to actually invoke the seeking. ;) (On desktop it should be able to use the existing controls in MwEmbedPlayer.)
[1] https://github.com/brion/ogv.js/commit/cad8ef8d067a509a2a18b4ed683e2c4a17b51 4e9
Wow, great work.
Yes using the ogg index would be the way to go, bisection seeking never worked too well for native implementations either. All derivatives created in WMF should include ogg skeleton / index.
michael
Got seeking with index working last weekend; spent some holiday time updating the TimedMediaHandler plugin to use it:
http://ogvjs-testing.wmflabs.org/wiki/Demo
I've also updated the compatibility detection code to use the JavaScript player for IE 10 and 11, since the Flash version cannot do HTTP Range requests (and thus cannot seek). These still use Flash for audio output, but all the decoding is done in JS.
There's a funky bug where the player state gets confused if you play through a second time after completion, still have to track that down.
Also still need to rework the MobileFrontend integration to be more stable; Jon's suggested refactoring it to just the internals so I can piggyback directly on the existing mobile media viewer overlay (perhaps with an extensible hook?) without trying to extend the still-fragile overlay API. Will also need to add seek controls on mobile. :)
-- brion
On Tue, Oct 28, 2014 at 6:41 AM, Michael Dale mdale@wikimedia.org wrote:
From: Brion Vibber bvibber@wikimedia.org Date: Sunday, October 26, 2014 at 8:15 PM To: Michael Dale mdale@wikimedia.org Cc: Wikimedia Foundation Multimedia Team multimedia@lists.wikimedia.org, mobile-l mobile-l@lists.wikimedia.org Subject: Re: [WikimediaMobile] [Multimedia] Review needed for mobile video overlay (native and ogv.js playback)
Spent some weekend time on this and got both chunked streaming for Safari and a bisection-based seek working[1]. Haven't merged it into the TMH version yet as I'm not satisfied with the seek performance, but I have a pretty good idea how to seek based on Ogg Skeleton metadata by using liboggz on the C side and giving it custom i/o functions that manipulate the JS XHR wrapper.
Of course, now I need to redesign the controls for the mobile overlay so there's a way to actually invoke the seeking. ;) (On desktop it should be able to use the existing controls in MwEmbedPlayer.)
[1] https://github.com/brion/ogv.js/commit/cad8ef8d067a509a2a18b4ed683e2c4a17b51...
Wow, great work.
Yes using the ogg index would be the way to go, bisection seeking never worked too well for native implementations either. All derivatives created in WMF should include ogg skeleton / index.
—michael