These conversions have been running in the background, at a moderate load to ensure we don't slow down newer uploads much, and are up to the "P"s alphabetically so we're on schedule. :) I've temporarily stopped the batch process pending the upcoming datacenter switchover test, and will continue once it's switched. -- brion On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; wrote: > Quick update... no major problems noticed so far. One configuration > change I made was to re-enable VP8 transcodes for new files, since > disabling them completely lead to the existing ones not being used. Later > in the transition, or afterwards, we'll clean up the now-unused VP8 > transcodes. > > If I'm reading numbers in > https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_ > transition/reports right, almost 5% of files by count or 12% by duration > have been converted so far. > > That leads to a remaining duration somewhere between 9 weeks (assuming 9 > days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on > actual factors like whether the remaining files are the same mix of > resolutions, and how we adjust the load balance on the job queue. > > -- brion > > On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; > wrote: > >> Batch process is continuing... over 2200 source videos have been >> compressed to VP9 so far, of the 120k+ total in the system. >> >> Seeing big improvements in overall compression, though it's hard to tell >> how representative the subset of conversions done so far is. I'll post >> occasional updates to the transcode report charts at: >> >> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/r… >> >> In the histograms you can see that not only is the average bitrate down >> for VP9 vs VP8, but there's a larger "spread" between low and high >> bandwidth in the VP9 versions thanks to the constrained-quality >> configuration vs the old fixed bitrate target. This allows files that >> compress well to use fewer bits, while those that have high frame rates or >> lots of detail/motion use more bits to encode higher quality than they did >> before. >> >> It will take at least a few weeks to recompress everything, and we may >> or may not want to increase the number of simultaneous jobs (so far the >> servers are running beautifully, but are a bit under-loaded). >> >> -- brion >> >> >> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; >> wrote: >> >>> Configuration has been updated and VP9 mode swapped in. Newly uploaded >>> videos should start encoding as VP9 starting now. >>> >>> I'll start the backfill batch process tonight or tomorrow, and will >>> likely stop and restart it a few times over the coming weeks if anything >>> needs adjustment. Please file tasks in phabricator and/or give me a ping on >>> IRC if any issues come up with the new conversions, or with old files! >>> (Existing VP8 files will be left as-is for now until we're sure >>> everything's up to snuff.) >>> >>> Big thanks to everybody who's helped prepping this, with libvpx and >>> ffmpeg deployments, with the patches and review, and with final deployment >>> which always takes longer than you expect. :) This'll be a nice improvement >>> to our video output for now, and lays a lot of groundwork for next steps. >>> >>> -- brion >>> >>> >>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; >>> wrote: >>> >>>> Oh and one more thing! >>>> >>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K") >>>> resolutions, which people can manually bump up to when watching videos with >>>> a suitable 4K source on a high-res screen. They use higher data rates, but >>>> only a small fraction of input files are 4K so should not significantly >>>> increase disk space projections for now. >>>> >>>> These can take a long time to compress, so if we find it's problematic >>>> we'll turn them back off until the jobs can be split into tiny chunks >>>> (future work planned!), but it works in my testing and shouldn't clog the >>>> servers now that we have more available. >>>> >>>> (Note that the ogv.js player shim for Safari will not handle >>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or >>>> iPad; for best results for 4K playback use Firefox, Chrome, or a >>>> Chromium-based browser.) >>>> >>>> -- brion >>>> >>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; >>>> wrote: >>>> >>>>> Ok, after some delay for re-tweaking the encoding settings for higher >>>>> quality when needed, and pulling in some other improvements to the config >>>>> system, all related updates to TimedMediaHandler have been merged. :D >>>>> >>>>> If all goes well with the general deployments in the next few days, >>>>> expect the beginning of VP9 rollout starting next week. >>>>> >>>>> Changes since the earlier announcement: >>>>> * the new row-multithreading will be available, which allows higher >>>>> threading usage at all resolutions; encoding times will be more like 1.5-2x >>>>> slower instead of 3-4x slower. >>>>> * switch to constrained quality with a larger max bitrate: many files >>>>> will become significantly smaller in their VP9 versions, but some will >>>>> actually increase in exchange for a huge increase in quality -- this is >>>>> mostly 60fps high-rate files, and those with lots of motion and detail that >>>>> didn't compress well at the default low data rates. >>>>> >>>>> -- brion >>>>> >>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber &lt;bvibber(a)wikimedia.org&gt; >>>>> wrote: >>>>> >>>>>> Awesome sauce. Thanks Moritz! >>>>>> >>>>>> -- brion >>>>>> >>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff < >>>>>> mmuhlenhoff(a)wikimedia.org&gt; wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote: >>>>>>> > Current state on this: >>>>>>> > >>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we >>>>>>> start with >>>>>>> > best performance; Moritz made a start on libvpx but we still have >>>>>>> to >>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all >>>>>>> the way to >>>>>>> > 3.4) >>>>>>> >>>>>>> I've completed this today. We now have a separate repository >>>>>>> component >>>>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg >>>>>>> 3.2.10 >>>>>>> (thus allowing us to follow the ffmpeg security updates released in >>>>>>> Debian >>>>>>> with a local rebuild) with backported row-mt support and linked >>>>>>> against >>>>>>> libvpx 1.7.0. >>>>>>> >>>>>>> I tested re-encoding >>>>>>> >>>>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_201… >>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which >>>>>>> results in >>>>>>> a size reduction from 48M to 31M. >>>>>>> >>>>>>> When using eight CPU cores on one of our video scaler servers, >>>>>>> enabling row-mt >>>>>>> gives a significant performance boost; encoding time went down from >>>>>>> 5:31 mins >>>>>>> to 3:36 mins. >>>>>>> >>>>>>> All the details can be found at >>>>>>> https://phabricator.wikimedia.org/T190333#4324995 >>>>>>> >>>>>>> Cheers, >>>>>>> Moritz >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Wikitech-l mailing list >>>>>>> Wikitech-l(a)lists.wikimedia.org >>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >>>>>> >>>>>> _______________________________________________