These conversions have been running in the background, at a moderate load
to ensure we don't slow down newer uploads much, and are up to the "P"s
alphabetically so we're on schedule. :)
I've temporarily stopped the batch process pending the upcoming datacenter
switchover test, and will continue once it's switched.
-- brion
On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber <bvibber(a)wikimedia.org> wrote:
Quick update... no major problems noticed so far. One
configuration change
I made was to re-enable VP8 transcodes for new files, since disabling them
completely lead to the existing ones not being used. Later in the
transition, or afterwards, we'll clean up the now-unused VP8 transcodes.
If I'm reading numbers in
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
transition/reports right, almost 5% of files by count or 12% by duration
have been converted so far.
That leads to a remaining duration somewhere between 9 weeks (assuming 9
days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
actual factors like whether the remaining files are the same mix of
resolutions, and how we adjust the load balance on the job queue.
-- brion
On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber <bvibber(a)wikimedia.org>
wrote:
> Batch process is continuing... over 2200 source videos have been
> compressed to VP9 so far, of the 120k+ total in the system.
>
> Seeing big improvements in overall compression, though it's hard to tell
> how representative the subset of conversions done so far is. I'll post
> occasional updates to the transcode report charts at:
>
>
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/r…
>
> In the histograms you can see that not only is the average bitrate down
> for VP9 vs VP8, but there's a larger "spread" between low and high
> bandwidth in the VP9 versions thanks to the constrained-quality
> configuration vs the old fixed bitrate target. This allows files that
> compress well to use fewer bits, while those that have high frame rates or
> lots of detail/motion use more bits to encode higher quality than they did
> before.
>
> It will take at least a few weeks to recompress everything, and we may or
> may not want to increase the number of simultaneous jobs (so far the
> servers are running beautifully, but are a bit under-loaded).
>
> -- brion
>
>
> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber <bvibber(a)wikimedia.org>
> wrote:
>
>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>> videos should start encoding as VP9 starting now.
>>
>> I'll start the backfill batch process tonight or tomorrow, and will
>> likely stop and restart it a few times over the coming weeks if anything
>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>> IRC if any issues come up with the new conversions, or with old files!
>> (Existing VP8 files will be left as-is for now until we're sure
>> everything's up to snuff.)
>>
>> Big thanks to everybody who's helped prepping this, with libvpx and
>> ffmpeg deployments, with the patches and review, and with final deployment
>> which always takes longer than you expect. :) This'll be a nice improvement
>> to our video output for now, and lays a lot of groundwork for next steps.
>>
>> -- brion
>>
>>
>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber <bvibber(a)wikimedia.org>
>> wrote:
>>
>>> Oh and one more thing!
>>>
>>> For the VP9 configuration I'll be enabling 1440p and 2160p
("4K")
>>> resolutions, which people can manually bump up to when watching videos with
>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>> only a small fraction of input files are 4K so should not significantly
>>> increase disk space projections for now.
>>>
>>> These can take a long time to compress, so if we find it's problematic
>>> we'll turn them back off until the jobs can be split into tiny chunks
>>> (future work planned!), but it works in my testing and shouldn't clog
the
>>> servers now that we have more available.
>>>
>>> (Note that the ogv.js player shim for Safari will not handle
>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>> Chromium-based browser.)
>>>
>>> -- brion
>>>
>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber <bvibber(a)wikimedia.org>
>>> wrote:
>>>
>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>> quality when needed, and pulling in some other improvements to the
config
>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>
>>>> If all goes well with the general deployments in the next few days,
>>>> expect the beginning of VP9 rollout starting next week.
>>>>
>>>> Changes since the earlier announcement:
>>>> * the new row-multithreading will be available, which allows higher
>>>> threading usage at all resolutions; encoding times will be more like
1.5-2x
>>>> slower instead of 3-4x slower.
>>>> * switch to constrained quality with a larger max bitrate: many files
>>>> will become significantly smaller in their VP9 versions, but some will
>>>> actually increase in exchange for a huge increase in quality -- this is
>>>> mostly 60fps high-rate files, and those with lots of motion and detail
that
>>>> didn't compress well at the default low data rates.
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber
<bvibber(a)wikimedia.org>
>>>> wrote:
>>>>
>>>>> Awesome sauce. Thanks Moritz!
>>>>>
>>>>> -- brion
>>>>>
>>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>>> mmuhlenhoff(a)wikimedia.org> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>>> > Current state on this:
>>>>>> >
>>>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so
we
>>>>>> start with
>>>>>> > best performance; Moritz made a start on libvpx but we still
have
>>>>>> to
>>>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating
all
>>>>>> the way to
>>>>>> > 3.4)
>>>>>>
>>>>>> I've completed this today. We now have a separate repository
>>>>>> component
>>>>>> for stretch-wikimedia (named component/vp9) which includes
ffmpeg
>>>>>> 3.2.10
>>>>>> (thus allowing us to follow the ffmpeg security updates released
in
>>>>>> Debian
>>>>>> with a local rebuild) with backported row-mt support and linked
>>>>>> against
>>>>>> libvpx 1.7.0.
>>>>>>
>>>>>> I tested re-encoding
>>>>>>
>>>>>>
https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_201…
>>>>>> (which is a nice fast-paced test file) from VP8 to VP9, which
>>>>>> results in
>>>>>> a size reduction from 48M to 31M.
>>>>>>
>>>>>> When using eight CPU cores on one of our video scaler servers,
>>>>>> enabling row-mt
>>>>>> gives a significant performance boost; encoding time went down
from
>>>>>> 5:31 mins
>>>>>> to 3:36 mins.
>>>>>>
>>>>>> All the details can be found at
>>>>>>
https://phabricator.wikimedia.org/T190333#4324995
>>>>>>
>>>>>> Cheers,
>>>>>> Moritz
>>>>>>
>>>>>> _______________________________________________
>>>>>> Wikitech-l mailing list
>>>>>> Wikitech-l(a)lists.wikimedia.org
>>>>>>
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>>>
>>>>>