[Engineering] MediaWiki `scap clean`

[Chad Horohoe]

Hi,

As you may or not have noticed, in recent weeks there's a scap plugin
available for MediaWiki deployments. I wanted to give everyone a quick
heads up on this.

`scap clean` is a plugin that I wrote that automates the *incredibly* messy
and time-consuming process of cleaning up obsolete branches of MediaWiki
from production. A couple of big reasons why we want this:
- Stop wasting space
- Stop wasting time in rsync on dumb stuff
- Have the deployed directories mirror what we actually care about (sanity)

Ok, so we had a bunch of dsh commands that did this...it was ugly and
required copy+pasting every single week. Now: if you're wanting to cleanup,
it's as simple as:

`scap cleanup 1.29.0-wmf.4`

This does a couple of things
1) Prunes old wmf/* branches from Gerrit (yay finally automated cleanup!)
2) Prunes the directory from /srv/mediawiki-staging/* on the active master
3) Prunes the directory from /srv/mediawiki/* on all apaches (masters,
debug, etc)
4) Prunes the junk l10nupdate cache files (/var/cache/l10nupdate/*) that
were **never** being cleaned up before

(4) especially has bit us in the past with causing tin/mira to run out of
disk space. Additionally, you may see a --keep-static option. This is for
soon-to-be-dead branches where we only care about static assets
(CSS/JS/images/etc) but the PHP is not being executed. --keep-static will
prune the non-static assets, clean the l10nupdate cache, and still keep the
branch intact (we go from ~4.4GB -> ~30MB when we go static-only!)

Basically, a branch gets `scap clean` run twice: once when it goes out of
service with --keep-static, and a second time a few weeks later when we
really don't want it anymore.

Sooooo, all this being said: you probably won't ever need/want to run this.
But I wanted to make sure everyone knew what it does, just in case they saw
it and was curious. And don't worry, I have safeguards to keep you from
accidentallying the active branches ;-)

-Chad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/engineering/attachments/20170404/78ae6f13/attachment.html>