On Fri, Mar 7, 2014 at 5:27 PM, Bryan Davis bd808@wikimedia.org wrote:
I recommend that we add two new weekly cleanup steps:
- When we deploy a new branch to group0 (Thursdays), all branches
retired more than 5 weeks ago should be removed. This should really only include multiple branches the first time it's done to catch up. After that it will be an "add a branch, kill a branch" situation. With the current release cadence this will keep us at 7 checked out branches on tin, 2 versions in active use and 5 waiting for potential cache references to expire.
I did this process today for the first time. I ended up leaving 8 branches on tin instead of 7. I removed these branches:
* 1.23wmf6 * 1.23wmf7 * 1.23wmf8 * 1.23wmf9 * 1.23wmf10
1.23wmf11 was last live on 2014-02-06. I should be safe to delete today, but I didn't want to tempt fate or anger any deity that may control front-end 404 generation.
Here's what I did to delete each branch:
* Cleanup bits symlinks to branch ** /a/common/multiversion/deleteMediaWiki php-1.23wmfX * Create symlinks cleanup patch ** git rm -r docroot/bits/static-1.23wmfX ** git rm -r w/static-1.23wmfX ** NOLOGMSG=1 git commit -m 'Remove 1.23wmfX symlinks' * Revert change on tin ** NOLOGMSG=1 git reset HEAD^ --hard * Repeat steps above for each branch as needed * Approve and pull symlink cleanup patch to tin * Delete branch checkout in /a/common ** rm -rf /a/common/php-1.23wmfX * Repeat steps above for each branch as needed * scap
When scap runs it will leave the php-1.23wmfX/cache/l10n/*.cdb files in place because we ignore *.cdb purposefully in the rsync command run by sync-common. This can be cleaned up with dsh:
dsh -g mediawiki-installation -M -F 40 -- \ 'sudo -u mwdeploy -- rm -r /usr/local/apache/common-local/php-1.23wmfX'
Chad has suggested that an additional step of tagging the end of the branch and deleting it be added to this process. Right now there is a minor blocker on that as the branch delete can only be done via the gerrit web ui and requires user permissions that I do not currently have. I will make the tags though and let Chad either delete the branches or figure out how to let me and Sam do it.
- When we move group1 to the newest branch (Tuesdays), the cache/l10n
directory of all non-active branches should be purged. By this point there is little chance that we will be reverting the wikipedias to the N-2 branch and thus the l10n cache is just taking up disk space and slowing down rsync comparisons.
I did this procedure for the first time this week as well purging the l10n cache for 1.23wmf13, 1.23wmf14 and 1.23wmf15. The process I followed was:
* Delete l10n files in /a/common ** sudo -u l10nupdate rm --recursive ${COMMON}/php-${VERSION}/cache/l10n/* * Delete l10n files on cluster hosts ** dsh "${MW_DSH_ARGS[@]}" -- \ "sudo -u mwdeploy rm --recursive ${SHARED}/php-${VERSION}/cache/l10n/*" * !log deletion
This has been turned into a script in the scap project that is awaiting code review [0]. Once it is merged the process will be reduced to:
scap-purge-l10n-cache --version 1.23wmfX
[0]: https://gerrit.wikimedia.org/r/#/c/118337/