Brion Vibber wrote:

Got up with code review and went to put things live on test.wikipedia for some live testing....

Test is completely borked due to missing magic words; for some reason it seems to be loading the currently-deployed localization files instead of the ones in its master copy.

This has been a general annoyance for a long time, but is particularly disruptive when every page view is blocked by a missing magic word that isn't even needed for anything. :)

We'll want to figure out what the hell is causing that so we can make test work before finishing up the pre-deployment testing and updating the code.

(Sysadmins -- I've disabled scap. You can sync-file individual files as necessary, but do *not* do a scap or sync-common-all until we've confirmed everything works!)

Fixed all that and scapped, then saw your post on wikitech-l a couple of hours later.

Test was loading the serialized data files from /home, which were out of date. When you rebuild them, it draws the source data from /usr, so you have to sync the local common directory first. Scap handles this automatically. Apparently someone rebuilt them without doing this, because their timestamps had been updated but they still had old data. Syncing to srv35 and then doing "make -B" fixed it.

There were a number of issues with the new codebase, most of which I fixed throughout the day. Details are in the server admin log.

-- Tim Starling