Brion Vibber wrote:
Got up with code review and went to put things live on
test.wikipedia
for some live testing....
Test is completely borked due to missing magic words; for some reason it
seems to be loading the currently-deployed localization files instead of
the ones in its master copy.
This has been a general annoyance for a long time, but is particularly
disruptive when every page view is blocked by a missing magic word that
isn't even needed for anything. :)
We'll want to figure out what the hell is causing that so we can make
test work before finishing up the pre-deployment testing and updating
the code.
(Sysadmins -- I've disabled scap. You can sync-file individual files as
necessary, but do *not* do a scap or sync-common-all until we've
confirmed everything works!)
Fixed all that and scapped, then saw your post on wikitech-l a couple of
hours later.
Test was loading the serialized data files from /home, which were out of
date. When you rebuild them, it draws the source data from /usr, so you
have to sync the local common directory first. Scap handles this
automatically. Apparently someone rebuilt them without doing this, because
their timestamps had been updated but they still had old data. Syncing to
srv35 and then doing "make -B" fixed it.
There were a number of issues with the new codebase, most of which I fixed
throughout the day. Details are in the server admin log.
-- Tim Starling