Tim Starling wrote:
We're planning on doing another master split this
weekend.
Looks like I've run out of time. I've repartitioned all the slaves, that
was the main thing that required off-peak traffic conditions. We can do
the master switch itself during the day on Monday UTC, when there are more
sysadmins around.
The master partitions are designated s1, s2 and s3 (s for section),
assigned as follows:
s1: enwiki
s2: bgwiki, bgwiktionary, commonswiki, cswiki, dewiki, enwikiquote,
enwiktionary, eowiki, fiwiki, idwiki, itwiki, nlwiki, nowiki, plwiki,
ptwiki, svwiki, thwiki, trwiki, zhwiki
s3: everything else
There will also be two additional slave partitions -- sets of servers
which only replicate a reduced set of databases:
s2a: dewiki
s3a: frwiki, jawiki
We do this so that holbach and webster, slave servers with only 4GB of
memory, will have something useful to do. Holbach will be doing s2a and
webster will be doing s3a.
The server assignments at present are:
s1: db2, db3, ariel, db4, db6, db7
s2: db8, ixia, lomaria
s2a: holbach
s3: samuel, thistle, db1, db5
s3a: webster
The main potential pitfall I can see in the master switch is the
possibility that a client with an old configuration may write to the stale
copy of the databases in the old master. To avoid this, I'm proposing that
we change masters for both s2 and s3, leaving old clients harmlessly
attempting to write to a read-only slave. I've enabled binlogs on db8 and
thistle so that they can become the new masters. I'm hoping a Dell such as
db8 will be able to cope with the master load, which is typically less
than the full slave load.
It should be fairly straightforward:
* switch wiki to read-only
* change master assignment in MW
* switch master for s2
* switch master for s3
* switch wiki to r/w
It should only take a couple of minutes if we script the master switches.
-- Tim Starling