-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi,
the user database server (zedler) has run out of space on the MySQL filesystem; however, there is still 250GB free in the InnoDB table space. to reclaim this free space, we need to dump the reload the database. the database will be inaccessible while this is in progress.
based on the time taken to run nightly backups, the dump should take about 5 hours. however, i will run it in parallel to improve performance. it will take a bit longer to reload it afterwards.
after the reload, i will not restore the commonswiki replica that was previously on the sql server. the correct way to join user databases to commons is to create them on the sql-s4 server.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
we successfully dumped the databases, but after a reboot, the SCSI card attaching the server to its external disk array malfunctioned (it's likely this happened previously and wasn't noticed until a reboot).
i am currently looking for a temporary solution to bring user databases back online until the server can be repaired or replaced.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
i have started a MySQL instance on another server which will temporarily serve as 'sql', and have started importing the databases. this will probably take a while.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
today we replaced the SCSI card in zedler (sql); during the maintenance, yarrow's disk array unexpectedly rebooted which caused MySQL to crash. it is currently running database recovery and will be available again shortly.
at some point in the next week we will move the user databases from the temporary server back to zedler (by copying the data files directly, which will be much faster).
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
the import is now finished and user databases are accessible again.
- river.
toolserver-announce@lists.wikimedia.org