On Tue, Mar 24, 2020 at 2:36 AM Manuel Arostegui marostegui@wikimedia.org wrote:
So far we have had normal 1 instance hosts upgraded, multi-instance (2 mysqld processes) upgraded, and we need to have a multisource (labsdb) host upgrade, to make sure 10.4 works fine or to know what might need work (mysqld-exporter https://phabricator.wikimedia.org/T247290 or whatever), better to know in advance.
10.4 also fixes some bugs that are hitting labsdb hosts specifically:
- Grants race condition: https://jira.mariadb.org/browse/MDEV-14732
- GTID works on multisource: https://jira.mariadb.org/browse/MDEV-12012 this is one of the early bugs we filed with MariaDB almost 3 years ago and looks like it is now working even though - this requires some work on the master's side, but my last tests are looking good and if we can enable GTID on labsdb hosts that'd we be a BIG improvement towards avoiding corruption during a crash.
These all sound like good things. And thank you very much, seriously, for the effort you have been putting into thinking about and caring for the wiki replicas.
So, any objections to reimage labsdb1011 as Buster and 10.4 (/srv won't be formatted, so we don't have to rebuild that host).
Any idea what the roll back plan would look like if it turns out that something about 10.4 and multisource do not work well together? Would it be less risky to do labsdb1012 first and see how it works there?
Bryan