Hi everybody,
the Analytics team has been working with the SRE Data Persistence team during the last months to replace dbstore1002 with three brand new nodes, dbstore100[3-5]. We are moving from a single mysql instance (multi-source) to a multi-instance environment.
For more info please check: * T210478 and related subtasks. * https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas
We are planning to decommission the dbstore1002 host (namely stopping mysql and shutting down the server) on Monday March 4th (EU morning). We have recently been following up with a lot of users to help them migrate to the new environment, so we are reasonably sure that this move should not heavily impact anybody, but if we have left some use case aside please let us know in https://phabricator.wikimedia.org/T215589. If we don't hear anything before the March 4th deadline we'll proceed with the host decommission maintenance.
Luca (on behalf of the Analytics team)
On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano ltoscano@wikimedia.org wrote:
the Analytics team has been working with the SRE Data Persistence team during the last months to replace dbstore1002 with three brand new nodes, dbstore100[3-5]. We are moving from a single mysql instance (multi-source) to a multi-instance environment.
This has been an incredible amount of work, both in socializing the idea and also implementation and making sure workflows don't break as much as possible. Thank you to all of you who worked on this over the past months and to those of you who maintained the single mysql instance for all the past years. I hope that the maintenance workflows become easier for those of you who continue to maintain these systems for us.
Thank you! :)
Best, Leila
+1 to everything Leila said. This is very difficult work and I'm amazed to see it happen so smoothly -- at least from my point of view :)
On Fri, Feb 22, 2019 at 10:07 AM Leila Zia leila@wikimedia.org wrote:
On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano ltoscano@wikimedia.org wrote:
the Analytics team has been working with the SRE Data Persistence team
during the last months to replace dbstore1002 with three brand new nodes, dbstore100[3-5]. We are moving from a single mysql instance (multi-source) to a multi-instance environment.
This has been an incredible amount of work, both in socializing the idea and also implementation and making sure workflows don't break as much as possible. Thank you to all of you who worked on this over the past months and to those of you who maintained the single mysql instance for all the past years. I hope that the maintenance workflows become easier for those of you who continue to maintain these systems for us.
Thank you! :)
Best, Leila
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Let me join in with a late appreciation of all the work that Analytics has done making this migration happen.
As probably the heaviest users of the MediaWiki replicas, we in the Product Analytics team were a bit worried when we heard the initial plan and had lot of requests for how to make the new infrastructure usable. Analytics listened to us in meeting after meeting and in long Phab conversations and worked *hard *to come up with creative, feasible solutions for our challenges, even as dbstore1002 slowly failed under them.
In particular, Luca did amazing work during this process: tracking all five of us analysts down during the mayhem of all hands to see how we worked with dbstore1002, coming up with smart hacks with SRV records and command-line clients to make it easier for us to work across the multiple replacement hosts, exhaustively documenting the new setup https://wikitech.wikimedia.org/w/index.php?title=Analytics/Data_access#MariaDB_replicas, and more.
Thank you so much to all of you for providing the infrastructure that literally makes our jobs possible! 🎉
On Fri, 22 Feb 2019 at 21:52, Aaron Halfaker ahalfaker@wikimedia.org wrote:
+1 to everything Leila said. This is very difficult work and I'm amazed to see it happen so smoothly -- at least from my point of view :)
On Fri, Feb 22, 2019 at 10:07 AM Leila Zia leila@wikimedia.org wrote:
On Fri, Feb 22, 2019 at 2:45 AM Luca Toscano ltoscano@wikimedia.org wrote:
the Analytics team has been working with the SRE Data Persistence team
during the last months to replace dbstore1002 with three brand new nodes, dbstore100[3-5]. We are moving from a single mysql instance (multi-source) to a multi-instance environment.
This has been an incredible amount of work, both in socializing the idea and also implementation and making sure workflows don't break as much as possible. Thank you to all of you who worked on this over the past months and to those of you who maintained the single mysql instance for all the past years. I hope that the maintenance workflows become easier for those of you who continue to maintain these systems for us.
Thank you! :)
Best, Leila
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
--
Aaron Halfaker
Principal Research Scientist
Head of the Scoring Platform team Wikimedia Foundation