This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org mailto:bstorm@wikimedia.org IRC: bstorm_
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
The change-over was successful. Thank you for your patience and understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_... https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_and_Replication https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org mailto:bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org wrote:
This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org mailto:bstorm@wikimedia.org IRC: bstorm_
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hoi, Thank you. Working with so many people depending on a good result is stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved. Thanks, GerardM
On Mon, 18 Feb 2019 at 20:11, Brooke Storm bstorm@wikimedia.org wrote:
The change-over was successful. Thank you for your patience and understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_... https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org wrote:
This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Thanks Brooke, everything looks great to me, I really appreciate everybody's work to get this back!
Arthur
On Mon, Feb 18, 2019 at 2:39 PM Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, Thank you. Working with so many people depending on a good result is stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved. Thanks, GerardM
On Mon, 18 Feb 2019 at 20:11, Brooke Storm bstorm@wikimedia.org wrote:
The change-over was successful. Thank you for your patience and understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_... https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org wrote:
This is an update on the ongoing problems with the toolsdb service. We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Thank you guys, all my tools.labsdb dependent dashboards are running smoothly again. Thanks for the hard work!
Best, Goran
On Tue, Feb 19, 2019, 15:52 Arthur Smith arthurpsmith@gmail.com wrote:
Thanks Brooke, everything looks great to me, I really appreciate everybody's work to get this back!
Arthur
On Mon, Feb 18, 2019 at 2:39 PM Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, Thank you. Working with so many people depending on a good result is
stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved.
Thanks, GerardM
On Mon, 18 Feb 2019 at 20:11, Brooke Storm bstorm@wikimedia.org wrote:
The change-over was successful. Thank you for your patience and
understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not
replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_...
https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org wrote:
This is an update on the ongoing problems with the toolsdb service. We
are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down by
some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly
labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Thanks, all looking good!
On Tue, Feb 19, 2019 at 2:57 PM Goran Milovanovic < goran.milovanovic_ext@wikimedia.de> wrote:
Thank you guys, all my tools.labsdb dependent dashboards are running smoothly again. Thanks for the hard work!
Best, Goran
On Tue, Feb 19, 2019, 15:52 Arthur Smith arthurpsmith@gmail.com wrote:
Thanks Brooke, everything looks great to me, I really appreciate everybody's work to get this back!
Arthur
On Mon, Feb 18, 2019 at 2:39 PM Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, Thank you. Working with so many people depending on a good result is
stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved.
Thanks, GerardM
On Mon, 18 Feb 2019 at 20:11, Brooke Storm bstorm@wikimedia.org
wrote:
The change-over was successful. Thank you for your patience and
understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not
replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_...
https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org
wrote:
This is an update on the ongoing problems with the toolsdb service.
We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down
by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly
labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Thanks much! Everything works properly, it seems that a great effort has been made.
уто, 19. феб 2019. у 16:23 Magnus Manske via Cloud < cloud@lists.wikimedia.org> је написао/ла:
Thanks, all looking good!
On Tue, Feb 19, 2019 at 2:57 PM Goran Milovanovic < goran.milovanovic_ext@wikimedia.de> wrote:
Thank you guys, all my tools.labsdb dependent dashboards are running smoothly again. Thanks for the hard work!
Best, Goran
On Tue, Feb 19, 2019, 15:52 Arthur Smith arthurpsmith@gmail.com wrote:
Thanks Brooke, everything looks great to me, I really appreciate everybody's work to get this back!
Arthur
On Mon, Feb 18, 2019 at 2:39 PM Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, Thank you. Working with so many people depending on a good result is
stressful. I thank you for getting this far. I can imagine that you are not done yet but do get your rest. It is deserved.
Thanks, GerardM
On Mon, 18 Feb 2019 at 20:11, Brooke Storm bstorm@wikimedia.org
wrote:
The change-over was successful. Thank you for your patience and
understanding. ToolsDB should now be fully operational. Please report issues to the #wikimedia-cloud IRC channel.
In the course of recovery efforts, the four databases that are not
replicated were left behind. This ticket is there to track further recovery efforts on those databases as is desired or possible relating to those four databases listed here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups_...
https://phabricator.wikimedia.org/T216441
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
On Feb 18, 2019, at 9:57 AM, Brooke Storm bstorm@wikimedia.org
wrote:
This is an update on the ongoing problems with the toolsdb service.
We are preparing to move to a new server, which is now a functioning replica of the toolsdb server. The first step here is to restart the service in read-only mode, and then we will move the DNS. Expect writes to stop working and connections to drop. When we are moved to the new DNS, services that use this database will need to be restarted.
This will be happening within the next hour unless it is slowed down
by some issues or caution.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm_
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly
labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud