Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the operating system that hosts the shared Toolsdb servers. This upgrade may take an hour or more, during which time the databases will not be available.
This outage will be VERY DISRUPTIVE to many toolforge tools, as all database access will fail during the upgrade. Toolforge users may want to disable tools before the outage and/or check in to verify proper recovery after service is restored.
There is likely to be a similar (or longer) outage in subsequent weeks as we also need to upgrade the database servers themselves. Toolsdb has grown to an ungainly size and can't be easily handled using standard rolling upgrade procedures; the WMCS team is in ongoing discussions about long-term solutions for this issue. In the meantime you can help us out by engaging in periodic cleanup of your database usage and dumping or dropping data that's no longer of use.
-Andrew + the WMCS team
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...
Reminder: this outage will happen today in about 90 minutes.
On 4/13/22 11:15 AM, Andrew Bogott wrote:
Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the operating system that hosts the shared Toolsdb servers. This upgrade may take an hour or more, during which time the databases will not be available.
This outage will be VERY DISRUPTIVE to many toolforge tools, as all database access will fail during the upgrade. Toolforge users may want to disable tools before the outage and/or check in to verify proper recovery after service is restored.
There is likely to be a similar (or longer) outage in subsequent weeks as we also need to upgrade the database servers themselves. Toolsdb has grown to an ungainly size and can't be easily handled using standard rolling upgrade procedures; the WMCS team is in ongoing discussions about long-term solutions for this issue. In the meantime you can help us out by engaging in periodic cleanup of your database usage and dumping or dropping data that's no longer of use.
-Andrew + the WMCS team
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...
This is starting now.
On 4/13/22 11:15 AM, Andrew Bogott wrote:
Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the operating system that hosts the shared Toolsdb servers. This upgrade may take an hour or more, during which time the databases will not be available.
This outage will be VERY DISRUPTIVE to many toolforge tools, as all database access will fail during the upgrade. Toolforge users may want to disable tools before the outage and/or check in to verify proper recovery after service is restored.
There is likely to be a similar (or longer) outage in subsequent weeks as we also need to upgrade the database servers themselves. Toolsdb has grown to an ungainly size and can't be easily handled using standard rolling upgrade procedures; the WMCS team is in ongoing discussions about long-term solutions for this issue. In the meantime you can help us out by engaging in periodic cleanup of your database usage and dumping or dropping data that's no longer of use.
-Andrew + the WMCS team
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...
This upgrade is complete and toolsdb should be available read/write now. Some tools may need comfort or restarts to deal with the outage.
Please let me know if you see any new, bad behavior with this service. And, stay tuned for future maintenance windows; this recent window revealed several fragile and out-of-date aspects that we hope to address soon.
-Andrew
On 4/19/22 10:01 AM, Andrew Bogott wrote:
This is starting now.
On 4/13/22 11:15 AM, Andrew Bogott wrote:
Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the operating system that hosts the shared Toolsdb servers. This upgrade may take an hour or more, during which time the databases will not be available.
This outage will be VERY DISRUPTIVE to many toolforge tools, as all database access will fail during the upgrade. Toolforge users may want to disable tools before the outage and/or check in to verify proper recovery after service is restored.
There is likely to be a similar (or longer) outage in subsequent weeks as we also need to upgrade the database servers themselves. Toolsdb has grown to an ungainly size and can't be easily handled using standard rolling upgrade procedures; the WMCS team is in ongoing discussions about long-term solutions for this issue. In the meantime you can help us out by engaging in periodic cleanup of your database usage and dumping or dropping data that's no longer of use.
-Andrew + the WMCS team
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...