Because bad things come in threes (I'm hoping it's threes and not sevens) the server that hosts toolsdb is now also misbehaving. Brooke just now disabled a troubled drive which may have resolved things, but if the last few hours are any indication then the vast majority of connection or query attempts are likely to fail until we have a better solution in place.
We're working on multiple fronts, trying to diagnose and fix the primary issue while also working to get new hardware online as a possible replacement server. Neither of those things are likely to get done until tomorrow, though, so toolforge will be in pretty bad shape in the meantime.
It has been a rough couple of days, but rest assured we're taking notes about how to prevent outages like these in the future. Thank you for your patience in the meantime!
-Andrew + the cloud team
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Thanks for the updates and for trying to extinguish the fires.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
On Fri, Feb 15, 2019 at 4:59 AM Andrew Bogott abogott@wikimedia.org wrote:
Because bad things come in threes (I'm hoping it's threes and not sevens) the server that hosts toolsdb is now also misbehaving. Brooke just now disabled a troubled drive which may have resolved things, but if the last few hours are any indication then the vast majority of connection or query attempts are likely to fail until we have a better solution in place.
We're working on multiple fronts, trying to diagnose and fix the primary issue while also working to get new hardware online as a possible replacement server. Neither of those things are likely to get done until tomorrow, though, so toolforge will be in pretty bad shape in the meantime.
It has been a rough couple of days, but rest assured we're taking notes about how to prevent outages like these in the future. Thank you for your patience in the meantime!
-Andrew + the cloud team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
I'm assuming things are still broken? Let us know when all is well, and good luck!
Arthur
On Fri, Feb 15, 2019 at 2:28 AM Pine W wiki.pine@gmail.com wrote:
Thanks for the updates and for trying to extinguish the fires.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
On Fri, Feb 15, 2019 at 4:59 AM Andrew Bogott abogott@wikimedia.org wrote:
Because bad things come in threes (I'm hoping it's threes and not sevens) the server that hosts toolsdb is now also misbehaving. Brooke just now disabled a troubled drive which may have resolved things, but if the last few hours are any indication then the vast majority of connection or query attempts are likely to fail until we have a better solution in place.
We're working on multiple fronts, trying to diagnose and fix the primary issue while also working to get new hardware online as a possible replacement server. Neither of those things are likely to get done until tomorrow, though, so toolforge will be in pretty bad shape in the meantime.
It has been a rough couple of days, but rest assured we're taking notes about how to prevent outages like these in the future. Thank you for your patience in the meantime!
-Andrew + the cloud team
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud