As discussed previously in this list [1] and on phabricator [2], I've
just removed the Ubuntu Trusty image as a default option when creating
new VMs. This is part of a longterm foundation-wide process to
standardize on Debian as the distribution of choice.
Existing Trusty VMs are unaffected by this change, as are present
ToolForge workflows. WMCS operators still have the ability to create
Trusty VMs in a pinch, so if you need one please create a phabricator
task with an explanation of what you need and why and we'll create it as
soon as we're able.
-Andrew
[1] https://lists.wikimedia.org/pipermail/cloud/2017-October/000056.html
[2] https://phabricator.wikimedia.org/T161899
On Thu, Nov 2, 2017 at 6:13 PM, Maximilian Doerr
<maximilian.doerr(a)gmail.com> wrote:
> Can you provide a list of tools/users impacted by the drive failure? Or is there a redundant drive covering for this?
As long as c1 stays up, <https://tools.wmflabs.org/tool-db-usage/>
will show the users with user-owned databases there. These users
should have all also received a MassMessage spam from me on their
Wikitech talk page about a week ago.
There is no drive or data redundancy for user-created tables on
c1.labsdb or c3.labsdb. The tools.db.svc.eqiad.wmflabs databases
however are replicated to a secondary server. See
<https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups…>
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
TL;DR:
* c1.labsdb (labsdb1001.eqiad.wmnet) is down due to hardware issues
* *.labsdb are pointing to c3.labsdb (labsdb1003.eqiad.wmnet)
The physical server behind c1.labsdb (labsdb1001.eqiad.wmnet)
experienced a hard drive failure around 2017-11-01T03:30 UTC. This
failure is preventing the MySQL service on that host from starting.
The *.labsdb service names that were pointed at that server have been
updated to point to c3.labsdb (labsdb1003.eqiad.wmnet) instead.
See <https://phabricator.wikimedia.org/T179464> for more information
and additional updates.
Expect slower than normal performance as all traffic is handled by a
single server. Now would be a great time to update the configuration
for your tools to use the new database cluster [0][1].
[0]: https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_serve…
[1]: https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855