The <wikidb>.analytics.db.svc.eqiad.wmflabs database servers have been
experiencing some stability issues in the last two to three weeks that
we have reason to believe are related to query volume. The DBA team at
the Wikimedia Foundation is looking into various changes that may help
with these problems including software upgrades for our MariaDB
deployments.
Today we took an initial step of reducing the maximum time allowed for
a query to complete on the <wikidb>.analytics.db.svc.eqiad.wmflabs
hosts to 1 hour. We were using an upper limit of 4 hours previously.
Our hope is that this change will relieve some stress on the shared
servers and allow us more time to look into other changes to restore
stability. Ideally we will be able to increase the limit again after
making other changes to these systems.
Bryan, on behalf of the Cloud Services team
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808