The <wikidb>.analytics.db.svc.eqiad.wmflabs database servers have been experiencing some stability issues in the last two to three weeks that we have reason to believe are related to query volume. The DBA team at the Wikimedia Foundation is looking into various changes that may help with these problems including software upgrades for our MariaDB deployments.

Today we took an initial step of reducing the maximum time allowed for a query to complete on the <wikidb>.analytics.db.svc.eqiad.wmflabs hosts to 1 hour. We were using an upper limit of 4 hours previously. Our hope is that this change will relieve some stress on the shared servers and allow us more time to look into other changes to restore stability. Ideally we will be able to increase the limit again after making other changes to these systems.

Bryan, on behalf of the Cloud Services team