2024

2023

2022

2021

2020

2019

2018

2017

List overview
Download

Cloud-admin September 2020

cloud-admin@lists.wikimedia.org

1 participants
1 discussions

Start a nNew thread

s4 on labsdb on the analytics role.
by Manuel Arostegui 07 Oct '20

07 Oct '20

Hello, This is a heads up about the current situation with s4 (commons) and labsdb. There's been more activity lately on s4, and that had made labsdb1011 (analytics role) start lagging behind. https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I have tried to ease its weight a couple of days ago, to help it recovering: https://gerrit.wikimedia.org/r/c/operations/puppet/+/630392 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630531 https://gerrit.wikimedia.org/r/c/operations/puppet/+/630770 The last change has (as sort of expected) made labsdb1010 lag: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=6&orgId=1&from=no… I am going to decrease the pt-kill query time from 3600 to 1800 to see if that helps labsdb1010 to guard the fort a bit. There's not much else we can do at the moment, but just keep all these issues in mind if people complain about lag on s4 (commons) on the analytics role. The web role is doing fine (labsdb1009 isn't lagging). Manuel.

2 6