August 2018 - Cloud-admin - lists.wikimedia.org

Grants and views creationg bug update
by Manuel Arostegui 17 Jun '19

17 Jun '19

Hello, Pretty much everyone who's dealt with creating views for new wikis on the labs hosts have experienced issues with "Access denied" sometimes. This was usually due to the MariaDB grant role being missed. We tried to workaround this by including the grant addition on the maintain-views script. Unfortunately, we ran into very weird problems when doing so and this is an example: https://phabricator.wikimedia.org/T193187#4273281 After lots of back and forth we decided to create a bug to MariaDB ( https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB yesterday and pointed to a similar issue ( https://jira.mariadb.org/browse/MDEV-14732). The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of us. So, for now, the workaround before adding new views is to manually add the GRANT on the DB and then run the script: GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser'; Hopefully with this email everyone is on the same page now. Thanks everyone (specially Brooke for helping me out with the troubleshooting!) Manuel.

1 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 16 Aug '18

16 Aug '18

2018-08-15 20:00:02,328 INFO force is enabled 2018-08-15 20:00:02,394 INFO removing misc-project-backup 2018-08-15 20:00:02,511 INFO removing misc-project-backup 2018-08-15 20:00:02,991 INFO creating misc-project-backup at 2T 2018-08-15 20:00:04,013 INFO force is enabled 2018-08-15 20:00:04,047 INFO removing misc-snap 2018-08-15 20:00:04,090 INFO removing misc-snap 2018-08-15 20:00:04,483 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 15 Aug '18

15 Aug '18

2018-08-14 20:00:02,818 INFO force is enabled 2018-08-14 20:00:02,868 INFO removing tools-project-backup 2018-08-14 20:00:02,951 INFO removing tools-project-backup 2018-08-14 20:00:03,589 INFO creating tools-project-backup at 2T 2018-08-14 20:00:04,565 INFO force is enabled 2018-08-14 20:00:04,623 INFO removing tools-snap 2018-08-14 20:00:04,662 INFO removing tools-snap 2018-08-14 20:00:06,198 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 09 Aug '18

09 Aug '18

2018-08-08 20:00:03,064 INFO force is enabled 2018-08-08 20:00:03,111 INFO removing misc-project-backup 2018-08-08 20:00:03,194 INFO removing misc-project-backup 2018-08-08 20:00:03,636 INFO creating misc-project-backup at 2T 2018-08-08 20:00:04,620 INFO force is enabled 2018-08-08 20:00:04,687 INFO removing misc-snap 2018-08-08 20:00:04,744 INFO removing misc-snap 2018-08-08 20:00:05,166 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 08 Aug '18

08 Aug '18

2018-08-07 20:00:03,018 INFO force is enabled 2018-08-07 20:00:03,036 INFO removing tools-project-backup 2018-08-07 20:00:03,093 INFO removing tools-project-backup 2018-08-07 20:00:03,662 INFO creating tools-project-backup at 2T 2018-08-07 20:00:04,592 INFO force is enabled 2018-08-07 20:00:04,640 INFO removing tools-snap 2018-08-07 20:00:04,678 INFO removing tools-snap 2018-08-07 20:00:07,181 INFO creating tools-snap at 1T

1 0

clinic-duty summary
by Brooke Storm 07 Aug '18

07 Aug '18

For the preceding week, things were relatively quiet. * Two new wikis were created and needed setup on the replicas (satwiki and zhwikiversity). * Efforts to build dumps torrents caused very high load on labstore1007 * Some pages came in from a puppet issue, which appears to be on the production puppet server. From the SRE meeting: * Jaime proposed standardization around monitoring naming. https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503 <https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503> * prometheus-logstash-exporter is packaged and rolling out * Most other things don’t directly affect cloud, yet. (progress on datacenter switch over, php7 work moving forward, etc) Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_

1 0

Serving grafana through cache_text
by Emanuele Rocca 06 Aug '18

06 Aug '18

Hey there, we've just pushed the DNS changes required to serve grafana, grafana-labs, and grafana-labs-admin through cache_text instead of cache_misc (see T164609 for context). Please do let Traffic know if you notice anything strange while using grafana. Thanks, ema

1 0

2018 purge
by Andrew Bogott 04 Aug '18

04 Aug '18

In anticipation of the upcoming eqiad1 migration, I propose we have another opt-in VPS project purge. I've made a dump of all projects and instances with a proposed purge timeline here: https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge I'm running this by all of you before I send anything to cloud-announce in case you have suggestions for making this a smoother process and/or think my proposed timeline is too draconian. Thanks! -A

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 02 Aug '18

02 Aug '18

2018-08-01 20:00:02,755 INFO force is enabled 2018-08-01 20:00:02,819 INFO removing misc-project-backup 2018-08-01 20:00:02,926 INFO removing misc-project-backup 2018-08-01 20:00:03,342 INFO creating misc-project-backup at 2T 2018-08-01 20:00:04,407 INFO force is enabled 2018-08-01 20:00:04,507 INFO removing misc-snap 2018-08-01 20:00:04,656 INFO removing misc-snap 2018-08-01 20:00:05,114 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 01 Aug '18

01 Aug '18

2018-07-31 20:00:02,798 INFO force is enabled 2018-07-31 20:00:02,832 INFO removing tools-project-backup 2018-07-31 20:00:02,877 INFO removing tools-project-backup 2018-07-31 20:00:03,514 INFO creating tools-project-backup at 2T 2018-07-31 20:00:04,534 INFO force is enabled 2018-07-31 20:00:04,642 INFO removing tools-snap 2018-07-31 20:00:04,693 INFO removing tools-snap 2018-07-31 20:00:06,475 INFO creating tools-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin August 2018