- Cloud-admin - lists.wikimedia.org

clinic-duty summary
by Brooke Storm 08 Aug '18

08 Aug '18

For the preceding week, things were relatively quiet. * Two new wikis were created and needed setup on the replicas (satwiki and zhwikiversity). * Efforts to build dumps torrents caused very high load on labstore1007 * Some pages came in from a puppet issue, which appears to be on the production puppet server. From the SRE meeting: * Jaime proposed standardization around monitoring naming. https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503 <https://gerrit.wikimedia.org/r/c/operations/puppet/+/448503> * prometheus-logstash-exporter is packaged and rolling out * Most other things don’t directly affect cloud, yet. (progress on datacenter switch over, php7 work moving forward, etc) Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_

1 0

Serving grafana through cache_text
by Emanuele Rocca 06 Aug '18

06 Aug '18

Hey there, we've just pushed the DNS changes required to serve grafana, grafana-labs, and grafana-labs-admin through cache_text instead of cache_misc (see T164609 for context). Please do let Traffic know if you notice anything strange while using grafana. Thanks, ema

1 0

2018 purge
by Andrew Bogott 05 Aug '18

05 Aug '18

In anticipation of the upcoming eqiad1 migration, I propose we have another opt-in VPS project purge. I've made a dump of all projects and instances with a proposed purge timeline here: https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge I'm running this by all of you before I send anything to cloud-announce in case you have suggestions for making this a smoother process and/or think my proposed timeline is too draconian. Thanks! -A

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 02 Aug '18

02 Aug '18

2018-08-01 20:00:02,755 INFO force is enabled 2018-08-01 20:00:02,819 INFO removing misc-project-backup 2018-08-01 20:00:02,926 INFO removing misc-project-backup 2018-08-01 20:00:03,342 INFO creating misc-project-backup at 2T 2018-08-01 20:00:04,407 INFO force is enabled 2018-08-01 20:00:04,507 INFO removing misc-snap 2018-08-01 20:00:04,656 INFO removing misc-snap 2018-08-01 20:00:05,114 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 01 Aug '18

01 Aug '18

2018-07-31 20:00:02,798 INFO force is enabled 2018-07-31 20:00:02,832 INFO removing tools-project-backup 2018-07-31 20:00:02,877 INFO removing tools-project-backup 2018-07-31 20:00:03,514 INFO creating tools-project-backup at 2T 2018-07-31 20:00:04,534 INFO force is enabled 2018-07-31 20:00:04,642 INFO removing tools-snap 2018-07-31 20:00:04,693 INFO removing tools-snap 2018-07-31 20:00:06,475 INFO creating tools-snap at 1T

1 0

Ops meeting update
by Brooke Storm 31 Jul '18

31 Jul '18

Ops meeting updates — cherry picking what seems most interesting for cloud: * Microcode with protections for SSBD uploaded to apt.wikimedia.org ** Packages upgraded on all servers which have microcode enabled ** Reboots will be coordinated separately (not sure how much impact cloud will be seeing for this) Which I presume is for: https://access.redhat.com/security/vulnerabilities/ssbd <https://access.redhat.com/security/vulnerabilities/ssbd> Also on the network side, * Interface damping is being enabled on primary links Wednesday https://phabricator.wikimedia.org/T196432 <https://phabricator.wikimedia.org/T196432> Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 26 Jul '18

26 Jul '18

2018-07-25 20:00:02,513 INFO force is enabled 2018-07-25 20:00:02,577 INFO removing misc-project-backup 2018-07-25 20:00:02,702 INFO removing misc-project-backup 2018-07-25 20:00:03,166 INFO creating misc-project-backup at 2T 2018-07-25 20:00:04,139 INFO force is enabled 2018-07-25 20:00:04,189 INFO removing misc-snap 2018-07-25 20:00:04,241 INFO removing misc-snap 2018-07-25 20:00:04,907 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 25 Jul '18

25 Jul '18

2018-07-24 20:00:02,546 INFO force is enabled 2018-07-24 20:00:02,577 INFO removing tools-project-backup 2018-07-24 20:00:02,627 INFO removing tools-project-backup 2018-07-24 20:00:03,152 INFO creating tools-project-backup at 2T 2018-07-24 20:00:04,022 INFO force is enabled 2018-07-24 20:00:04,075 INFO removing tools-snap 2018-07-24 20:00:04,125 INFO removing tools-snap 2018-07-24 20:00:06,502 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 19 Jul '18

19 Jul '18

2018-07-18 20:00:03,153 INFO force is enabled 2018-07-18 20:00:03,207 INFO removing misc-project-backup 2018-07-18 20:00:03,341 INFO removing misc-project-backup 2018-07-18 20:00:03,973 INFO creating misc-project-backup at 2T 2018-07-18 20:00:04,834 INFO force is enabled 2018-07-18 20:00:04,869 INFO removing misc-snap 2018-07-18 20:00:04,913 INFO removing misc-snap 2018-07-18 20:00:05,448 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 18 Jul '18

18 Jul '18

2018-07-17 20:00:02,609 INFO force is enabled 2018-07-17 20:00:02,651 INFO removing tools-project-backup 2018-07-17 20:00:02,725 INFO removing tools-project-backup 2018-07-17 20:00:03,292 INFO creating tools-project-backup at 2T 2018-07-17 20:00:04,122 INFO force is enabled 2018-07-17 20:00:04,179 INFO removing tools-snap 2018-07-17 20:00:04,251 INFO removing tools-snap 2018-07-17 20:00:06,387 INFO creating tools-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin