July 2018 - Cloud-admin - lists.wikimedia.org

Grants and views creationg bug update
by Manuel Arostegui 17 Jun '19

17 Jun '19

Hello, Pretty much everyone who's dealt with creating views for new wikis on the labs hosts have experienced issues with "Access denied" sometimes. This was usually due to the MariaDB grant role being missed. We tried to workaround this by including the grant addition on the maintain-views script. Unfortunately, we ran into very weird problems when doing so and this is an example: https://phabricator.wikimedia.org/T193187#4273281 After lots of back and forth we decided to create a bug to MariaDB ( https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB yesterday and pointed to a similar issue ( https://jira.mariadb.org/browse/MDEV-14732). The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of us. So, for now, the workaround before adding new views is to manually add the GRANT on the DB and then run the script: GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser'; Hopefully with this email everyone is on the same page now. Thanks everyone (specially Brooke for helping me out with the troubleshooting!) Manuel.

1 1

Ops meeting update
by Brooke Storm 30 Jul '18

30 Jul '18

Ops meeting updates — cherry picking what seems most interesting for cloud: * Microcode with protections for SSBD uploaded to apt.wikimedia.org ** Packages upgraded on all servers which have microcode enabled ** Reboots will be coordinated separately (not sure how much impact cloud will be seeing for this) Which I presume is for: https://access.redhat.com/security/vulnerabilities/ssbd <https://access.redhat.com/security/vulnerabilities/ssbd> Also on the network side, * Interface damping is being enabled on primary links Wednesday https://phabricator.wikimedia.org/T196432 <https://phabricator.wikimedia.org/T196432> Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm(a)wikimedia.org IRC: bstorm_

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 25 Jul '18

25 Jul '18

2018-07-25 20:00:02,513 INFO force is enabled 2018-07-25 20:00:02,577 INFO removing misc-project-backup 2018-07-25 20:00:02,702 INFO removing misc-project-backup 2018-07-25 20:00:03,166 INFO creating misc-project-backup at 2T 2018-07-25 20:00:04,139 INFO force is enabled 2018-07-25 20:00:04,189 INFO removing misc-snap 2018-07-25 20:00:04,241 INFO removing misc-snap 2018-07-25 20:00:04,907 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 25 Jul '18

25 Jul '18

2018-07-24 20:00:02,546 INFO force is enabled 2018-07-24 20:00:02,577 INFO removing tools-project-backup 2018-07-24 20:00:02,627 INFO removing tools-project-backup 2018-07-24 20:00:03,152 INFO creating tools-project-backup at 2T 2018-07-24 20:00:04,022 INFO force is enabled 2018-07-24 20:00:04,075 INFO removing tools-snap 2018-07-24 20:00:04,125 INFO removing tools-snap 2018-07-24 20:00:06,502 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 18 Jul '18

18 Jul '18

2018-07-18 20:00:03,153 INFO force is enabled 2018-07-18 20:00:03,207 INFO removing misc-project-backup 2018-07-18 20:00:03,341 INFO removing misc-project-backup 2018-07-18 20:00:03,973 INFO creating misc-project-backup at 2T 2018-07-18 20:00:04,834 INFO force is enabled 2018-07-18 20:00:04,869 INFO removing misc-snap 2018-07-18 20:00:04,913 INFO removing misc-snap 2018-07-18 20:00:05,448 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 18 Jul '18

18 Jul '18

2018-07-17 20:00:02,609 INFO force is enabled 2018-07-17 20:00:02,651 INFO removing tools-project-backup 2018-07-17 20:00:02,725 INFO removing tools-project-backup 2018-07-17 20:00:03,292 INFO creating tools-project-backup at 2T 2018-07-17 20:00:04,122 INFO force is enabled 2018-07-17 20:00:04,179 INFO removing tools-snap 2018-07-17 20:00:04,251 INFO removing tools-snap 2018-07-17 20:00:06,387 INFO creating tools-snap at 1T

1 0

Cloud[labs] network and policy shared practice [first pass]
by Chase Pettet 12 Jul '18

12 Jul '18

Since I will be spending my time soon working most directly on things other than Cloud Services goals I have attempted to persist my understanding of shared/best practice to a wiki page. This is less a declaration than it is an attempt to formalize current expectations as I understand them. I know this is imperfect but the lack of a page like this I think has been making life difficult for awhile and we gotta start somewhere. This is mostly as-is but there are sprinklings of hope-to-be with links to tasks where changes are in progress or being tracked. https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/NetworkandPolicy -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 11 Jul '18

11 Jul '18

2018-07-11 20:00:02,778 INFO force is enabled 2018-07-11 20:00:02,830 INFO removing misc-project-backup 2018-07-11 20:00:02,922 INFO removing misc-project-backup 2018-07-11 20:00:03,521 INFO creating misc-project-backup at 2T 2018-07-11 20:00:04,552 INFO force is enabled 2018-07-11 20:00:04,587 INFO removing misc-snap 2018-07-11 20:00:04,656 INFO removing misc-snap 2018-07-11 20:00:05,060 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 11 Jul '18

11 Jul '18

2018-07-10 20:00:02,797 INFO force is enabled 2018-07-10 20:00:02,825 INFO removing tools-project-backup 2018-07-10 20:00:02,867 INFO removing tools-project-backup 2018-07-10 20:00:03,410 INFO creating tools-project-backup at 2T 2018-07-10 20:00:04,392 INFO force is enabled 2018-07-10 20:00:04,444 INFO removing tools-snap 2018-07-10 20:00:04,511 INFO removing tools-snap 2018-07-10 20:00:06,795 INFO creating tools-snap at 1T

1 0

Re: [Cloud-admin] heads up--we killed a copy job
by Chase Pettet 09 Jul '18

09 Jul '18

(I'm not sure if Brooke already had a chance to respond) Right now things are in a bit of a failure mode for dumps hosting which may be complicating the issue, it's hard to say what is sane at the moment. If you could a) drop us a note in #wikimedia-cloud when you are going to kick off grabbing the files, and b) download from the web interface. That could be the sanest course of action. In general nothing going on should have been a problem, but we have a situation on our hands otherwise that is probably complicating issues. Best, Chase > Begin forwarded message: > > *From: *Stas Malyshev <smalyshev(a)wikimedia.org> > *Subject: **Re: heads up--we killed a copy job* > *Date: *July 8, 2018 at 1:14:05 PM MST > *To: *Brooke Storm <bstorm(a)wikimedia.org> > > Hi! > > On 7/8/18 1:00 PM, Brooke Storm wrote: > > Hello Stas, > We killed your cp job on wdqs-test.wikidata-query.eqiad.wmflabs today at > 19:30-ish UTC. We wanted to give you a heads up. Somehow, it was > causing huge load on the server, which we are trying to fix with some > new traffic shaping settings because that really shouldn’t cause that. > > Sure. I was planning to do some experiments with dump > processing/compression, which involve moving some large amounts of data > around. Most of these should happen on the wdqs-test, locally, but I > needed the initial file of course. Please tell me if you have any > objections to that - i.e. if I should postpone this, or not do it on > labs at all, or any other limitation? Is there a best way/place to run > such things? I can download the initial file from web storage instead of > using nfs if that's a problem. > > Thanks, > -- > Stas Malyshev > smalyshev(a)wikimedia.org > > > -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin July 2018