- Cloud-admin - lists.wikimedia.org

Cloud[labs] network and policy shared practice [first pass]
by Chase Pettet 13 Jul '18

13 Jul '18

Since I will be spending my time soon working most directly on things other than Cloud Services goals I have attempted to persist my understanding of shared/best practice to a wiki page. This is less a declaration than it is an attempt to formalize current expectations as I understand them. I know this is imperfect but the lack of a page like this I think has been making life difficult for awhile and we gotta start somewhere. This is mostly as-is but there are sprinklings of hope-to-be with links to tasks where changes are in progress or being tracked. https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/NetworkandPolicy -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 12 Jul '18

12 Jul '18

2018-07-11 20:00:02,778 INFO force is enabled 2018-07-11 20:00:02,830 INFO removing misc-project-backup 2018-07-11 20:00:02,922 INFO removing misc-project-backup 2018-07-11 20:00:03,521 INFO creating misc-project-backup at 2T 2018-07-11 20:00:04,552 INFO force is enabled 2018-07-11 20:00:04,587 INFO removing misc-snap 2018-07-11 20:00:04,656 INFO removing misc-snap 2018-07-11 20:00:05,060 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 11 Jul '18

11 Jul '18

2018-07-10 20:00:02,797 INFO force is enabled 2018-07-10 20:00:02,825 INFO removing tools-project-backup 2018-07-10 20:00:02,867 INFO removing tools-project-backup 2018-07-10 20:00:03,410 INFO creating tools-project-backup at 2T 2018-07-10 20:00:04,392 INFO force is enabled 2018-07-10 20:00:04,444 INFO removing tools-snap 2018-07-10 20:00:04,511 INFO removing tools-snap 2018-07-10 20:00:06,795 INFO creating tools-snap at 1T

1 0

Re: [Cloud-admin] heads up--we killed a copy job
by Chase Pettet 09 Jul '18

09 Jul '18

(I'm not sure if Brooke already had a chance to respond) Right now things are in a bit of a failure mode for dumps hosting which may be complicating the issue, it's hard to say what is sane at the moment. If you could a) drop us a note in #wikimedia-cloud when you are going to kick off grabbing the files, and b) download from the web interface. That could be the sanest course of action. In general nothing going on should have been a problem, but we have a situation on our hands otherwise that is probably complicating issues. Best, Chase > Begin forwarded message: > > *From: *Stas Malyshev <smalyshev(a)wikimedia.org> > *Subject: **Re: heads up--we killed a copy job* > *Date: *July 8, 2018 at 1:14:05 PM MST > *To: *Brooke Storm <bstorm(a)wikimedia.org> > > Hi! > > On 7/8/18 1:00 PM, Brooke Storm wrote: > > Hello Stas, > We killed your cp job on wdqs-test.wikidata-query.eqiad.wmflabs today at > 19:30-ish UTC. We wanted to give you a heads up. Somehow, it was > causing huge load on the server, which we are trying to fix with some > new traffic shaping settings because that really shouldn’t cause that. > > Sure. I was planning to do some experiments with dump > processing/compression, which involve moving some large amounts of data > around. Most of these should happen on the wdqs-test, locally, but I > needed the initial file of course. Please tell me if you have any > objections to that - i.e. if I should postpone this, or not do it on > labs at all, or any other limitation? Is there a best way/place to run > such things? I can download the initial file from web storage instead of > using nfs if that's a problem. > > Thanks, > -- > Stas Malyshev > smalyshev(a)wikimedia.org > > > -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Analytics plans for FY18/19 related to Cloud Services
by Bryan Davis 07 Jul '18

07 Jul '18

I met with Nuria and some of her folks a couple of weeks ago to sync up on things that the Analytics team is planning on for FY18/19 that have specific overlap with the Cloud Services environment. The TL;DR is that they are planning on provisioning servers to expose a some of their Data Lake (<https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake>) data to the public via Cloud Services. There will be some help that they need in rolling this out, but the expectation is that most of the work will be handled by their team. Here are my raw notes from that meeting: === Public data lake === * wiki replicas not good for research due to lack of normalization * data lake provides a normalized data set ** current data lake system is a bunch of hive tables ** looking at presto + sql layer or similar tech for cloud exposure ** dataset is all public * first publish the data on new bare metal hardware * next work on adapting a tool (quarry?) to be able to get to that data easier * had some data quality issues that kept this from happening in sooner * Start in Q2; hope that the hard work has already been done * Aim for a Q3 launch * Bryan brought up possibility of wanting named user accounts for tracking who is using the service === Asks from others to analytics ==== * folks from Audiences asking for data sooner, but that data needs to be built from public replicas ** replicas get so busy that building the data set takes longer ** More wiki replica hardware might help, but if not reserved capacity then all other usage would grow to fill available compute power ** no plan to fix this yet, but aware of the problem === Future plans === * 3 node Kafka cluster for eventstream data Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 05 Jul '18

05 Jul '18

2018-07-04 20:00:02,385 INFO force is enabled 2018-07-04 20:00:02,446 INFO removing misc-project-backup 2018-07-04 20:00:02,554 INFO removing misc-project-backup 2018-07-04 20:00:03,044 INFO creating misc-project-backup at 2T 2018-07-04 20:00:04,148 INFO force is enabled 2018-07-04 20:00:04,203 INFO removing misc-snap 2018-07-04 20:00:04,282 INFO removing misc-snap 2018-07-04 20:00:04,755 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 04 Jul '18

04 Jul '18

2018-07-03 20:00:03,213 INFO force is enabled 2018-07-03 20:00:03,258 INFO removing tools-project-backup 2018-07-03 20:00:03,359 INFO removing tools-project-backup 2018-07-03 20:00:03,956 INFO creating tools-project-backup at 2T 2018-07-03 20:00:04,921 INFO force is enabled 2018-07-03 20:00:04,984 INFO removing tools-snap 2018-07-03 20:00:05,046 INFO removing tools-snap 2018-07-03 20:00:06,892 INFO creating tools-snap at 1T

1 0

SRE meeting 2018-07-02
by Arturo Borrero Gonzalez 04 Jul '18

04 Jul '18

I attended the SRE meeting yesterday as part of my clinic-duty/on-call week. * Hiring: 2 people on the final stage of the hiring pipeline. I can give more details. * Running a diversity and inclusion survey (I'm not sure by who) * A reminder of expense reports, mind the new FY! * Phab vandalism: discussion of measures, releng is taking care for now. * Summary of the dumps incident by Ariel. * Q1 goals were a big chunk of the meeting. Nothing seems to have a direct WMCS dependency or actionable in our side. Some goals were discussed, in many cases they were already mentioned in Prague: * PHP7 & mediawiki * K8s (aka. deployment pipeline) * DB backups improvements * Netbox vs racktables * Central certificate management for LE * Increasing networking capacity * Datacenter switchover. Some concerns: it requires a lot of time, coordination, commitment... Further discussion is probably required to choose concrete goals, as Mark pointed out that doing all of them is probably too much work.

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 28 Jun '18

28 Jun '18

2018-06-27 20:00:03,212 INFO force is enabled 2018-06-27 20:00:03,262 INFO removing misc-project-backup 2018-06-27 20:00:03,371 INFO removing misc-project-backup 2018-06-27 20:00:03,819 INFO creating misc-project-backup at 2T 2018-06-27 20:00:04,751 INFO force is enabled 2018-06-27 20:00:04,801 INFO removing misc-snap 2018-06-27 20:00:04,844 INFO removing misc-snap 2018-06-27 20:00:05,148 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 27 Jun '18

27 Jun '18

2018-06-26 20:00:03,103 INFO force is enabled 2018-06-26 20:00:03,155 INFO removing tools-project-backup 2018-06-26 20:00:03,254 INFO removing tools-project-backup 2018-06-26 20:00:03,810 INFO creating tools-project-backup at 2T 2018-06-26 20:00:04,653 INFO force is enabled 2018-06-26 20:00:04,693 INFO removing tools-snap 2018-06-26 20:00:04,755 INFO removing tools-snap 2018-06-26 20:00:06,585 INFO creating tools-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin