- Cloud-admin - lists.wikimedia.org

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 04 Apr '18

04 Apr '18

2018-04-03 20:00:02,895 INFO force is enabled 2018-04-03 20:00:02,943 INFO removing tools-project-backup 2018-04-03 20:00:03,001 INFO removing tools-project-backup 2018-04-03 20:00:03,447 INFO creating tools-project-backup at 2T 2018-04-03 20:00:04,310 INFO force is enabled 2018-04-03 20:00:04,347 INFO removing tools-snap 2018-04-03 20:00:04,396 INFO removing tools-snap 2018-04-03 20:00:06,032 INFO creating tools-snap at 1T

1 0

hiera hints for labmon
by Arturo Borrero Gonzalez 02 Apr '18

02 Apr '18

Hi folks! I'm trying to setup labmon1002 as a cold standby for labmon1001. We need to sync the whisper files from one server to another, so in case we lost labmon1001 we don't lost all metrics. Regarding hiera, in my mind it was as simpler as having 2 hiera keys (names aren't set in stone): * wmcs::monitoring::server labmon1001.eqiad.wmnet * wmcs::monitoring::server_standby labmon1002.eqiad.wmnet And then: * have all clients send data to 'wmcs::monitoring::server' * In case of outage, simple flip the keys * the rsync cronjob is in server 'wmcs::monitoring::server_standby' If you grep the ops/puppet.git repo, you may find *a lot* of calls to 'labmon1001.eqiad.wmnet'. Examples: * hieradata/common/profile/openstack/labtest.yaml profile::openstack::labtest::statsd_host: 'labmon1001.eqiad.wmnet' * hieradata/common/profile/openstack/main.yaml profile::openstack::main::statsd_host: 'labmon1001.eqiad.wmnet' * hieradata/labs/deployment-prep/common.yaml service::configuration::statsd_host: labmon1001.eqiad.wmnet * hieradata/labs/deployment-prep/common.yaml graphite_host: labmon1001.eqiad.wmnet To improve a bit maintainability, I thought of using a single hiera key, the toplevel 'wmcs::monitoring::server', so in case of an outage, we don't have to update a lot of LOCs to point to the standby server. This is, some kind of code factorization. Hiera is a new thing to me, and I've been doing some testing, test compilations and playing with tools/hiera_lookup [0]. And at the end, this doesn't seem to work because: * my new hiera keys are not found (why hieradata/labs.yaml is never read?) * some other weirdness unknown to me * isn't there a way to introduce a global hiera key for all our environment? So, would you please share some hints? What do you think about this whole picture? Do you have any suggestion for the hiera keys layout? Thanks in advance for your time! :-) Relevant phabricator tasks: * labmon1002 as cold standby for labmon1001 ** https://phabricator.wikimedia.org/T189871 * labmon: syncronize whisper files between labmon1001 and labmon1002 ** https://phabricator.wikimedia.org/T190512 [0] cmdline used are things like: % utils/hiera_lookup --fqdn=labmon1002.eqiad.wmnet --roles=labs::monitoring profile::labs::monitoring::master -v % utils/hiera_lookup --fqdn=labmon1002.eqiad.wmnet profile::labs::monitoring::master -v

2 1

Re: [Cloud-admin] ** PROBLEM alert - labstore1006/Incoming network saturation is CRITICAL **
by Chase Pettet 31 Mar '18

31 Mar '18

I silenced this as well On Fri, Mar 30, 2018, 3:01 PM <icinga(a)einsteinium.wikimedia.org> wrote: > Notification Type: PROBLEM > > Service: Incoming network saturation > Host: labstore1006 > Address: 208.80.154.7 > State: CRITICAL > > Date/Time: Fri Mar 30 20:01:08 UTC 2018 > > Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring > > Additional Info: > > CRITICAL: 22.22% of data above the critical threshold [106250000.0] >

1 0

Re: [Cloud-admin] ** PROBLEM alert - labstore1007/Incoming network saturation is CRITICAL **
by Chase Pettet 30 Mar '18

30 Mar '18

I silenced this system to Sunday as it is not in service but it seems we need to adjust thresholds for when it is? On Fri, Mar 30, 2018, 7:22 AM <icinga(a)einsteinium.wikimedia.org> wrote: > Notification Type: PROBLEM > > Service: Incoming network saturation > Host: labstore1007 > Address: 208.80.155.106 > State: CRITICAL > > Date/Time: Fri Mar 30 12:22:11 UTC 2018 > > Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring > > Additional Info: > > CRITICAL: 22.22% of data above the critical threshold [106250000.0] >

2 1

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 29 Mar '18

29 Mar '18

2018-03-28 20:00:02,503 INFO force is enabled 2018-03-28 20:00:02,578 INFO removing misc-project-backup 2018-03-28 20:00:02,711 INFO removing misc-project-backup 2018-03-28 20:00:03,268 INFO creating misc-project-backup at 2T 2018-03-28 20:00:04,159 INFO force is enabled 2018-03-28 20:00:04,195 INFO removing misc-snap 2018-03-28 20:00:04,257 INFO removing misc-snap 2018-03-28 20:00:04,805 INFO creating misc-snap at 1T

1 0

Ops meeting/clinic duty update
by Brooke Storm 27 Mar '18

27 Mar '18

Hello all, From the ops meeting yesterday: * Most important part for cloud that the DBAs will be out on the 29th and 30th, so no serious DB changes during that time. Clinic Duty update: * /tmp cleanup on tools-webgrid-lighttpd-1422 - https://phabricator.wikimedia.org/T190185 <https://phabricator.wikimedia.org/T190185> ** Will be adding tmpreaper on web nodes after messing with it a bit. * novaproxy01 filled up with map tiles nginx logs https://phabricator.wikimedia.org/T190218 <https://phabricator.wikimedia.org/T190218> ** Added new settings to logrotate, but I still need to change the cron for logrotate for them to work. * Fixed broken NFS mount on tools-exec-1408 -- Brooke Storm Operations Engineer, Cloud Services

1 0

Reminder I am away :)
by Chase Pettet 22 Mar '18

22 Mar '18

Traveling and wedding family stuff through Sunday. Also, I seem to not be getting emails from phabricator. I don't know why but it must be somewhat recent.

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 22 Mar '18

22 Mar '18

2018-03-21 20:00:03,213 INFO force is enabled 2018-03-21 20:00:03,287 INFO removing misc-project-backup 2018-03-21 20:00:03,420 INFO removing misc-project-backup 2018-03-21 20:00:04,131 INFO creating misc-project-backup at 2T 2018-03-21 20:00:05,027 INFO force is enabled 2018-03-21 20:00:05,055 INFO removing misc-snap 2018-03-21 20:00:05,102 INFO removing misc-snap 2018-03-21 20:00:05,689 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 21 Mar '18

21 Mar '18

2018-03-20 20:00:03,402 INFO force is enabled 2018-03-20 20:00:03,435 INFO removing tools-project-backup 2018-03-20 20:00:03,485 INFO removing tools-project-backup 2018-03-20 20:00:04,045 INFO creating tools-project-backup at 2T 2018-03-20 20:00:04,831 INFO force is enabled 2018-03-20 20:00:04,884 INFO removing tools-snap 2018-03-20 20:00:04,941 INFO removing tools-snap 2018-03-20 20:00:05,958 INFO creating tools-snap at 1T

1 0

[clinic-duty] weekly update
by Arturo Borrero Gonzalez 21 Mar '18

21 Mar '18

Hi there! an update my clinic-duty week: * reboot tools-webgrid-lighttpd-1421, tools-webgrid-lighttpd-1415, tools-webgrid-lighttpd-1420 due to /tmp almost full * reboot tools-exec-1408 due to high load * request creation of WCDO VPS project https://phabricator.wikimedia.org/T189165 approved toolforge access: * fakejames https://toolsadmin.wikimedia.org/tools/membership/status/268 * Jim Smith https://toolsadmin.wikimedia.org/tools/membership/status/269 * Rguptawiki (rgupta) https://toolsadmin.wikimedia.org/tools/membership/status/271 * Ross Mallett (rmallett) https://toolsadmin.wikimedia.org/tools/membership/status/273 * Tigraan (tigraan) https://toolsadmin.wikimedia.org/tools/membership/status/272 declined toolforge access: * https://toolsadmin.wikimedia.org/tools/membership/status/242

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin