- Cloud-admin - lists.wikimedia.org

Toolforge: new mail relay
by Giovanni Tirloni 14 Nov '18

14 Nov '18

Hello, A new mail relay (tools-mail-02) based on Stretch was deployed today (T208579). It replaces the old tools-mail server which was running on Trusty. The new server adds TLS support (using Let's Encrypt certificates), which wasn't available in tools-mail before. Please let us know if you notice any regressions. Regards, -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

Proposal to have wikibugs bot only in #wikimedia-cloud-feed
by Giovanni Tirloni 31 Oct '18

31 Oct '18

Hello, Today the wikibugs bot sends patches to #wikimedia-cloud and bugs to #wikimedia-cloud-feed. I would like to propose that we only have the wikibugs bot broadcasting things on #wikimedia-cloud-feed. This would have a few benefits, in my opinion: * #wikimedia-cloud gets more space for undistracted conversations * #wikimedia-cloud-feed becomes the sole source for updates (easier mental processing for some people like me) Does anyone have any concerns with this proposal? Thank you, -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

5 5

Toolforge puppet refactor options
by Arturo Borrero 27 Oct '18

27 Oct '18

Hi, Brooke already started refactoring puppet code for toolforge-stretch, and both Giovanni and me are following with other side of the code (currently, service nodes T207591). I'm starting with the refactor myself and I have some doubts on how to better organize the code. I think it's really important we agree on how to do this beforehand, and also I may lack some knowledge on best practices etc, so I'm sharing some ideas for you to comment on. The following wikitech page contains 3 model options I think can be applied: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/puppet_refactor Brooke has probably some ideas on this, since she already started. Please comment, and forgive my ignorance if the answer to my questions are obvious. Regards. PD: CC'ing Chase, because he did the massive CloudVPS refactor in the past.

2 1

tools-mail - Deployed stricter rules against spam
by Giovanni Tirloni 29 Sep '18

29 Sep '18

Hello, I've deployed change 463611 which converts the DNSBL checks introduced in 462472 (thanks Keith!) from just warnings to actually taking action against spammers. Also enabled sender verification checks (on top of the already enabled recipient checks). This should help with bounce emails that can't get delivered (frozen in Exim's queue). I tested email delivery to/from various hosts, tool's email, watched emails going to other tools, etc. It seems to be working fine but I'm sure there will be some false positives. Infamous @qq.com spam being blocked: 2018-09-29 14:03:38 H=(0voc.cn) [122.237.40.138] F=<5281141(a)qq.com> rejected RCPT <Mailer-Daemon(a)tools.wmflabs.org>: Blocked by DNSBL (see https://www.spamhaus.org/query/ip/122.237.40.138) Sender addresses check: 2018-09-29 14:07:07 H=(ABC) [138.99.254.197] sender verify fail for <noreply(a)i.dont.exist>: Unrouteable address 2018-09-29 14:07:07 H=(ABC) [138.99.254.197] F=<noreply(a)i.dont.exist> rejected RCPT <gtirloni-sandbox.anything(a)tools.wmflabs.org>: Sender verify failed Hopefully, this decreases the amount of spam we're getting today until the new SMTP servers are in place. Please let us know if something broke due to this change. Thanks -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

Naming VMs
by Giovanni Tirloni 18 Sep '18

18 Sep '18

Hello, I want to deploy a VM instance on a project to be the test NFS server we have been talking about it. According to the server naming conventions [0], I should name this "cloudstore-XXXX-dev", right? Also, are we increasing the numbers from the latest labstore servers or starting fresh for cloudstore? Is there some other VM naming convention I should follow (or that our tenants are encouraged to follow)? Thanks 0 - https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions#Serve… -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

2 1

SRE meeting notes
by Andrew Bogott 11 Sep '18

11 Sep '18

- Lots of folks are absorbed in the DC migration test coming up on 10-10 and 10-11 . That's unlikely to affect us but it would be good for some of us to be available in case of unexpected messes. - Moritz is a bit interested in the NFS vs. Kernel issues; he's been messing with the 4.14 kernel and might want to give that a try in case it happens to resolve the issue. - Procurement people have a new scheme to get our new labvirts working (swapping in older hardware) but it's not yet a sure thing; and even if that works Moritz may have to backport some things to Jessie. - Arzhel will make the cross-region routing network changes in the next few hours. - Giovanni's root approved!

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 16 Aug '18

16 Aug '18

2018-08-15 20:00:02,328 INFO force is enabled 2018-08-15 20:00:02,394 INFO removing misc-project-backup 2018-08-15 20:00:02,511 INFO removing misc-project-backup 2018-08-15 20:00:02,991 INFO creating misc-project-backup at 2T 2018-08-15 20:00:04,013 INFO force is enabled 2018-08-15 20:00:04,047 INFO removing misc-snap 2018-08-15 20:00:04,090 INFO removing misc-snap 2018-08-15 20:00:04,483 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 15 Aug '18

15 Aug '18

2018-08-14 20:00:02,818 INFO force is enabled 2018-08-14 20:00:02,868 INFO removing tools-project-backup 2018-08-14 20:00:02,951 INFO removing tools-project-backup 2018-08-14 20:00:03,589 INFO creating tools-project-backup at 2T 2018-08-14 20:00:04,565 INFO force is enabled 2018-08-14 20:00:04,623 INFO removing tools-snap 2018-08-14 20:00:04,662 INFO removing tools-snap 2018-08-14 20:00:06,198 INFO creating tools-snap at 1T

1 0

Cron <root@labstore2004> /usr/local/sbin/block_sync 10.64.37.20 misc misc-project misc-snap backup misc-project misc-project-backup 2T
by root＠labstore2004.codfw.wmnet 09 Aug '18

09 Aug '18

2018-08-08 20:00:03,064 INFO force is enabled 2018-08-08 20:00:03,111 INFO removing misc-project-backup 2018-08-08 20:00:03,194 INFO removing misc-project-backup 2018-08-08 20:00:03,636 INFO creating misc-project-backup at 2T 2018-08-08 20:00:04,620 INFO force is enabled 2018-08-08 20:00:04,687 INFO removing misc-snap 2018-08-08 20:00:04,744 INFO removing misc-snap 2018-08-08 20:00:05,166 INFO creating misc-snap at 1T

1 0

Cron <root@labstore2003> /usr/local/sbin/block_sync 10.64.37.20 tools tools-project tools-snap backup tools-project tools-project-backup 2T
by root＠labstore2003.codfw.wmnet 08 Aug '18

08 Aug '18

2018-08-07 20:00:03,018 INFO force is enabled 2018-08-07 20:00:03,036 INFO removing tools-project-backup 2018-08-07 20:00:03,093 INFO removing tools-project-backup 2018-08-07 20:00:03,662 INFO creating tools-project-backup at 2T 2018-08-07 20:00:04,592 INFO force is enabled 2018-08-07 20:00:04,640 INFO removing tools-snap 2018-08-07 20:00:04,678 INFO removing tools-snap 2018-08-07 20:00:07,181 INFO creating tools-snap at 1T

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin