Cloud-admin November 2018

cloud-admin@lists.wikimedia.org

2 participants
8 discussions

Grants and views creationg bug update
by Manuel Arostegui 17 Jun '19

17 Jun '19

Hello, Pretty much everyone who's dealt with creating views for new wikis on the labs hosts have experienced issues with "Access denied" sometimes. This was usually due to the MariaDB grant role being missed. We tried to workaround this by including the grant addition on the maintain-views script. Unfortunately, we ran into very weird problems when doing so and this is an example: https://phabricator.wikimedia.org/T193187#4273281 After lots of back and forth we decided to create a bug to MariaDB ( https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB yesterday and pointed to a similar issue ( https://jira.mariadb.org/browse/MDEV-14732). The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of us. So, for now, the workaround before adding new views is to manually add the GRANT on the DB and then run the script: GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser'; Hopefully with this email everyone is on the same page now. Thanks everyone (specially Brooke for helping me out with the troubleshooting!) Manuel.

1 1

Daily Puppet alerts are back (T210432)
by Giovanni Tirloni 26 Nov '18

26 Nov '18

Just a heads up that tomorrow at the usual 8:15 time, project members may start to receive emails about their Cloud VPS instances that have Puppet failures. https://gerrit.wikimedia.org/r/c/operations/puppet/+/475875 -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

Fwd: Cron <root@labcontrol1001> /usr/bin/mysql keystone -hm5-master.eqiad.wmnet -ukeystone -pauheeri5eiQuoo5 -e 'DELETE FROM token WHERE user_id="novaobserver" AND NOW() + INTERVAL 7 day > expires LIMIT 10000; '
by Chase Pettet 24 Nov '18

24 Nov '18

Fyi ---------- Forwarded message --------- From: Cron Daemon <root(a)labcontrol1001.wikimedia.org> Date: Sat, Nov 24, 2018, 7:30 AM Subject: Cron <root@labcontrol1001> /usr/bin/mysql keystone -hm5-master.eqiad.wmnet -ukeystone -pauheeri5eiQuoo5 -e 'DELETE FROM token WHERE user_id="novaobserver" AND NOW() + INTERVAL 7 day > expires LIMIT 10000;' To: <root(a)labcontrol1001.wikimedia.org> ERROR 1040 (08004): Too many connections

1 0

New debian-9.6-stretch image
by Giovanni Tirloni 22 Nov '18

22 Nov '18

Hello, I've deployed a new image in OpenStack: debian-9.6-stretch Also reviewed the docs as I went along: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images Debian 9.6 Release Notes: https://www.debian.org/News/2018/20181110 Everything went fine in my tests but please let me know if something doesn't look right. Regards -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

Fwd: Cron <root@gtirloni-stretch-01> /usr/local/sbin/tools-clush-generator /etc/clustershell/tools.yaml --observer-pass hiera('profile::openstack::eqiad1::observer_password')
by Chase Pettet 20 Nov '18

20 Nov '18

FYI :) ---------- Forwarded message --------- From: Cron Daemon <root(a)tools.wmflabs.org> Date: Tue, Nov 20, 2018 at 7:31 AM Subject: Cron <root@gtirloni-stretch-01> /usr/local/sbin/tools-clush-generator /etc/clustershell/tools.yaml --observer-pass hiera('profile::openstack::eqiad1::observer_password') To: <root(a)tools.wmflabs.org> /bin/sh: 1: Syntax error: "(" unexpected -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Toolforge: New clushmaster
by Giovanni Tirloni 20 Nov '18

20 Nov '18

Hello, A new clushmaster host (tools-clushmaster-02) has been deployed using the refactored Puppet code and Debian Stretch (T209701). The old clushmaster host (tools-clushmaster-01) has been shut down and will be deleted in a few days if there are no issues with the new one. If you find any issues with it, please let us know. Thanks, -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

cloud-admin-feed mailing list
by Chase Pettet 15 Nov '18

15 Nov '18

I added the new icinga@ sender for hte new icinga boxes to the approved sender list here and trashed about 170 seemingly spam subscription requests today while doing other things FYI. -- Chase Pettet chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and IRC

1 0

Toolforge: new mail relay
by Giovanni Tirloni 13 Nov '18

13 Nov '18

Hello, A new mail relay (tools-mail-02) based on Stretch was deployed today (T208579). It replaces the old tools-mail server which was running on Trusty. The new server adds TLS support (using Let's Encrypt certificates), which wasn't available in tools-mail before. Please let us know if you notice any regressions. Regards, -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin November 2018