Cloud-admin March 2019

cloud-admin@lists.wikimedia.org

3 participants
6 discussions

Grants and views creationg bug update
by Manuel Arostegui 17 Jun '19

17 Jun '19

Hello, Pretty much everyone who's dealt with creating views for new wikis on the labs hosts have experienced issues with "Access denied" sometimes. This was usually due to the MariaDB grant role being missed. We tried to workaround this by including the grant addition on the maintain-views script. Unfortunately, we ran into very weird problems when doing so and this is an example: https://phabricator.wikimedia.org/T193187#4273281 After lots of back and forth we decided to create a bug to MariaDB ( https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB yesterday and pointed to a similar issue ( https://jira.mariadb.org/browse/MDEV-14732). The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of us. So, for now, the workaround before adding new views is to manually add the GRANT on the DB and then run the script: GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser'; Hopefully with this email everyone is on the same page now. Thanks everyone (specially Brooke for helping me out with the troubleshooting!) Manuel.

1 1

How to add NFS to a Cloud VPS project
by Giovanni Tirloni 23 Mar '19

23 Mar '19

Hello, Could someone help me add NFS access to project `paws`? I couldn't find any documentation on this. We need access to `home` and `project` shares so it can migrate out of the `tools` project. Thank you, -- Giovanni Tirloni Operations Engineer Wikimedia Foundation

1 0

Should ldap disaster strike...
by Andrew Bogott 22 Mar '19

22 Mar '19

I have just moved all tools nodes to the new read-only ldap service. This change was made via the tools project puppet panel in horizon. Everything looks fine so far, but I'm going to be AFK for much of the weekend. So, if things go pear-shaped over the weekend, here's how to revert: 1) In Horizon, go to the 'Project Puppet' tab in the 'tools' project 2) Scroll all the way to the bottom, to the 'Hiera Config' section 3) Remove these lines: labsldapconfig: hostname: ldap-ro.eqiad.wikimedia.org secondary_hostname: ldap-ro.eqiad.wikimedia.org 4) Force puppet runs as needed, or just wait a bit. Assuming things go well, eventually I'll remove the project-specific hiera hacks and just make this change cloud-wide -A

1 0

T210818 - Converted OpenStack admin cronjobs to systemd timers
by Giovanni Tirloni 21 Mar '19

21 Mar '19

Hi, Just a heads up, I've merged a change that converts OpenStack admin cronjobs to systemd timers. List all timers: ``` root@cloudcontrol1003:~# systemctl list-timers --no-pager NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2019-03-21 14:00:00 UTC 6min left Thu 2019-03-21 13:50:01 UTC 3min 31s ago designate_floating_ip_ptr_records_updater.timer designate_floating_ip_ptr_records_updater.service Thu 2019-03-21 14:15:00 UTC 21min left n/a n/a glance_rsync_images.timer glance_rsync_images.service Thu 2019-03-21 14:20:00 UTC 26min left n/a n/a keystone_delete_expired_tokens.timer keystone_delete_expired_tokens.service Thu 2019-03-21 14:30:00 UTC 36min left n/a n/a keystone_novaobserver_delete_tokens.timer keystone_novaobserver_delete_tokens.service Thu 2019-03-21 14:40:00 UTC 46min left n/a n/a keystone_novaadmin_delete_tokens.timer keystone_novaadmin_delete_tokens.service Fri 2019-03-22 13:30:52 UTC 23h left Thu 2019-03-21 13:30:52 UTC 22min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service 6 timers listed. ``` Now all logs can be accessed by `journalctl -u $unit` and Icinga can monitor failures as well. If you find any issues, please let me know. Regards, -- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services

2 2

q4 goals etherpad
by Andrew Bogott 13 Mar '19

13 Mar '19

I just now scratched out an etherpad with proposed Q4 goals: https://etherpad.wikimedia.org/p/wmcs_q4_2019 I think that includes everything we discussed in the meeting yesterday, with most bullet points off the top of my head. Feel free to revise as you see fit, but also please add notes at the bottom about which goals you think are realistic for the next quarter. We should try to be modest in our ambitions given that we'll be training in at least one new person at the same time. -A

1 0

Toolforge stretch bastion rebooted
by Arturo Borrero Gonzalez 13 Mar '19

13 Mar '19

Hi, Sorry, for the short notice, but I just had to reboot the Toolforge stretch bastion server (tools-sgebastion-07). It should be back online already. Context is: https://phabricator.wikimedia.org/T215154 regards -- Arturo Borrero Gonzalez Operations Engineer / Wikimedia Cloud Services Wikimedia Foundation

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin March 2019