Hello,
Pretty much everyone who's dealt with creating views for new wikis on the
labs hosts have experienced issues with "Access denied" sometimes.
This was usually due to the MariaDB grant role being missed. We tried to
workaround this by including the grant addition on the maintain-views
script.
Unfortunately, we ran into very weird problems when doing so and this is an
example: https://phabricator.wikimedia.org/T193187#4273281
After lots of back and forth we decided to create a bug to MariaDB (
https://jira.mariadb.org/browse/MDEV-16466) which was confirmed by MariaDB
yesterday and pointed to a similar issue (
https://jira.mariadb.org/browse/MDEV-14732).
The expected fix will come in 10.4 (we are in 10.1), so quite long ahead of
us.
So, for now, the workaround before adding new views is to manually add the
GRANT on the DB and then run the script:
GRANT SELECT, SHOW VIEW ON `newiki\_p`.* to labsdbuser';
Hopefully with this email everyone is on the same page now.
Thanks everyone (specially Brooke for helping me out with the
troubleshooting!)
Manuel.
Hello,
Could someone help me add NFS access to project `paws`? I couldn't
find any documentation on this. We need access to `home` and `project`
shares so it can migrate out of the `tools` project.
Thank you,
--
Giovanni Tirloni
Operations Engineer
Wikimedia Foundation
I have just moved all tools nodes to the new read-only ldap service.
This change was made via the tools project puppet panel in horizon.
Everything looks fine so far, but I'm going to be AFK for much of the
weekend. So, if things go pear-shaped over the weekend, here's how to
revert:
1) In Horizon, go to the 'Project Puppet' tab in the 'tools' project
2) Scroll all the way to the bottom, to the 'Hiera Config' section
3) Remove these lines:
labsldapconfig:
hostname: ldap-ro.eqiad.wikimedia.org
secondary_hostname: ldap-ro.eqiad.wikimedia.org
4) Force puppet runs as needed, or just wait a bit.
Assuming things go well, eventually I'll remove the project-specific
hiera hacks and just make this change cloud-wide
-A
Hi,
Just a heads up, I've merged a change that converts OpenStack admin cronjobs to systemd timers.
List all timers:
```
root@cloudcontrol1003:~# systemctl list-timers --no-pager
NEXT LEFT LAST PASSED UNIT ACTIVATES
Thu 2019-03-21 14:00:00 UTC 6min left Thu 2019-03-21 13:50:01 UTC 3min 31s ago designate_floating_ip_ptr_records_updater.timer designate_floating_ip_ptr_records_updater.service
Thu 2019-03-21 14:15:00 UTC 21min left n/a n/a glance_rsync_images.timer glance_rsync_images.service
Thu 2019-03-21 14:20:00 UTC 26min left n/a n/a keystone_delete_expired_tokens.timer keystone_delete_expired_tokens.service
Thu 2019-03-21 14:30:00 UTC 36min left n/a n/a keystone_novaobserver_delete_tokens.timer keystone_novaobserver_delete_tokens.service
Thu 2019-03-21 14:40:00 UTC 46min left n/a n/a keystone_novaadmin_delete_tokens.timer keystone_novaadmin_delete_tokens.service
Fri 2019-03-22 13:30:52 UTC 23h left Thu 2019-03-21 13:30:52 UTC 22min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
6 timers listed.
```
Now all logs can be accessed by `journalctl -u $unit` and Icinga can monitor failures as well.
If you find any issues, please let me know.
Regards,
--
Giovanni Tirloni
Operations Engineer
Wikimedia Cloud Services
I just now scratched out an etherpad with proposed Q4 goals:
https://etherpad.wikimedia.org/p/wmcs_q4_2019
I think that includes everything we discussed in the meeting yesterday,
with most bullet points off the top of my head. Feel free to revise as
you see fit, but also please add notes at the bottom about which goals
you think are realistic for the next quarter. We should try to be
modest in our ambitions given that we'll be training in at least one new
person at the same time.
-A
Hi,
Sorry, for the short notice, but I just had to reboot the Toolforge stretch
bastion server (tools-sgebastion-07).
It should be back online already.
Context is: https://phabricator.wikimedia.org/T215154
regards
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Foundation