Sorry about the noise with the alerts. After deploying, I discovered a few small errors in the timer/service definitions that have been fixed.
Finally, there is an error in Wikitech that is causing `mwscript runJobs.php` to fail frequently (T218935). That was being hidden in the noise with cronjobs as was surfaced by adopting timers, so I think that's a win. That job has been downtimed in Icinga for now.
On Thu, Mar 21, 2019 at 11:00 AM Giovanni Tirloni gtirloni@wikimedia.org wrote:
Hi,
Just a heads up, I've merged a change that converts OpenStack admin cronjobs to systemd timers.
List all timers:
root@cloudcontrol1003:~# systemctl list-timers --no-pager NEXT LEFT LAST PASSED UNIT ACTIVATES Thu 2019-03-21 14:00:00 UTC 6min left Thu 2019-03-21 13:50:01 UTC 3min 31s ago designate_floating_ip_ptr_records_updater.timer designate_floating_ip_ptr_records_updater.service Thu 2019-03-21 14:15:00 UTC 21min left n/a n/a glance_rsync_images.timer glance_rsync_images.service Thu 2019-03-21 14:20:00 UTC 26min left n/a n/a keystone_delete_expired_tokens.timer keystone_delete_expired_tokens.service Thu 2019-03-21 14:30:00 UTC 36min left n/a n/a keystone_novaobserver_delete_tokens.timer keystone_novaobserver_delete_tokens.service Thu 2019-03-21 14:40:00 UTC 46min left n/a n/a keystone_novaadmin_delete_tokens.timer keystone_novaadmin_delete_tokens.service Fri 2019-03-22 13:30:52 UTC 23h left Thu 2019-03-21 13:30:52 UTC 22min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service 6 timers listed.Now all logs can be accessed by `journalctl -u $unit` and Icinga can monitor failures as well.
If you find any issues, please let me know.
Regards,
-- Giovanni Tirloni Operations Engineer Wikimedia Cloud Services