We had issues with NFS mounts in Toolforge as an unintended side
effect of OpenStack upgrades earlier today. While NFS mounts were
broken, many (all?) cronjobs failed. The emails telling maintainers
about these failures did not go out in real time because the mail
system also relies on NFS. When we fixed the mail server thousands of
queued messages started going out. There are still a few emails
sending as I write this, but the queue is almost empty.
As far as we can tell, cronjobs are starting as expected now and the
emails are just left over noise.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808