On 12/12/19 11:50 AM, Andrew Bogott wrote:
We are still chasing down stray issues (in particular,
some of the dump and
scratch mounts on toolforge or now wrong) but for almost all use cases things
should be back to normal.
-Andrew
We consider the operations finished. Everything has been done.
NFS, being one of the weakest components of our infra, suffered during today's
operation. We were force to reboot most of Toolforge servers, so grid jobs and
webservices in both the web grid and kubernetes have most likely been restarted
and may present error log entries corresponding to the window of this operation.
Other CloudVPS projects users of NFS (dumps shares, maps, etc) might also
require some checking. Please get in touch if you are a project admin of such
project.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation