Hi everyone,

stat1004 has been reimaged (with /srv preserved), and is back in operation. It should
work just as before, but with newer versions of kernel and system software.

Luca has prepared a document that explains how to reset virtual envs if needed:

https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter#Resetting_user_virtualenvs

If there is anything out of place, just let us know.

The updates to stat1005 and stat1007 will happen next week at the earliest, preceded
by separate announcements at least 24h before the individual reinstalls.

Best,
Tobias

On Thu, Sep 17, 2020 at 1:11 PM Tobias Klausmann <tklausmann@wikimedia.org> wrote:
Dear users of stat1004,

as it turns out, the backup of /srv will take a total of about eight hours or so.
Thus, we will only do the backup step today (UTC/CEST), and the reimaging first thing
tomorrow.

Apologies for this taking longer than expected. I'll send an update here as things
progress.

Best,
Tobias

On Tue, Sep 15, 2020 at 2:20 PM Tobias Klausmann <tklausmann@wikimedia.org> wrote:
Dear users of stat100{4,6,7},

we are planning on upgrading stat1004 to Debian Buster this Thursday
(2020-09-17) after 12:00 CEST (10:00 UTC). We will reinstall the machine,
preserving user data (home directories, /srv), but to be on the safe side,
we will backup that data. After the reinstall and a few tests, we will send
an all-clear to this list.

A few things of note:

- It would be greatly appreciated if you cleaned out unneeded data before the
  backup time mentioned above, thus speeding up backup (and restore if we need
  it).
- Any changes made to the file system contents after the time mentioned above
  may be lost.
- Around the time of the backup, both cron and systemd timers will be
  disabled, and still-running process may be ungracefully terminated.

If this process works well, the remaining stat100x machines in need of update
(6, 7) will be processed in a similar manner.

As always, if there are questions, do not hesitate to contact us.

Best,
Tobias

--
Tobias Klausmann, SRE, Wikimedia Foundation



--
Tobias Klausmann, SRE, Wikimedia Foundation



--
Tobias Klausmann, SRE, Wikimedia Foundation