[Labs-l] Filesystem downtime to schedule

[Marc A. Pelletier]

Hello Labs,

Many of you may recall that until some point late 2013, one of the 
features of the labs file server was that it provided time travel 
snapshots (you could see a consistent view of the filesystem as it 
existed 1h, 2h, 3h, 1d, 2d, 3d and 1 week ago).

This was disabled at that time - despite being generally considered 
valuable - because it was suspected to be (part of) the stability 
problems the NFS server suffered at the time.  This turns out to not 
have been the case, and we could turn it back on now.

Indeed, doing so is a prerequisite to the planned replication of the 
filesystem in the new datacenter where a redundant Labs installation is 
slated to be deployed[1].

The issue is that turning that feature back on requires changing the way 
the disk space is currently allocated at a low level[2] and necessitates 
a fairly long period of partial downtime during which data is being 
copied from one part of the disk subsystem to the other.  In practice, 
this would require the primary partitions (/home and /data/project) to 
be set readonly for a period on the order of a day (24-30 hours).

That downtime is pretty much unavoidable eventually as it is a 
requirement of expanding labs and improving data resillience and 
reliability, but the /timing/ of that is flexible.  I wanted to "poll" 
labs users as to when the possibility of disruption is minimized, and 
give everyone plenty of time to make contingency planning and/or notify 
their endusers of the expected period of reduced availability.

Provided there is a good consensus that the week is a better time than 
the weekend (I am guessing here that volunteer coders and users are more 
active during the weekend) then I would suggest starting the operation 
on Tuesday, January 13 at 18:00 UTC.  The period of downtime is expected 
to last until January 14, 18:00 UTC but may extend a few hours beyond that.

The expected impacts are:

* Starting at the beginning of the window, /home and /data/project will 
switch to readonly mode; any attempt to write to files to those trees 
will result in EROFS errors being thrown.  Reading from those 
filesystems will still work as expected, so would writing to other 
filesystems;
* Read performance may degrade noticably as the disk subsystem will be 
loaded to capacity;
* It will not be possible to manipulate the gridengine queue - 
specifically, starting or stopping jobs will not work; and
* At the end of the window, when the operation is complete, the "old" 
file system will go away and be replaced by the new one - this will 
cause any access to files or directories that were previously opened 
(including working directories) on the affected filesystems to error out 
with ESTALE.  Reopening files by name will access the new copy identical 
to the one at the time the filesystems became readonly.

In practice, that latter impact has the effect that most running 
programs will be unable to continue unless they have special handling 
for this situation, and most gridengine jobs will no longer be able to 
log output.  It may be a good idea to restart any continuous tool at 
that time.  All webservices that were running at the start of the 
maintenance window will be restarted at that time.

If you have tools or other processes running that do not rely on being 
able to write to /data/project, they may be able to continue running 
during the downtime without interruption.  Jobs that only access the 
network (for instance, the Mediawiki API) or the databases will not 
likely be affected.  Because of this, no automatic or forcible restart 
of running (non-webservice) jobs will be made.

In particular, if you have a tool whose continued operation is 
important, temporarily modifying it so that it works from /data/scratch 
may be a good workaround.

Finally, in order to avoid risks of the filesystem move taking longer 
than expected and increasing downtime significantly, LOG FILES OVER 1G 
WILL BE NOT BE COPIED.  If you have critical files that are not simple 
log files but whose names end in .log, .err or .out then you MUST 
compress those files if you absolutely require them to survive the 
transition.  Alternately, truncating them to some size comfortably 
smaller than 1G will work if the file must remain uncompressed.

The speed and reliability of the maintenance process depends on the 
total data to copy.  If you can clean up both your home and project 
directories of extraneous files, you'll help the process greatly.  :-)

Thanks all,

-- Marc