[Labs-admin] 2017-07-02 Toolforge data loss for permissive data

[Chase Pettet]

On 2017-07-02 some users experienced a loss of data in their projects.

We estimate 126 Tools out of 1,766 saw at least one file removed.  We have
reason to believe a user acting under a Tool account issued the command 'rm
-fr *' at the wrong point in the directory structure.  Anyone who had files
that were removable by this user was effected.  Thankfully, we have a
backup from before the command was run, a minority of users have overly
permissive files, and a further minority were severely impacted.
https://phabricator.wikimedia.org/T169774 was created in response to
inquiries surrounding data loss.

We do not guarantee any level of user backups for day-to-day operations,
but in this case since we do have the data I have restored it to
/data/scratch/T169774/ so users can retrieve what was removed.  We intend
to make this restored data available until at least 2017-08-08.  A Warning:
Please do not rely on NFS for backups of code or critical data.  We only
have capacity to keep 2 weeks of historical backups at the moment and
cannot guarantee timely retrieval or availability.  Every Tool account can
use https://phabricator.wikimedia.org/diffusion/ for code hosting, and
creation of the repository is handled by going to
https://toolsadmin.wikimedia.org/tools/id/<mytool>.

This calamity was almost entirely caused by directories with o+w set
allowing 'other' or 'everyone' write access.  Do not use permissions such
as '777' or that look like 'drwxrwxrwx' as it will allow other users to
remove your files.  This is especially dangerous in a shared hosting
environment as this incident has shown.

A brief explanation of why this happened to users who have given write
permissions to 'other' for a directory in their Tool:

*Because directories are not used in the same way as regular files, the
permissions work slightly (but only slightly) differently.  An attempt to
list the files in a directory requires read permission for the directory,
but not on the files within.  An attempt to add a file to a directory,
delete a file from a directory, or to rename a file, all require write
permission for the directory, but (perhaps surprisingly) not for the files
within. *

*-  *Unix File and Directory Permissions and Modes (
https://wpollock.com/AUnix1/FilePermissions.htm)

ACTION ITEMS:
- Make sure you have backups of code and data needed
- Check for removed data you want to restore on login.tools.wmflabs.org at
/data/scratch/T169774/
- Check your Tools files and directories for o+w permission and remove if
possible (chmod -R o-w <directory>).
- Ask for help on the labs-l mailing list, Phabricator, or in the
#wikimedia-cloud IRC channel if you cannot figure out how to do without o+w
(someone may have a different solution).

-- 
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-admin/attachments/20170707/8ce377dc/attachment.html>