Hi all,
*Who: *This mostly applies to people who have access to the stat1002 and stat1003 statistics machines on the production cluster, and publish datasets as static files.
*What:* We are no longer using datasets.wikimedia.org to serve static datasets. We have set up a redirect, so requests like https://datasets.wikimedia.org/ $1 will be sent to https://analytics.wikimedia.org/datasets/archive/ $1. Most importantly, publishing datasets is now much easier. Any files you put in published-datasets on either machine:
stat1002:/a/published-datasets stat1003:/srv/published-datasets
Are going to be merged together and served together on:
https://analytics.wikimedia.org/datasets/
One request as we all enjoy this much simpler process: let's use README files in these directories to let future versions of us know what the datasets are all about. That will make the repository more fun for others to browse and ease future cleanups. Thank you!
*TODO*
If something of yours got lost, let us know, we have backups. If you had stuff that we might have cleaned up, we put it in /srv/otto-to-delete-datasets-cleanup and /a/otto-to-delete-datasets-cleanup. Take a look there and you can move files as you see fit into published-datasets
*Context*
For a long time, publishing files from stat1002 and stat1003 was quite painful. There were three folders, some on both boxes, some only on one box, symlinks, rsyncs, it was bad. We talked to everyone who had files in these folders and gathered consensus for this deprecation. If this message catches you by surprise, please let us know what channel we should reach you in next time and we'll add it to our communication plan.
This work is tracked in T159409 https://phabricator.wikimedia.org/T159409