As was previously announced on the xmldatadumps-l list, the sql/xml dumps
generated twice a month will be written to an internal server, starting
with the November run. This is in part to reduce load on the web/rsync/nfs
server which has been doing this work also until now. We want separation
of roles for some other reasons too.
Because I want to get this right, and there are a lot of moving parts, and
I don't want to rsync all the prefetch data over to these boxes again next
month after cancelling the move:
********
If needed, the November full run will be delayed for a few days.
If the November full run takes too long, the partial run, usually starting
on the 20th of the month, will not take place.
*********
Additionally, as described in an earlier email on the xmldatadumps-l list:
*********
files will show up on the web server/rsync server with a substantial
delay. Initially this may be a day or more. This includes index.html and
other status files.
*********
You can keep track of developments here:
https://phabricator.wikimedia.org/T178893
If you know folks not on the lists in the recipients field for this email,
please forward it to them and suggest that they subscribe to this list.
Thanks,
Ariel
This issue will be tracked here. https://phabricator.wikimedia.org/T178893
As it says on the ticket, I hope to get this done in time for the Nov 1 run.
Here is what it means for folks who download the dumps:
* First off, the host where the dumps are generated will no longer be the
host that serves them to the web (or that serves them internally via NFS).
This means that you won't see automatic minute-to-minute updates of how the
dumps are doing. I'll be rsyncing over files to the web server, probably
with a several hour delay.
* Second, it's possible that the index.html or other status files that you
check will point to things that aren't rsynced over yet. If so, try again
in a few hours and the files should have arrived.
* Datasets that arrive via weekly or daily cron jobs, such as the wikidata
dumps or the adds/changes dumps, will not be affected at this stage. The
plan is to move them later.
* Mirrors and web service will continue to remain where they are.
I may have forgotten some things; if so I'll update as they occur to me.
Questions and/or comments welcome.
Ariel