Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Hello,
Ariel noted last month that the dumps are currently being ran manually. [1] Having delays in the dumps production should thus be expected.
Hope this helps.
[1]: https://lists.wikimedia.org/pipermail/xmldatadumps-l/2015-July/001153.html
On 5 Aug 2015, at 07:26, Bryan White bgwhite@gmail.com wrote:
Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
I did mention that dumps are now run manually. I was on the message you mentioned. Only one dump has started in the last 18 days, this is in addition to larger languages never completing. Going from two dumps a month to no dumps a month is not a "delay", it is dead.
Bryan
On Tue, Aug 4, 2015 at 6:08 PM, Hydriz Scholz admin@alphacorp.tk wrote:
Hello,
Ariel noted last month that the dumps are currently being ran manually. [1] Having delays in the dumps production should thus be expected.
Hope this helps.
On 5 Aug 2015, at 07:26, Bryan White bgwhite@gmail.com wrote:
Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Hello,
I apologize if I was not clear enough. The dumps are being run manually because Ariel is currently testing the new framework. It is thus to be expected if the dumps are not being produced according to schedule. Also, I believe the enwiki dump is still being produced, so I doubt that the process is "dead", it's just slow.
Perhaps Ariel is able to provide more information on this issue, as I am unable to comment further than this.
On 5 Aug 2015, at 08:38, Bryan White bgwhite@gmail.com wrote:
I did mention that dumps are now run manually. I was on the message you mentioned. Only one dump has started in the last 18 days, this is in addition to larger languages never completing. Going from two dumps a month to no dumps a month is not a "delay", it is dead.
Bryan
On Tue, Aug 4, 2015 at 6:08 PM, Hydriz Scholz admin@alphacorp.tk wrote: Hello,
Ariel noted last month that the dumps are currently being ran manually. [1] Having delays in the dumps production should thus be expected.
Hope this helps.
On 5 Aug 2015, at 07:26, Bryan White bgwhite@gmail.com wrote:
Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Again, I was on the message you mentioned. I was the person Ariel was responding too. I have been in touch with Ariel in the past, but he not responded to any emails from me or others for almost three weeks about the situation. Problems in Greece, on holiday, real-life, ...?
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
I use over 40 languages, not just enwiki. The one's that didn't finish are useless for the majority of people. It takes two weeks to dump all languages except for enwiki and only one has started in 18 days. By "started", I mean re-started a pre-existing dump to completion. No new dump has commenced in over a month. For many languages (French, German, etc) it has been two months. Things are dead.
Does anybody know how to get dumps working who's name isn't Ariel?
Bryan
On Tue, Aug 4, 2015 at 7:36 PM, Hydriz Scholz admin@alphacorp.tk wrote:
Hello,
I apologize if I was not clear enough. The dumps are being run manually because Ariel is currently testing the new framework. It is thus to be expected if the dumps are not being produced according to schedule. Also, I believe the enwiki dump is still being produced, so I doubt that the process is "dead", it's just slow.
Perhaps Ariel is able to provide more information on this issue, as I am unable to comment further than this.
On 5 Aug 2015, at 08:38, Bryan White bgwhite@gmail.com wrote:
I did mention that dumps are now run manually. I was on the message you mentioned. Only one dump has started in the last 18 days, this is in addition to larger languages never completing. Going from two dumps a month to no dumps a month is not a "delay", it is dead.
Bryan
On Tue, Aug 4, 2015 at 6:08 PM, Hydriz Scholz admin@alphacorp.tk wrote: Hello,
Ariel noted last month that the dumps are currently being ran manually. [1] Having delays in the dumps production should thus be expected.
Hope this helps.
On 5 Aug 2015, at 07:26, Bryan White bgwhite@gmail.com wrote:
Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Probably no one else does know (how to run these dumps).
So to clear up a little of the confusion/lack of information, automating staged dumps is currently tracked in Phabricator here: https://phabricator.wikimedia.org/T107750
Just got the configuration files puppetized for the scheduler, which was started manually this time but won't need any other intervention. I'll be keeping an eye on it anyways since this too is new code.
I've been playing whack-a-mole with bugs in the streaming stubs/logs production, which is why the delay again last month in completing enwp and why only one set of dumps went off. This run should be better, cross fingers that nothing else crept in.
After one complete run finishes for the non en wikipedia dumps, we'll be able to do a second run, probably leaving off revision history since most folks don't need those more than once a month. <-- If this is wrong, speak up now.
The order in which things are run has been redone a bit, hopefully to get the parts of the dumps people need done sooner in the cycle.
I'm still reluctant to do partial dump copies over labs unless we can guarantee a little more space over there to keep e.g. the last full good dump along with whatever new files are being copied. I've opened a Phabricator task for that here https://phabricator.wikimedia.org/T108077 and added Coren so we can figure out the space issue.
I've not wanted to weigh in earlier until I had things in place to kick off the new run. My apologies if that just made folks more annoyed.
Ariel
Στις 04-08-2015, ημέρα Τρι, και ώρα 20:08 -0600, ο/η Bryan White έγραψε:
Again, I was on the message you mentioned. I was the person Ariel was responding too. I have been in touch with Ariel in the past, but he not responded to any emails from me or others for almost three weeks about the situation. Problems in Greece, on holiday, real-life, ...?
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
I use over 40 languages, not just enwiki. The one's that didn't finish are useless for the majority of people. It takes two weeks to dump all languages except for enwiki and only one has started in 18 days. By "started", I mean re-started a pre-existing dump to completion. No new dump has commenced in over a month. For many languages (French, German, etc) it has been two months. Things are dead.
Does anybody know how to get dumps working who's name isn't Ariel?
Bryan
On Tue, Aug 4, 2015 at 7:36 PM, Hydriz Scholz admin@alphacorp.tk wrote:
Hello,
I apologize if I was not clear enough. The dumps are being run manually because Ariel is currently testing the new framework. It is thus to be expected if the dumps are not being produced according to schedule. Also, I believe the enwiki dump is still being produced, so I doubt that the process is "dead", it's just slow.
Perhaps Ariel is able to provide more information on this issue, as I am unable to comment further than this.
On 5 Aug 2015, at 08:38, Bryan White bgwhite@gmail.com wrote:
I did mention that dumps are now run manually. I was on the message you mentioned. Only one dump has started in the last 18 days, this is in addition to larger languages never completing. Going from two dumps a month to no dumps a month is not a "delay", it is dead.
Bryan
On Tue, Aug 4, 2015 at 6:08 PM, Hydriz Scholz < admin@alphacorp.tk> wrote: Hello,
Ariel noted last month that the dumps are currently being ran manually. [1] Having delays in the dumps production should thus be expected.
Hope this helps.
July/001153.html
On 5 Aug 2015, at 07:26, Bryan White bgwhite@gmail.com wrote:
Three months ago, "they" changed the way dumps are done. "They" wanted stubs for every language done first and work on the rest of the dump later.
Old way: Two dumps a month for all languages except enwiki. A new dump was started when one completed. Except for larger languages, dumps were done in under a day.
New way: One dump a month for majority of languages. Dumps have to be started manually. All dumps take 2-4 weeks to complete.
Last month, French, German, Japanese, Russian, Spanish and others were never completed.
It's about to be August 5. No dumps have started this month. Only one dump has started in over two weeks.
Does anybody know how to get dumps working again.
Bryan
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Xmldatadumps-l mailing list Xmldatadumps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
xmldatadumps-l@lists.wikimedia.org