Hi Ariel,
Great news to hear about the progress of the project. I have 2 questions in mind:
1) Is there any deadline for this project that we aim to work towards? Having a deadline
sounds good in ensuring that the project stays on track.
2) How are volunteers able to help in the software development? Is the repository for the
project hosted somewhere visible? (Or, are patches welcome?)
Thanks.
On 5 Sep 2016, at 19:35, Ariel Glenn WMF
<ariel(a)wikimedia.org> wrote:
Hello folks,
I know a number of you have subscribed to the Dumps Rewrite project
(
https://phabricator.wikimedia.org/tag/dumps-rewrite/) but I bet none of you actually
watch it or any of its tasks. So here's a heads up.
I'm getting started on work on the job scheduler/workflow manager piece; this would
accept lists of dump tasks (in the current setup, "dump stubs for el
wikipedia"), call a callback to turn each of them into small jobs that can be
completed in less than an hour, submit and monitor these jobs with retries, dependencies
etc, call a callback to recombine the outputs of the jobs, and notify some caller on
success of te whole operation.
First up is evaluating existing packages and choosing one to use as a foundation. Please
contribute! See the following tasks:
https://phabricator.wikimedia.org/T143205: Draft usage scenarios for job/workflow manager
https://phabricator.wikimedia.org/T143206: List requirements needed for task/job/workflow
manager
https://phabricator.wikimedia.org/T143207: Evaluate software packages for
job/task/workflow management
Also, can someone please forward this on to analytics-l and research-l? I'm not on
those lists but they will no doubt have a lot of useful expertise here.
Thanks!
Ariel
_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l