Hey Hydriz,
1) I'd like to get the scheduler eval done in the next quarter and
hopefully also convert the current dumps to use it during that quarter as
well. (Code for that, when the time comes, will live in the regular dumps
repo.)
2) There is not yet a repo for the dumps rewrite; we are not yet to the
point of writing code. When we do, there sure will be.
Ariel
On Tue, Sep 6, 2016 at 2:02 AM, Hydriz Scholz <hydriz(a)jorked.com> wrote:
Hi Ariel,
Great news to hear about the progress of the project. I have 2 questions
in mind:
1) Is there any deadline for this project that we aim to work towards?
Having a deadline sounds good in ensuring that the project stays on track.
2) How are volunteers able to help in the software development? Is the
repository for the project hosted somewhere visible? (Or, are patches
welcome?)
Thanks.
On 5 Sep 2016, at 19:35, Ariel Glenn WMF <ariel(a)wikimedia.org> wrote:
Hello folks,
I know a number of you have subscribed to the Dumps Rewrite project (
https://phabricator.wikimedia.org/tag/dumps-rewrite/) but I bet none of
you actually watch it or any of its tasks. So here's a heads up.
I'm getting started on work on the job scheduler/workflow manager piece;
this would accept lists of dump tasks (in the current setup, "dump stubs
for el wikipedia"), call a callback to turn each of them into small jobs
that can be completed in less than an hour, submit and monitor these jobs
with retries, dependencies etc, call a callback to recombine the outputs of
the jobs, and notify some caller on success of te whole operation.
First up is evaluating existing packages and choosing one to use as a
foundation. Please contribute! See the following tasks:
https://phabricator.wikimedia.org/T143205: Draft usage scenarios for
job/workflow manager <https://phabricator.wikimedia.org/T143205>
https://phabricator.wikimedia.org/T143206: List requirements needed for
task/job/workflow manager <https://phabricator.wikimedia.org/T143206>
https://phabricator.wikimedia.org/T143207: Evaluate software packages for
job/task/workflow management <https://phabricator.wikimedia.org/T143207>
Also, can someone please forward this on to analytics-l and research-l?
I'm not on those lists but they will no doubt have a lot of useful
expertise here.
Thanks!
Ariel
_______________________________________________
Xmldatadumps-l mailing list
Xmldatadumps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l