FYI
---------- Forwarded message ----------
From: Ariel Glenn WMF <ariel(a)wikimedia.org>
Date: Mon, Sep 12, 2016 at 9:07 AM
Subject: [Research-Internal] Fwd: Dumps Rewrite getting underway (help
needed!)
To: research-internal(a)lists.wikimedia.org
---------- Forwarded message ----------
From: Ariel Glenn WMF <ariel(a)wikimedia.org>
Date: Mon, Sep 5, 2016 at 2:35 PM
Subject: Dumps Rewrite getting underway (help needed!)
To: Wikipedia Xmldatadumps-l <Xmldatadumps-l(a)lists.wikimedia.org>
Hello folks,
I know a number of you have subscribed to the Dumps Rewrite project (
https://phabricator.wikimedia.org/tag/dumps-rewrite/) but I bet none of you
actually watch it or any of its tasks. So here's a heads up.
I'm getting started on work on the job scheduler/workflow manager piece;
this would accept lists of dump tasks (in the current setup, "dump stubs
for el wikipedia"), call a callback to turn each of them into small jobs
that can be completed in less than an hour, submit and monitor these jobs
with retries, dependencies etc, call a callback to recombine the outputs of
the jobs, and notify some caller on success of te whole operation.
First up is evaluating existing packages and choosing one to use as a
foundation. Please contribute! See the following tasks:
https://phabricator.wikimedia.org/T143205: Draft usage scenarios for
job/workflow manager <https://phabricator.wikimedia.org/T143205>
https://phabricator.wikimedia.org/T143206: List requirements needed for
task/job/workflow manager <https://phabricator.wikimedia.org/T143206>
https://phabricator.wikimedia.org/T143207: Evaluate software packages for
job/task/workflow management <https://phabricator.wikimedia.org/T143207>
Also, can someone please forward this on to analytics-l and research-l?
I'm not on those lists but they will no doubt have a lot of useful
expertise here.
Thanks!
Ariel
_______________________________________________
Research-Internal mailing list
Research-Internal(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/research-internal