Hello folks,

I know a number of you have subscribed to the Dumps Rewrite project (https://phabricator.wikimedia.org/tag/dumps-rewrite/) but I bet none of you actually watch it or any of its tasks. So here's a heads up.

I'm getting started on work on the job scheduler/workflow manager piece; this would accept lists of dump tasks (in the current setup, "dump stubs for el wikipedia"), call a callback to turn each of them into small jobs that can be completed in less than an hour, submit and monitor these jobs with retries, dependencies etc, call a callback to recombine the outputs of the jobs, and notify some caller on success of te whole operation.

First up is evaluating existing packages and choosing one to use as a foundation. Please contribute! See the following tasks:

https://phabricator.wikimedia.org/T143205: Draft usage scenarios for job/workflow manager
https://phabricator.wikimedia.org/T143206: List requirements needed for task/job/workflow manager
https://phabricator.wikimedia.org/T143207: Evaluate software packages for job/task/workflow management

Also, can someone please forward this on to analytics-l and research-l? I'm not on those lists but they will no doubt have a lot of useful expertise here.

Thanks!

Ariel