Planning for the full redesign to kick off early 2016. When I say "full redesign" I mean it, so let's not talk small stuff about whether json or xml is a batter format; let's talk about a design that will allow us to plug in various formats easily, for example.
Starting point for discussion here:
https://phabricator.wikimedia.org/T114019
See the last link on the description.
What we can start on now: are these the right components to break the dumps into? Can we indeed cover all sorts of dumps this way? What existing software can we re-use? WHat sort of grid computing or other package can we use for the "black box" in the diagram? Etc. etc. We want to get as much of this hashed out ahead of time so that we can get the maximum out of the session.
Ariel