Initial thoughts -
At a high level, we need to make sure that our analytics engineering work is organized in a manner that is
a) architecturally sane; b) iterative; c) customer-driven.
The relatively high level of detachment of the analytics cluster work from ops (partially a function of how the team is situated in the org, partially a result of a high desire for expediency that's led to some bad habits) is clearly not sustainable and won't satisfy a) or b).
So there's no question that the structure of the work needs to change going forward. I doubt anyone in analytics would disagree with that. :-)
Moreover, we've already identified a clear near term deliverable: a minimally viable Hadoop cluster setup that passes architectural muster, with availability of the data currently in HDFS so that existing research (mobile PV + Wikipedia Zero analysis) can continue and be expanded upon.
My opening view is that we should strive to empower a cross-functional (i.e. ops+analytics) team to provide this deliverable as it sees fit, and ensure that said team is staffed so that its decisions will generally be trusted and accepted by both ops & analytics. This, IMO, is the best way to satisfy all three criteria above. If that approach sounds viable, then we'll just need to decide who's on that team, and trust them to do the work -- they would decide, ideally in consensus, how much of the work done so far to re-use vs. reboot.
I want to emphasize that the Hadoop cluster and related infrastructure is just a small part of the analytics engineering work we need to do. So we shouldn't agonize too much about who's part of that effort as long as we think it's a viable team that'll get the job done with minimal drama and at a level of quality that'll be satisfactory.
Oddly enough, I think all of the above is consistent with the outcomes of the meeting today. :-)
As for the long term ownership/support of the system, again, I think we just need to break down the responsibilities a bit more. Clearly we'll want ops to be in a position to maintain analytics services like any others. There may be larger changes that again require more cross-functional work. And there may be parts of the cluster administration and policy that can be handled entirely by analytics.
Erik