Howdy,
After having spent some time reviewing the analytics github repo and
playing observer to the quarterly review last December, and today's
security/architecture mixup, I have a few opinions and suggestions that I'd
like to share. They may upset some or step on toes. Sorry about that.
Main suggestion - all logging, etl, storage, and compute infrastructure
should be owned, implemented, and maintained by the operations team. There
should be a clear set of deliverables for ops: the entirety of the current
udp stream ingested, processed via an extensible etl layer with a minimum
of IP anonymization in place, and stored in hdfs in a standardized format
with logical access controls. Technology and implementation choices should
ultimately rest with ops so long as all deliverables are met, though
external advice and assistance (including from industry experts outside of
wmf) will be welcome and solicited.
The analytics team owns everything above this. Is pig the best tool to
analyze log data in hdfs? Does hive make sense for some things? Want to
add and analyze wiki revisions via map reduce jobs? Visualize everything
imaginable? Add more sophisticated transforms to the etl pipeline? Go,
go, analytics!
I see the work accomplished to date under the heading of kraken as falling
into three categories:
1) Data querying. This includes pig integration, repeatable queries run
via pig, and ad hoc map reduce jobs meant to analyze data written by folks
like Diederik. While modifications may be needed if there are changes to
how data is stored in hdfs (such as file name conventions or format) or to
access controls, this category of work isn't tied to infrastructure details
and should be reusable on any generic hadoop implementation containing wmf
log data.
2) Devops work. This includes everything Andrew Otto has done to puppetize
various pieces of the existing infrastructure. I'd consider all of this
experimental. Some might be reusable, some may need refactoring, some
should be chalked up as a learning exercise and abandoned. Even if the
majority was to fall under that last category, this has undoubtedly been a
valuable learning experience. Were Andrew to join the ops team and
collaborate with others on a from scratch implementation (let's say I'd
prefer us using the beta branch of actual apache hadoop instead of
cloudera), I'm sure the experience he's gained to date will be of use to
all.
3) Bound for mordor. Never happened, never to be spoken of again. This
includes things like the map reduce job executed via cron to transfer data
from kafka to hdfs, and... oh wait, never happened, never to be spoken of
again.
Unless I'm missing anything major, I don't see any reasons not to pursue
this new approach, nor does it appear that any significant amount of work
would be lost. Instead, the most useful bits (category 1) should still be
useful. And since that seems to be where analytics has been most
successful, perhaps it makes sense to let them focus fully on this sort of
thing instead of infrastructure.
-Asher