Hi everybody, here's a quick summary of notes / take-aways from the
Analytics (Kraken) security review meeting.
* analytics1001 has been wiped and reimaged (restoring /home from backup)
* All proxies and externally-facing services have been disabled.
* Work is under way to bring everything that was puppetized under the
analytics1001 puppetmaster into operations-puppet after proper review.
Andrew is working closely with a number of people in ops to make this
happen.
* All future deployments to the cluster will be puppetized and go through
normal code review. Other than performance testing, these puppet confs will
be tested in labs.
* The rest of the cluster will be wiped and reimaged out of puppet; data in
HDFS will be preserved. This can be a rolling process allowing work to
proceed while its under way.
* Schedule an Architectural Review meeting sometime during the SF Ops
hackathon, including a look at additional services and auth methods that
provide access to internal dashboards like Hue &such.
* Ensure all current "application" code (stuff written by WMF) gets
reviewed:
** Cron doing HDFS import from Kafka
** Pig UDFs and other data tools used in processing
** Future: Storm ETL layer
We all agreed the overall goal is to get to an acceptable security state.
During that process, the Analytics team still needs to continue to meet
stakeholder needs and deliver on promises. We decided on keeping running a
"minimum viable cluster" while reimaging boxes and civilizing cluster
configuration:
* Wall off some portion of the boxes to continue recieving data and running
jobs; all other boxes can be wiped (preserving HDFS partitions). Boxes
would be incrementally removed from the "unsanitary" cluster, reimaged, and
then added to the "sanitary" cluster. Stupid bathroom-related jokes to be
avoided.
* Team Analytics to enumerate data processing jobs that will be running in
the intermediate period; their configurations and tooling will be reviewed.
* Analytics and Ops engineers continue to have shell access. Jobs can be
submitted and managed using the CLI tools; internal dashboards can be
accessed via SSH tunnelling. Analysts working on the cluster will be
approved for shell access on a case-by-case basis (afaik, just Evan Rosen
(full-time analyst for Grantmaking & Programs), and Stefan Petrea
(contractor for Analytics)). If more analysts desire access in the interim,
we can work it out in a case-by-case basis.
* No public, external access of any box in either zone (including proxied,
dashboards like Hue, or even static files) that hasn't gone through review.
* Analytics and Ops will work together to find a simple, acceptable
mechanism for data export.
== Next Steps ==
* Analytics puppet manfiests fully reviewed and merged into master
operations-puppet repository
** Andrew to come pow-wow before the SF Ops hackathon and buddy it up with
ops to plow through some of this.
* Schedule Architecture Review
* Rolling reimaging of all analytics boxes (including hadoop data nodes but
preserving data) implementing this "minimal viable cluster" plan.
Questions very welcome! It's entirely possible I've missed things or
mistranslated them.
Cheers,
Team Analytics
--
David Schoonover
dsc(a)wikimedia.org