Hello!
I just finished writing a short incident report on the maps issue of
last Friday [1].
Basically, I was stupid and reinitialized Cassandra on the wrong node.
This should have had almost no effect, except that the system_auth
keyspace of Cassandra is configured with a replication factor of 1.
Loosing that node means that we also lost authentication.
Thanks a lot to Brandon for the help in mitigating this!
[1]
https://wikitech.wikimedia.org/wiki/Incident_documentation/20161021-Maps
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST