Hi all you EventLogging users out there!

tl;dr

We will switch EventLogging MySQL ingestion to be based on a schema whitelist rather than blacklist.

As you know, we currently import EventLogging events into two locations for analysis: The MySQL ‘log’ database, and the Hive ‘event’ database.  MySQL is not able to handle high volume events.  We currently blacklist any schemas that we know have high volumes from being ingested into MySQL.  This can cause problems when a new high volume schema is deployed,  as it requires knowledge and communication from the schema owners to the Analytics team, and it requires an Analytics Operations engineer to make a Puppet commit to blacklist the schema.  To address this problem, we will switch the EventLogging MySQL schema blacklist to a whitelist.  All schemas that are actively being ingested into MySQL today will be whitelisted.  In the future, if you want an event schema to be ingested into MySQL, you’ll need to ask the Analytics team to whitelist it.

Hive has been working for EventLogging analysis for a while now.  It has almost all of the schemas that MySQL does, plus the high volume ones.  One day in the (distant?) future, we’d like to decommission MySQL storage of events. (Don’t worry yet, MySQL decommissioning has a lot of blockers and this work is not planned.)  By not ingesting events into MySQL by default, we hope to encourage more users to switch to Hive.

This switch to a whitelist will happen this week.  If you are deploying a new schema and expect it to show up in MySQL, let us know so we can whitelist it!

Thanks!

-Andrew + the Analytics Engineering team