Hi,
TL;DR: When consuming EventLogging data, only rely on the 'log' database available from m2 replicas, like analytics-store.eqiad.wmnet.
Other representations might not get updated, might not get fix-ups or may (on purpose) give you unvalidated data.
----------------------------------
Due to the versatile design of EventLogging, its data exists/existed in many different representations, which got me confused around the data quality expectations. Also I could not find them publicly documented. After talking about different aspects with a few people, I wanted to put my current understanding of it up for public discussion.
Please let me know (either in private or on list), if something looks wrong or does not match your use of EventLogging data.
* MySQL / MariaDB database on m2
This database is the best place to consume EventLogging data from.
Available as 'log' database on m2 replicas, such as analytics-store.eqiad.wmnet.
Only validated events enter the database.
In case of bugs, this database is the only place that gets fixes like cleanup of historic data, or live fixes.
* 'all-events' JSON log files [1]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are JSON objects.
Only validated events get written.
In case of bugs, historic data does not get fixed.
* Raw client and server side log files [2]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are parameters to the event.gif's request. They are not decoded at all.
In case of bugs, historic data does not get fixed. Neither need hot-fixes reach those files.
* Kafka: EventLogging data is no longer fed into Kafka since 2014-06-12 [3]. The EventLogging data in Kafka had no users. Turning it on again is tracked in bug 66528 [4].
* MongoDB: EventLogging data is no longer fed into MongoDB since 2014-02-13 [5]. The EventLogging data in MongoDB did not appear to get used. I am not aware of plans to revive feeding the data into MongoDB.
* ZMQ: ZMQ is available from vanadium. In case of bugs, historic data cannot get fixed :-) Data coming from the forwarders (ports 8421, 8422) is not validated and need not see hot-fixes. Data coming from processors (port 8521, 8522) and multiplexer (port 8600) is validated.
Have fun, Christian
[1] Available as stats1002:/a/eventlogging/archive/all-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/all-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[2] Available as stats1002:/a/eventlogging/archive/client-side-events.log-$DATE.gz stats1002:/a/eventlogging/archive/server-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/client-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/server-side-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[3] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/f85b1dbcd61bbb5...
[4] https://bugzilla.wikimedia.org/show_bug.cgi?id=66528
[5] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/05b4027973c59b0...
Thanks for such comprehensive notes!
On Friday, June 20, 2014, Christian Aistleitner christian@quelltextlich.at wrote:
Hi,
TL;DR: When consuming EventLogging data, only rely on the 'log' database available from m2 replicas, like analytics-store.eqiad.wmnet.
Other representations might not get updated, might not get fix-ups or may (on purpose) give you unvalidated data.
Due to the versatile design of EventLogging, its data exists/existed in many different representations, which got me confused around the data quality expectations. Also I could not find them publicly documented. After talking about different aspects with a few people, I wanted to put my current understanding of it up for public discussion.
Please let me know (either in private or on list), if something looks wrong or does not match your use of EventLogging data.
- MySQL / MariaDB database on m2
This database is the best place to consume EventLogging data from.
Available as 'log' database on m2 replicas, such as analytics-store.eqiad.wmnet.
Only validated events enter the database.
In case of bugs, this database is the only place that gets fixes like cleanup of historic data, or live fixes.
- 'all-events' JSON log files [1]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are JSON objects.
Only validated events get written.
In case of bugs, historic data does not get fixed.
- Raw client and server side log files [2]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are parameters to the event.gif's request. They are not decoded at all.
In case of bugs, historic data does not get fixed. Neither need hot-fixes reach those files.
- Kafka:
EventLogging data is no longer fed into Kafka since 2014-06-12 [3]. The EventLogging data in Kafka had no users. Turning it on again is tracked in bug 66528 [4].
- MongoDB:
EventLogging data is no longer fed into MongoDB since 2014-02-13 [5]. The EventLogging data in MongoDB did not appear to get used. I am not aware of plans to revive feeding the data into MongoDB.
- ZMQ:
ZMQ is available from vanadium. In case of bugs, historic data cannot get fixed :-) Data coming from the forwarders (ports 8421, 8422) is not validated and need not see hot-fixes. Data coming from processors (port 8521, 8522) and multiplexer (port 8600) is validated.
Have fun, Christian
[1] Available as stats1002:/a/eventlogging/archive/all-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/all-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[2] Available as stats1002:/a/eventlogging/archive/client-side-events.log-$DATE.gz stats1002:/a/eventlogging/archive/server-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/client-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/server-side-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[3] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/f85b1dbcd61bbb5...
[4] https://bugzilla.wikimedia.org/show_bug.cgi?id=66528
[5] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/05b4027973c59b0...
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at javascript:; 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Hi,
keeping archives connected.
The below email around different data representations of EventLogging data got wikified at
https://www.mediawiki.org/wiki/Extension:EventLogging/Data_representations
and will be maintained there.
Have fun, Christian
On Fri, Jun 20, 2014 at 03:28:10PM +0200, Christian Aistleitner wrote:
Hi,
TL;DR: When consuming EventLogging data, only rely on the 'log' database available from m2 replicas, like analytics-store.eqiad.wmnet.
Other representations might not get updated, might not get fix-ups or may (on purpose) give you unvalidated data.
Due to the versatile design of EventLogging, its data exists/existed in many different representations, which got me confused around the data quality expectations. Also I could not find them publicly documented. After talking about different aspects with a few people, I wanted to put my current understanding of it up for public discussion.
Please let me know (either in private or on list), if something looks wrong or does not match your use of EventLogging data.
- MySQL / MariaDB database on m2
This database is the best place to consume EventLogging data from.
Available as 'log' database on m2 replicas, such as analytics-store.eqiad.wmnet.
Only validated events enter the database.
In case of bugs, this database is the only place that gets fixes like cleanup of historic data, or live fixes.
- 'all-events' JSON log files [1]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are JSON objects.
Only validated events get written.
In case of bugs, historic data does not get fixed.
- Raw client and server side log files [2]
Use this data source only to debug issues around ingestion into the m2 database.
Entries are parameters to the event.gif's request. They are not decoded at all.
In case of bugs, historic data does not get fixed. Neither need hot-fixes reach those files.
- Kafka:
EventLogging data is no longer fed into Kafka since 2014-06-12 [3]. The EventLogging data in Kafka had no users. Turning it on again is tracked in bug 66528 [4].
- MongoDB:
EventLogging data is no longer fed into MongoDB since 2014-02-13 [5]. The EventLogging data in MongoDB did not appear to get used. I am not aware of plans to revive feeding the data into MongoDB.
- ZMQ:
ZMQ is available from vanadium. In case of bugs, historic data cannot get fixed :-) Data coming from the forwarders (ports 8421, 8422) is not validated and need not see hot-fixes. Data coming from processors (port 8521, 8522) and multiplexer (port 8600) is validated.
Have fun, Christian
[1] Available as stats1002:/a/eventlogging/archive/all-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/all-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[2] Available as stats1002:/a/eventlogging/archive/client-side-events.log-$DATE.gz stats1002:/a/eventlogging/archive/server-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/client-side-events.log-$DATE.gz stats1003:/srv/eventlogging/archive/server-side-events.log-$DATE.gz vanadium:/var/log/eventlogging/...
[3] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/f85b1dbcd61bbb5...
[4] https://bugzilla.wikimedia.org/show_bug.cgi?id=66528
[5] https://git.wikimedia.org/commitdiff/operations%2Fpuppet.git/05b4027973c59b0...
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics