A short post-mortem of the problem with missing notification data.
tl;dr: the drop is not a real one and is being fixed.

As some of you noticed, we had a sudden drop in Echo notification counts across all wikis for some categories of events as of November 7 [1].
The drop was only in the EventLogging data, not in the actual delivery of notifications on site or by mail.

Here’s what happened:

* a change was made to Echo’s instrumentation to log the revid associated with a notification [2]
* when the change got merged, the schema ID was not updated in the config file
* as a result, events started getting validated against a stale schema, which caused most types of notifications (anything except for page-linked and welcome) to fail validation

These are the next steps we’re considering

* Benny wrote a patch to bump the schema id [3] it hasn’t been pushed to production yet, but when it does we will have valid data in the MySQL log DB again (in a table called Echo_6081131)
* We have a copy of the raw events stored as JSON on stat1, we may want to restore this data and import it into the DB
* After the new log is in production, we will update the scripts generating the data dumps to query a union of tables (across multiple schema IDs). This is one of the potential limitations of using SQL as a store from which to extract EventLogging data to generate dashboards.
* We will need to figure out whether passing additional fields that are not required and not specified in a schema should invalidate an event or we should relax the validation criteria.

Dario

[1] http://ee-dashboard.wmflabs.org/graphs/frwiki_echo_category
[2] https://bugzilla.wikimedia.org/show_bug.cgi?id=46045
[3] https://gerrit.wikimedia.org/r/#/c/96901/

On Nov 21, 2013, at 2:51 PM, Dario Taraborelli <dtaraborelli@wikimedia.org> wrote:

yep, the majority of events are failing to validate against Schema:Echo since Nov 7. I’ll send a summary of our findings in a moment.

On Nov 21, 2013, at 2:00 PM, Ori Livneh <ori@wikimedia.org> wrote:

On Nov 21, 2013, at 12:50 PM, Steven Walling <swalling@wikimedia.org> wrote:

On Thu, Nov 21, 2013 at 12:49 PM, Jan Ainali <jan.ainali@wikimedia.se>

wrote:

What happened with the Notifications statistiscs on Swedish WP on Nov 8?

http://ee-dashboard.wmflabs.org/dashboards/svwiki-features

No thanks, mentions, talk, reverts or review notfications since then, only

link and system.

This problem is not unique to svwiki, you can see it also in de, it, etc.

Dario will know more I think?

On Thu, Nov 21, 2013 at 12:58 PM, Dario Taraborelli <dtaraborelli@wikimedia.org> wrote:

good catch, I’ll look into this

ERROR:root:Unable to decode: 159611 EventLogging

{"event":{"version":"1.5","eventId":1641964,"notificationType":"reverted","notificationGroup":"negative","sender":"XXX","recipientUserId":XXX,"recipientEditCount":XXX,"deliveryMethod":"web","revisionId":XXX},"schema":"Echo","revision":XXX,"clientValidated":false,"wiki":"frwiki","recvFrom":"mw1163","timestamp":1385071015,"webHost":"fr.wikipedia.org"}

ValidationError: Additional properties are not allowed (u'revisionId'

was unexpected)

(XXX = redacted)

_______________________________________________

Analytics mailing list

Analytics@lists.wikimedia.org

https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________
EE mailing list
EE@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ee