EventLogging suffered from performance problems and data loss from Tuesday 2015-05-05 22:00 UTC to Wednesday 2015-05-06 20:00 UTC (22 hours).
During that period, an exceptional amount of events were sent to EL server for a given schema. The system could not handle them properly, and this caused data loss (30%-40% during the period) and some small gaps in the db. All schemas were affected.
The missing data will be backfilled during this week.
Phab Task: https://phabricator.wikimedia.org/T98588 Incident documentation: https://wikitech.wikimedia.org/wiki/Incident_documentation/20150506-EventLog...
Cheers,
Marcel
Thank you!
On Fri, May 8, 2015 at 5:12 AM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
EventLogging suffered from performance problems and data loss from Tuesday 2015-05-05 22:00 UTC to Wednesday 2015-05-06 20:00 UTC (22 hours).
During that period, an exceptional amount of events were sent to EL server for a given schema. The system could not handle them properly, and this caused data loss (30%-40% during the period) and some small gaps in the db. All schemas were affected.
The missing data will be backfilled during this week.
Phab Task: https://phabricator.wikimedia.org/T98588 Incident documentation:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150506-EventLog...
Cheers,
Marcel
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
The data for this period has been back-filled with success.
Cheers!
On Fri, May 8, 2015 at 4:23 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thank you!
On Fri, May 8, 2015 at 5:12 AM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
EventLogging suffered from performance problems and data loss from Tuesday 2015-05-05 22:00 UTC to Wednesday 2015-05-06 20:00 UTC (22 hours).
During that period, an exceptional amount of events were sent to EL server for a given schema. The system could not handle them properly, and this caused data loss (30%-40% during the period) and some small gaps in the db. All schemas were affected.
The missing data will be backfilled during this week.
Phab Task: https://phabricator.wikimedia.org/T98588 Incident documentation:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150506-EventLog...
Cheers,
Marcel
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Thanks very much.
On Tue, May 19, 2015 at 5:05 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
The data for this period has been back-filled with success.
Cheers!
On Fri, May 8, 2015 at 4:23 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thank you!
On Fri, May 8, 2015 at 5:12 AM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
EventLogging suffered from performance problems and data loss from Tuesday 2015-05-05 22:00 UTC to Wednesday 2015-05-06 20:00 UTC (22 hours).
During that period, an exceptional amount of events were sent to EL server for a given schema. The system could not handle them properly, and this caused data loss (30%-40% during the period) and some small gaps in the db. All schemas were affected.
The missing data will be backfilled during this week.
Phab Task: https://phabricator.wikimedia.org/T98588 Incident documentation:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150506-EventLog...
Cheers,
Marcel
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Indeed, thank you, Marcel!
On Sat, May 23, 2015 at 6:06 PM, Ori Livneh ori@wikimedia.org wrote:
Thanks very much.
On Tue, May 19, 2015 at 5:05 PM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
The data for this period has been back-filled with success.
Cheers!
On Fri, May 8, 2015 at 4:23 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thank you!
On Fri, May 8, 2015 at 5:12 AM, Marcel Ruiz Forns mforns@wikimedia.org wrote:
EventLogging suffered from performance problems and data loss from Tuesday 2015-05-05 22:00 UTC to Wednesday 2015-05-06 20:00 UTC (22 hours).
During that period, an exceptional amount of events were sent to EL server for a given schema. The system could not handle them properly, and this caused data loss (30%-40% during the period) and some small gaps in the db. All schemas were affected.
The missing data will be backfilled during this week.
Phab Task: https://phabricator.wikimedia.org/T98588 Incident documentation:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150506-EventLog...
Cheers,
Marcel
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics