Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
It did cause data loss, and we can not backfill because the disk was full so the logs were not written.
On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
It did cause data loss, and we can not backfill because the disk was full so the logs were not written.
On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails.
On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc kevin@wikimedia.org wrote:
the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
It did cause data loss, and we can not backfill because the disk was full so the logs were not written.
On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
to clarify: does this affect all logs or client-side logs only?
On Apr 8, 2015, at 11:13 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails.
On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc <kevin@wikimedia.org mailto:kevin@wikimedia.org> wrote: the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog... https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLogging#Actionables
On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu <dandreescu@wikimedia.org mailto:dandreescu@wikimedia.org> wrote: It did cause data loss, and we can not backfill because the disk was full so the logs were not written.
On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker@wikimedia.org mailto:ahalfaker@wikimedia.org> wrote: Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria@wikimedia.org mailto:nuria@wikimedia.org> wrote: Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog... https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLogging
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Dario,
All kinds of event logs were affected. I updated the documentation.
On Thu, Apr 9, 2015 at 12:42 AM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
to clarify: does this affect all logs or client-side logs only?
On Apr 8, 2015, at 11:13 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails.
On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc kevin@wikimedia.org wrote:
the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu dandreescu@wikimedia.org wrote:
It did cause data loss, and we can not backfill because the disk was full so the logs were not written.
On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks Nuria.
Did this cause data loss and if so, is there a plan to backfill?
-Aaron
On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Team:
As you might know we have swapped EL old vanadium box to a a never, more resilient one.
This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium.
Details can be found here:
https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLog...
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics