Eventlogging outage

List overview All Threads
Download

newer

older

IdeaLab Grant Call for Participants

Re: [Analytics] stats.grok.se not...

Nuria Ruiz

8 Apr 2015 8 Apr '15

7:28 p.m.

Attachments:

attachment.htm (text/html — 635 bytes)

Show replies by date

Aaron Halfaker

8 Apr 8 Apr

7:37 p.m.

Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote:

...

Team: As you might know we have swapped EL old vanadium box to a a never, more resilient one. This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium. Details can be found here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLo… Thanks, Nuria _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Dan Andreescu

7:40 p.m.

It did cause data loss, and we can not backfill because the disk was full so the logs were not written. On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote:

...

Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote:

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Kevin Leduc

7:45 p.m.

the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLo… On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote:

...

It did cause data loss, and we can not backfill because the disk was full so the logs were not written. On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote:

Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote:

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Aaron Halfaker

8:13 p.m.

Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails. On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc <kevin(a)wikimedia.org> wrote:

...

It did cause data loss, and we can not backfill because the disk was full so the logs were not written. On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote:

Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote:

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Dario Taraborelli

9 Apr 9 Apr

12:42 a.m.

to clarify: does this affect all logs or client-side logs only?

...

On Apr 8, 2015, at 11:13 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote: Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails. On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc <kevin(a)wikimedia.org <mailto:kevin@wikimedia.org>> wrote: the data loss and no-backfilling are documented in the incident report https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLo… <https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLogging#Actionables> On Wed, Apr 8, 2015 at 10:40 AM, Dan Andreescu <dandreescu(a)wikimedia.org <mailto:dandreescu@wikimedia.org>> wrote: It did cause data loss, and we can not backfill because the disk was full so the logs were not written. On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker(a)wikimedia.org <mailto:ahalfaker@wikimedia.org>> wrote: Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org <mailto:nuria@wikimedia.org>> wrote: Team: As you might know we have swapped EL old vanadium box to a a never, more resilient one. This new box had less disk space and the move caused a small outage due to a bug already present on EL code that was not apparent on vanadium. Details can be found here: https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLo… <https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLogging> Thanks, Nuria _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics> _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics> _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics> _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics> _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Marcel Ruiz Forns

12:47 a.m.

Dario, All kinds of event logs were affected. I updated the documentation. On Thu, Apr 9, 2015 at 12:42 AM, Dario Taraborelli < dtaraborelli(a)wikimedia.org> wrote:

...

to clarify: does this affect all logs or client-side logs only? On Apr 8, 2015, at 11:13 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote: Thanks guys. As a frequent user of event logging, the dataloss and potential backfilling are of great importance to me. It would be helpful for me if, in the future, these could be summarized in announcement emails. On Wed, Apr 8, 2015 at 12:45 PM, Kevin Leduc <kevin(a)wikimedia.org> wrote:

It did cause data loss, and we can not backfill because the disk was full so the logs were not written. On Wed, Apr 8, 2015 at 1:37 PM, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote:

Thanks Nuria. Did this cause data loss and if so, is there a plan to backfill? -Aaron On Wed, Apr 8, 2015 at 12:28 PM, Nuria Ruiz <nuria(a)wikimedia.org> wrote: > Team: > > As you might know we have swapped EL old vanadium box to a a never, > more resilient one. > > This new box had less disk space and the move caused a small outage > due to a bug already present on EL code that was not apparent on vanadium. > > Details can be found here: > > > https://wikitech.wikimedia.org/wiki/Incident_documentation/20150406-EventLo… > > Thanks, > > Nuria > > _______________________________________________ > Analytics mailing list > Analytics(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > > _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

3305

days inactive

3305

days old

analytics@lists.wikimedia.org

Manage subscription

6 comments

6 participants

tags (0)

participants (6)

Aaron Halfaker
Dan Andreescu
Dario Taraborelli
Kevin Leduc
Marcel Ruiz Forns
Nuria Ruiz