Eventlogging for editing broken

List overview All Threads
Download

newer

older

Re: [Analytics] [Multimedia] ...

Survey tool for features

Jon Robson

25 Apr 2014 25 Apr '14

4:06 p.m.

Some of the graphs [1] on the report card are not rendering due to what seems like some sort of EventLogging outage When I query the database I get errors such as ""MySQL said: Table 'MobileWebEditing_7675117' is marked as crashed and should be repaired"" Let's keep an eye on this... [1] http://mobile-reportcard.wmflabs.org/#edits_daily-graphs-tab

Show replies by date

Jon Robson

25 Apr 25 Apr

4:08 p.m.

See https://bugzilla.wikimedia.org/show_bug.cgi?id=64445 On Fri, Apr 25, 2014 at 1:06 PM, Jon Robson <jrobson(a)wikimedia.org> wrote:

...

Ori Livneh

6:57 p.m.

Sean sent this update earlier (before Jon noticed the issue) and agreed to let me forward it here.

...

db1047 has been upgraded to MariaDB 10, however it is not yet replicating from m2 because I ran into a replication bug on db1046, the m2 slave I had intended to make db1047's master for the log db. Until I get that glitch sorted I have db1047 federating the log tables instead using the new CONNECT engine, which seems fine so far.

The data on db1048 looks healthy (and is continuing to accumulate incoming events), which supports Dan's hunch that the issue lies with replication. So there is still nothing that suggests data loss. But once the migration is over, we should cross-reference the database with the log files and confirm this definitively. On Fri, Apr 25, 2014 at 1:08 PM, Jon Robson <jrobson(a)wikimedia.org> wrote:

...

See https://bugzilla.wikimedia.org/show_bug.cgi?id=64445 On Fri, Apr 25, 2014 at 1:06 PM, Jon Robson <jrobson(a)wikimedia.org> wrote:

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Maryana Pinchuk

30 Apr 30 Apr

5:09 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

Are there any updates on this from Sean? It's not super critical at the moment; I just want to know what's going on and when we might expect the graphs and data to return to normal :) On Fri, Apr 25, 2014 at 3:57 PM, Ori Livneh <ori(a)wikimedia.org> wrote:

...

Sean sent this update earlier (before Jon noticed the issue) and agreed to let me forward it here.

db1047 has been upgraded to MariaDB 10, however it is not yet replicating from m2 because I ran into a replication bug on db1046, the m2 slave I

had

intended to make db1047's master for the log db. Until I get that glitch sorted I have db1047 federating the log tables instead using the new CONNECT engine, which seems fine so far.

See https://bugzilla.wikimedia.org/show_bug.cgi?id=64445 On Fri, Apr 25, 2014 at 1:06 PM, Jon Robson <jrobson(a)wikimedia.org> wrote:

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org

Dan Andreescu

5:19 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

...

Are there any updates on this from Sean? It's not super critical at the moment; I just want to know what's going on and when we might expect the graphs and data to return to normal :)

Arthur Richards

5:22 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

Is there a bug/ticket tracking this other than the one filed under MobileFrontend? If not, perhaps we should reassign the MobileFrontend bug to a more appropriate place. On Wed, Apr 30, 2014 at 2:19 PM, Dan Andreescu <dandreescu(a)wikimedia.org>wrote;wrote:

...

Are there any updates on this from Sean? It's not super critical at the

moment; I just want to know what's going on and when we might expect the graphs and data to return to normal :)

The latest I know is that the mobile-reportcard dashboards are updating again. That means db1047/log is ok to query. From what I can see there are data gaps but maybe Ori can shed more light on that. I'll try pinging him _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Dan Andreescu

5:28 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

...

Is there a bug/ticket tracking this other than the one filed under MobileFrontend? If not, perhaps we should reassign the MobileFrontend bug to a more appropriate place.

We should report a bug for every separate issue, and I think this one is "Limn Dashboards not working since EventLogging migration".

Jon Robson

6:05 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

There also do not seem to be any new entries being created in MobileWebUploads_8209043 which is slightly concerning... On Wed, Apr 30, 2014 at 2:28 PM, Dan Andreescu <dandreescu(a)wikimedia.org> wrote:

...

Is there a bug/ticket tracking this other than the one filed under MobileFrontend? If not, perhaps we should reassign the MobileFrontend bug to a more appropriate place.

We should report a bug for every separate issue, and I think this one is "Limn Dashboards not working since EventLogging migration". _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l

-- Jon Robson * http://jonrobson.me.uk * https://www.facebook.com/jonrobson * @rakugojon

Steven Walling

6:11 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

On Wed, Apr 30, 2014 at 3:05 PM, Jon Robson <jdlrobson(a)gmail.com> wrote:

...

There also do not seem to be any new entries being created in MobileWebUploads_8209043 which is slightly concerning...

Jon if you're looking on the usual log databse this is likely just the ~30 hours of lag. EventLogging was recently switched over to write to db1048 instead, and then federate to db1047 for analysis purposes (if I'm describing that correctly). -- Steven Walling, Product Manager https://wikimediafoundation.org/

Jon Robson

6:17 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

Toby Negrin

6:23 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

Let's give the database the time it needs to replicate and perform needed validation before we start troubleshooting other issues. I'm concerned that too many things are going on here. Thanks to everyone who is working on this right now. -Toby On Wed, Apr 30, 2014 at 3:17 PM, Jon Robson <jdlrobson(a)gmail.com> wrote:

...

I've not seen any events logged on this table since 26th April. I know for sure there have been some since then as I triggered some on the 27th (I saw them in the network tab)... I'm using stat1003 I just want to rule out schema validation errors. _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Ori Livneh

6:27 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

On Wed, Apr 30, 2014 at 3:05 PM, Jon Robson <jdlrobson(a)gmail.com> wrote:

...

There also do not seem to be any new entries being created in MobileWebUploads_8209043 which is slightly concerning...

I see entries in db1048. So, again: this is temporary lag in replication due to the setup being new.

Jon Robson

6:29 p.m.

New subject: [WikimediaMobile] Eventlogging for editing broken

Ori, thanks for checking and the reassurance and thanks for sorting this out guys! Keep us updated! On Wed, Apr 30, 2014 at 3:27 PM, Ori Livneh <ori(a)wikimedia.org> wrote:

...

On Wed, Apr 30, 2014 at 3:05 PM, Jon Robson <jdlrobson(a)gmail.com> wrote:

There also do not seem to be any new entries being created in MobileWebUploads_8209043 which is slightly concerning...

I see entries in db1048. So, again: this is temporary lag in replication due to the setup being new.

-- Jon Robson * http://jonrobson.me.uk * https://www.facebook.com/jonrobson * @rakugojon

3647

days inactive

3652

days old

analytics@lists.wikimedia.org

Manage subscription

12 comments

8 participants

tags (0)

participants (8)

Arthur Richards
Dan Andreescu
Jon Robson
Jon Robson
Maryana Pinchuk
Ori Livneh
Steven Walling
Toby Negrin