Re: [Analytics] Danger zone: EventLogging editing data (action needed) - Analytics

Jon Robson

24 Oct 24 Oct

7:46 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?)) On Thu, Oct 24, 2013 at 10:37 AM, Arthur Richards <arichards(a)wikimedia.org> wrote:

...

Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory). On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson <jrobson(a)wikimedia.org> wrote:

I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again. On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards <arichards(a)wikimedia.org> wrote: > > On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson <jrobson(a)wikimedia.org> > wrote: >> >> Note if the events are firing and there are no errors in the console >> then >> the change was successful :) If someone can double check they are >> showing up >> on stat1 though even better! > > > Are there details published somewhere on how to do this? After Kaldari > got > the changes out successuflly, we realized neither of us knew how to > check on > stat1 nor could I quickly find docs. > > -- > Arthur Richards > Software Engineer, Mobile > [[User:Awjrichards]] > IRC: awjr > +1-415-839-6885 x6687

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Reply

Arthur Richards

7:51 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

On Thu, Oct 24, 2013 at 10:46 AM, Jon Robson <jrobson(a)wikimedia.org> wrote:

...

Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?))

Yeah, that is correct. Wikitech is actually probably the right place for this if it's not already there somewhere. -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Reply

Yuvi Panda

8:04 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

Re-sending because it bounced the first time: https://wikitech.wikimedia.org/wiki/EventLogging is probably what you want. I added a section on 'how do I check to see if events are coming in?'. Feel free to edit :D On Thu, Oct 24, 2013 at 11:21 PM, Arthur Richards <arichards(a)wikimedia.org>wrote;wrote:

...

On Thu, Oct 24, 2013 at 10:46 AM, Jon Robson <jrobson(a)wikimedia.org>wrote;wrote:

Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?))

Yeah, that is correct. Wikitech is actually probably the right place for this if it's not already there somewhere. -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Reply

Arthur Richards

8:42 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto <otto(a)wikimedia.org> wrote:

...

So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok. On Oct 24, 2013, at 1:37 PM, Arthur Richards <arichards(a)wikimedia.org> wrote: Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory). On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson <jrobson(a)wikimedia.org>wrote;wrote:

I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again. On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards <arichards(a)wikimedia.org> wrote:

On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson <jrobson(a)wikimedia.org>

wrote:

> > Note if the events are firing and there are no errors in the console

then

> the change was successful :) If someone can double check they are

showing up

on stat1 though even better!

Are there details published somewhere on how to do this? After Kaldari

got

the changes out successuflly, we realized neither of us knew how to

check on

stat1 nor could I quickly find docs. -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Reply

Ori Livneh

9:12 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

Would it be useful to have a hangout / meeting and give an overview of the developer / operational aspect of EL? --- Ori Livneh ori(a)wikimedia.org On Thu, Oct 24, 2013 at 11:58 AM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

for reference, we also have: https://office.wikimedia.org/wiki/Data_access#EventLogging On Oct 24, 2013, at 11:42 AM, Arthur Richards <arichards(a)wikimedia.org> wrote: Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto <otto(a)wikimedia.org> wrote:

So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok. On Oct 24, 2013, at 1:37 PM, Arthur Richards <arichards(a)wikimedia.org> wrote: Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory). On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson <jrobson(a)wikimedia.org> wrote:

I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again. On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards <arichards(a)wikimedia.org> wrote: > > On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson <jrobson(a)wikimedia.org> > wrote: >> >> Note if the events are firing and there are no errors in the console >> then >> the change was successful :) If someone can double check they are >> showing up >> on stat1 though even better! > > > Are there details published somewhere on how to do this? After Kaldari > got > the changes out successuflly, we realized neither of us knew how to > check on > stat1 nor could I quickly find docs. > > -- > Arthur Richards > Software Engineer, Mobile > [[User:Awjrichards]] > IRC: awjr > +1-415-839-6885 x6687

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Reply

Arthur Richards

10:12 p.m.

New subject: Danger zone: EventLogging editing data (action needed)

Heh, thanks Dario and Ori - I don't think an operational overview is necessary (unless anyone else wants it?). I understand how EventLogging works, I just had no idea how to double check whether or not events were actually getting properly logged and I did not have time to dig and try to figure it out on my own. Now that documentation has been updated and we've been pointed to additional docs, I do not expect to find myself in the same situation I did last night, and hopefully no one else will either. On Thu, Oct 24, 2013 at 12:17 PM, Dario Taraborelli < dtaraborelli(a)wikimedia.org> wrote:

...

we also have http://www.youtube.com/watch?v=ZBuTjl0lh9Y for those who missed it :) On Oct 24, 2013, at 12:12 PM, Ori Livneh <ori(a)wikimedia.org> wrote:

Would it be useful to have a hangout / meeting and give an overview of the developer / operational aspect of EL? --- Ori Livneh ori(a)wikimedia.org On Thu, Oct 24, 2013 at 11:58 AM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote: > for reference, we also have: > https://office.wikimedia.org/wiki/Data_access#EventLogging > > On Oct 24, 2013, at 11:42 AM, Arthur Richards <arichards(a)wikimedia.org> > wrote: > > Far out, thanks everyone! And special thanks to Yuvi; that's exactly

what I

> was looking for :D > > > On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto <otto(a)wikimedia.org>

wrote:

>> >> So from what I know, there is a job that copies EventLogging files get >> copied from somewhere (vanadium?) over to

stat1:/a/eventlogging/archive. I

>> see recent data files there, of approximately the same size as previous >> ones, so at a glance things look ok. >> >> >> On Oct 24, 2013, at 1:37 PM, Arthur Richards <arichards(a)wikimedia.org> >> wrote: >> >> Greetings Ori and analytics team. Is there documentation somewhere

about

>> how and where to determine that EventLogging events are being properly >> recorded? We had to do a quick deployment last night to change schemas

and

>> how we handle schemas internally for MF and realized after deploying

the

>> changes that the best we could do was make sure that the events were

firing

>> - we had no idea how to inspect the pipeline, and everyone who did

know was

>> asleep/offline/etc (see below for more backstory). >> >> >> On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson <jrobson(a)wikimedia.org> >> wrote: >>> >>> I can confirm we are still logging. >>> In terms of stat1.wikimedia.org access I forget exactly how I did it >>> but you will need to talk to someone in analytics - maybe Dario to get >>> setup there. I'd recommend doing this sooner rather than later to >>> avoid this problem again. >>> >>> On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards >>> <arichards(a)wikimedia.org> wrote: >>>> >>>> On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson <jrobson(a)wikimedia.org> >>>> wrote: >>>>> >>>>> Note if the events are firing and there are no errors in the console >>>>> then >>>>> the change was successful :) If someone can double check they are >>>>> showing up >>>>> on stat1 though even better! >>>> >>>> >>>> Are there details published somewhere on how to do this? After

Kaldari

>> got >> the changes out successuflly, we realized neither of us knew how to >> check on >> stat1 nor could I quickly find docs. >> >> -- >> Arthur Richards >> Software Engineer, Mobile >> [[User:Awjrichards]] >> IRC: awjr >> +1-415-839-6885 x6687 -- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687

Reply