Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org
wrote:
Note if the events are firing and there are no errors in the console
then
the change was successful :) If someone can double check they are
showing up
on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari
got
the changes out successuflly, we realized neither of us knew how to
check on
stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?))
On Thu, Oct 24, 2013 at 10:37 AM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote:
Note if the events are firing and there are no errors in the console then the change was successful :) If someone can double check they are showing up on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
Is http://wikitech.wikimedia.org/wiki/EventLogging what you're looking for?
Is http://wikitech.wikimedia.org/wiki/EventLogging what you're looking for?
I think that's a good place to start, and to be explicit, from stat1 you can look at the live event logging stream like this:
zsub vanadium.eqiad.wmnet:8600
We wanted to feed this into hadoop at some point to handle larger streams. So Arthur, let us know if you need that :)
On Thu, Oct 24, 2013 at 10:46 AM, Jon Robson jrobson@wikimedia.org wrote:
Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?))
Yeah, that is correct. Wikitech is actually probably the right place for this if it's not already there somewhere.
Re-sending because it bounced the first time:
https://wikitech.wikimedia.org/wiki/EventLogging is probably what you want. I added a section on 'how do I check to see if events are coming in?'. Feel free to edit :D
On Thu, Oct 24, 2013 at 11:21 PM, Arthur Richards arichards@wikimedia.orgwrote:
On Thu, Oct 24, 2013 at 10:46 AM, Jon Robson jrobson@wikimedia.orgwrote:
Basically I think Arthur is asking how he can access the data that EventLogging collects on Wikimedia servers (stat1). I doubt there is documentation as this is Wikimedia centric (although maybe we should record this on office wiki (?))
Yeah, that is correct. Wikitech is actually probably the right place for this if it's not already there somewhere.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote: I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote:
Note if the events are firing and there are no errors in the console then the change was successful :) If someone can double check they are showing up on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D
On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto otto@wikimedia.org wrote:
So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing
- we had no idea how to inspect the pipeline, and everyone who did know was
asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.orgwrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org
wrote:
Note if the events are firing and there are no errors in the console
then
the change was successful :) If someone can double check they are
showing up
on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari
got
the changes out successuflly, we realized neither of us knew how to
check on
stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
for reference, we also have: https://office.wikimedia.org/wiki/Data_access#EventLogging
On Oct 24, 2013, at 11:42 AM, Arthur Richards arichards@wikimedia.org wrote:
Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D
On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto otto@wikimedia.org wrote: So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing - we had no idea how to inspect the pipeline, and everyone who did know was asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote: I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote:
Note if the events are firing and there are no errors in the console then the change was successful :) If someone can double check they are showing up on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Would it be useful to have a hangout / meeting and give an overview of the developer / operational aspect of EL? --- Ori Livneh ori@wikimedia.org
On Thu, Oct 24, 2013 at 11:58 AM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
for reference, we also have: https://office.wikimedia.org/wiki/Data_access#EventLogging
On Oct 24, 2013, at 11:42 AM, Arthur Richards arichards@wikimedia.org wrote:
Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D
On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto otto@wikimedia.org wrote:
So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing
- we had no idea how to inspect the pipeline, and everyone who did know was
asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote:
Note if the events are firing and there are no errors in the console then the change was successful :) If someone can double check they are showing up on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
we also have http://www.youtube.com/watch?v=ZBuTjl0lh9Y for those who missed it :)
On Oct 24, 2013, at 12:12 PM, Ori Livneh ori@wikimedia.org wrote:
Would it be useful to have a hangout / meeting and give an overview of the developer / operational aspect of EL?
Ori Livneh ori@wikimedia.org
On Thu, Oct 24, 2013 at 11:58 AM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
for reference, we also have: https://office.wikimedia.org/wiki/Data_access#EventLogging
On Oct 24, 2013, at 11:42 AM, Arthur Richards arichards@wikimedia.org wrote:
Far out, thanks everyone! And special thanks to Yuvi; that's exactly what I was looking for :D
On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto otto@wikimedia.org wrote:
So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to stat1:/a/eventlogging/archive. I see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere about how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas and how we handle schemas internally for MF and realized after deploying the changes that the best we could do was make sure that the events were firing
- we had no idea how to inspect the pipeline, and everyone who did know was
asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote:
Note if the events are firing and there are no errors in the console then the change was successful :) If someone can double check they are showing up on stat1 though even better!
Are there details published somewhere on how to do this? After Kaldari got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Heh, thanks Dario and Ori - I don't think an operational overview is necessary (unless anyone else wants it?). I understand how EventLogging works, I just had no idea how to double check whether or not events were actually getting properly logged and I did not have time to dig and try to figure it out on my own. Now that documentation has been updated and we've been pointed to additional docs, I do not expect to find myself in the same situation I did last night, and hopefully no one else will either.
On Thu, Oct 24, 2013 at 12:17 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
we also have http://www.youtube.com/watch?v=ZBuTjl0lh9Y for those who missed it :)
On Oct 24, 2013, at 12:12 PM, Ori Livneh ori@wikimedia.org wrote:
Would it be useful to have a hangout / meeting and give an overview of the developer / operational aspect of EL?
Ori Livneh ori@wikimedia.org
On Thu, Oct 24, 2013 at 11:58 AM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:
for reference, we also have: https://office.wikimedia.org/wiki/Data_access#EventLogging
On Oct 24, 2013, at 11:42 AM, Arthur Richards arichards@wikimedia.org wrote:
Far out, thanks everyone! And special thanks to Yuvi; that's exactly
what I
was looking for :D
On Thu, Oct 24, 2013 at 11:07 AM, Andrew Otto otto@wikimedia.org
wrote:
So from what I know, there is a job that copies EventLogging files get copied from somewhere (vanadium?) over to
stat1:/a/eventlogging/archive. I
see recent data files there, of approximately the same size as previous ones, so at a glance things look ok.
On Oct 24, 2013, at 1:37 PM, Arthur Richards arichards@wikimedia.org wrote:
Greetings Ori and analytics team. Is there documentation somewhere
about
how and where to determine that EventLogging events are being properly recorded? We had to do a quick deployment last night to change schemas
and
how we handle schemas internally for MF and realized after deploying
the
changes that the best we could do was make sure that the events were
firing
- we had no idea how to inspect the pipeline, and everyone who did
know was
asleep/offline/etc (see below for more backstory).
On Thu, Oct 24, 2013 at 10:21 AM, Jon Robson jrobson@wikimedia.org wrote:
I can confirm we are still logging. In terms of stat1.wikimedia.org access I forget exactly how I did it but you will need to talk to someone in analytics - maybe Dario to get setup there. I'd recommend doing this sooner rather than later to avoid this problem again.
On Wed, Oct 23, 2013 at 7:31 PM, Arthur Richards arichards@wikimedia.org wrote:
On Wed, Oct 23, 2013 at 6:57 PM, Jon Robson jrobson@wikimedia.org wrote: > > Note if the events are firing and there are no errors in the console > then > the change was successful :) If someone can double check they are > showing up > on stat1 though even better!
Are there details published somewhere on how to do this? After
Kaldari
got the changes out successuflly, we realized neither of us knew how to check on stat1 nor could I quickly find docs.
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687 _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics