I think it would be great if you could add some documentation to that wiki

page about the various files in >the /var/log/eventlogging folder. Without documenting the whole complexity of EL, I have described a bit more how to check for valid events: https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs#How_to_che...

• Is my assumption correct that we devs can no longer access production EL

data? Well, access to 1002 is no more or less restricted so that hasn't changed. The idea is to test in the beta cluster is to test before you send your code to prod. This is specially easy on apps as you just need to change the endpoint and you can get to try your new code. To troubleshoot prod issues you would still do it in 1002. So developers that need to troubleshoot production issues would need access to that machine just the same.

Also I think it would be useful to restructure the page based on types of

consumers of the page: one section on top for event logging >users, and another one for admins below. In other words I think the Database and "Raw stream of events" sections could be moved >up, and then everything afterwards is admin stuff. Please change as you see fit.

Thanks,

Nuria

On Mon, Apr 20, 2015 at 5:29 PM, Bernd Sitzmann bernd@wikimedia.org wrote:

Hi Nuria,

Thank you for proofreading this. I clearly misunderstood some of the files.

I think it would be great if you could add some documentation to that wiki page about the various files in the /var/log/eventlogging folder. It would be nice to sort them in order messages flow (unvalidated -> validated, ...). Also I think it would be useful to restructure the page based on types of consumers of the page: one section on top for event logging users, and another one for admins below. In other words I think the Database and "Raw stream of events" sections could be moved up, and then everything afterwards is admin stuff.

Here are a few more questions in the mean-time:

• Is my assumption correct that we devs can no longer access production EL

data?

• If so, do we client side devs even need access to stat1002/1003? (Or

what would it be useful for?) I'm asking because this is one of the onboarding steps we have for new developers, and next week we'll have another dev joining our team.

Thanks, Bernd

On Mon, Apr 20, 2015 at 5:50 PM, Nuria Ruiz nuria@wikimedia.org wrote:

...

(cc-ing analytics list)

...

Heads up, that the way we test Event Logging has changed. Details can be found here:

https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs Correct, analytics team will take care of maintaining these docs.

...

ssh eventlogging02.eqiad.wmflabs cd /var/log/eventlogging tail -f client-side-events.log

This will give you the WHOLE stream of client side events, correct, valid and invalid.

...

For unvalidated events check out the file all-events.log instead.

This log holds VALID events both client and server side.

...

2. Modify event logging code in client code

In Android app uncomment the following line in EventLoggingEvent // private static final String EVENTLOG_URL = "

http://bits.beta.wmflabs.org/event.gif"; Correct

...

3. Run your code, and see the output from step 1.

Correct

Thanks for testing before sending your code to prod!

Nuria

On Mon, Apr 20, 2015 at 2:03 PM, Bernd Sitzmann bernd@wikimedia.org wrote:

...

Hi Nuria,

I'm about to send this off to the mobile-tech list. But before I do that, is this info correct?

Thanks, Bernd

---

Heads up, that the way we test Event Logging has changed. Details can be found here: https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs

ssh eventlogging02.eqiad.wmflabs cd /var/log/eventlogging tail -f client-side-events.log

For unvalidated events check out the file all-events.log instead.

If you need access check out https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs#Give_peopl... .

2. Modify event logging code in client code

In Android app uncomment the following line in EventLoggingEvent // private static final String EVENTLOG_URL = " http://bits.beta.wmflabs.org/event.gif";

Please don't commit that change to keep the production value as is.

3. Run your code, and see the output from step 1.

As an aside:

Confused about beta and labs, like I was? Check out https://wikitech.wikimedia.org/wiki/Labs_labs_labs. For some reason when I joined it was called beta labs and that stuck with me, and for a lot of others, like the creator of the page https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs.

Cheers, Bernd

I've restructured the Wiki page in a way which is easier to understand

from my POV.Please check if it's still correct. Looks good, thank you for doing the changes.

RE:stat1002. I've never used it before. I've only used stat1003 to verify

event logging using zsub. I never used 1003 as I do not think I have permits there. The same set of logs for EL present in the beta cluster are available (for production data) in 1002 at a given location for some months. Please take a look: https://wikitech.wikimedia.org/wiki/Stat1002

On Mon, Apr 20, 2015 at 10:46 PM, Bernd Sitzmann bernd@wikimedia.org wrote:

Nuria,

I've restructured the Wiki page in a way which is easier to understand from my POV. Please check if it's still correct. The part I still was a bit confused about is https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs#Raw_stream...

If events do not appear they might not be valid, check /var/log/upstart/ for

...

either the server-side or the client-side processor logs. Validation errors will appear on those logs and they are very descriptive.

I don't think this is correct but I don't know which files you are actually talking about, how to verify event validation.

RE:stat1002. I've never used it before. I've only used stat1003 to verify event logging using zsub. Maybe you or someone else could show me sometimes how to actually verify event logging in production?

Thanks, Bernd