It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Ah, sorry, I was looking on the wrong server (deployment-bastion). Thanks!
On Wed, Jan 7, 2015 at 1:21 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Looks like I've lost permission to view those logs on Beta Labs again. Any chance you could fix them? Also, was any progress ever made on piping the live cluster errors into a dedicated log with easy access? I know it sounds like it wouldn't be that useful, but we have actually had cases where server-side EventLogging was failing on en.wiki, but working on Beta Labs and locally. It would also be useful for catching obscure failures that only happen for edge cases.
Kaldari
On Wed, Jan 7, 2015 at 1:27 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Ah, sorry, I was looking on the wrong server (deployment-bastion). Thanks!
On Wed, Jan 7, 2015 at 1:21 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Kaldari:
Looks like I've lost permission to view those logs on Beta Labs again. Any
chance you could fix them? I think it will be best for you to ask for sudo on the labs machine in question, which is what I did in order not to run into these issues, folks in the labs channel should be able to help you.
Also, was any progress ever made on piping the live cluster errors into a
dedicated log with easy access? Outcome of the meeting we had on this regard was for researchers and interested parties ask for access to vanadium (which if it is declined might mean we need to sync the EL logs from vanadium more frequently to stat1002)
On Fri, Jan 9, 2015 at 5:49 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Looks like I've lost permission to view those logs on Beta Labs again. Any chance you could fix them? Also, was any progress ever made on piping the live cluster errors into a dedicated log with easy access? I know it sounds like it wouldn't be that useful, but we have actually had cases where server-side EventLogging was failing on en.wiki, but working on Beta Labs and locally. It would also be useful for catching obscure failures that only happen for edge cases.
Kaldari
On Wed, Jan 7, 2015 at 1:27 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Ah, sorry, I was looking on the wrong server (deployment-bastion). Thanks!
On Wed, Jan 7, 2015 at 1:21 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Sun, Jan 11, 2015 at 5:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Kaldari:
Looks like I've lost permission to view those logs on Beta Labs again.
Any chance you could fix them? I think it will be best for you to ask for sudo on the labs machine in question, which is what I did in order not to run into these issues, folks in the labs channel should be able to help you.
It seems a bit silly for me to ask for sudo just so I can occasionally look at some logs, but I'll go ahead and do that.
Also, was any progress ever made on piping the live cluster errors into
a dedicated log with easy access? Outcome of the meeting we had on this regard was for researchers and interested parties ask for access to vanadium (which if it is declined might mean we need to sync the EL logs from vanadium more frequently to stat1002)
I don't remember that being the outcome of our meeting. I don't even have access to stat1002, much less vanadium. I only have access to stat1003. I'll try asking for access to those servers as well, but ideally I would like for all the developers on my team to have access to an EventLogging error log for the cluster.
Kaldari
On Fri, Jan 9, 2015 at 5:49 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Looks like I've lost permission to view those logs on Beta Labs again. Any chance you could fix them? Also, was any progress ever made on piping the live cluster errors into a dedicated log with easy access? I know it sounds like it wouldn't be that useful, but we have actually had cases where server-side EventLogging was failing on en.wiki, but working on Beta Labs and locally. It would also be useful for catching obscure failures that only happen for edge cases.
Kaldari
On Wed, Jan 7, 2015 at 1:27 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Ah, sorry, I was looking on the wrong server (deployment-bastion). Thanks!
On Wed, Jan 7, 2015 at 1:21 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Sun, Jan 11, 2015 at 7:54 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
On Sun, Jan 11, 2015 at 5:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Kaldari:
Looks like I've lost permission to view those logs on Beta Labs again.
Any chance you could fix them? I think it will be best for you to ask for sudo on the labs machine in question, which is what I did in order not to run into these issues, folks in the labs channel should be able to help you.
It seems a bit silly for me to ask for sudo just so I can occasionally look at some logs, but I'll go ahead and do that.
I agree that this does not make sense. Can you describe what would be ideal, from your perspective? How would you like to access this information?
It seems a bit silly for me to ask for sudo just so I can occasionally
look at some logs, but I'll go ahead and do that. Much agree, it is unfortunate that is the state of affairs.
I don't remember that being the outcome of our meeting. I don't even have
access to stat1002, much less vanadium. I only >have access to stat1003. I'll try asking for access to those servers as well, but ideally I would like for all the developers on my >team to have access to an EventLogging error log for the cluster.
mmmm.. that is why I think we should always have meeting notes, if someone that was also on that meeting can chime in it will be great. I might be misremembering this.
Thank you,
Nuria
On Sun, Jan 11, 2015 at 8:51 PM, Ori Livneh ori@wikimedia.org wrote:
On Sun, Jan 11, 2015 at 7:54 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
On Sun, Jan 11, 2015 at 5:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Kaldari:
Looks like I've lost permission to view those logs on Beta Labs again.
Any chance you could fix them? I think it will be best for you to ask for sudo on the labs machine in question, which is what I did in order not to run into these issues, folks in the labs channel should be able to help you.
It seems a bit silly for me to ask for sudo just so I can occasionally look at some logs, but I'll go ahead and do that.
I agree that this does not make sense. Can you describe what would be ideal, from your perspective? How would you like to access this information?
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I don't remember that being the outcome of our meeting. I don't even have
access to stat1002, much less vanadium. I only >have access to stat1003. I'll try asking for access to those servers as well, but ideally I would like for all the developers on my >team to have access to an EventLogging error log for the cluster.
Ok, Christian set me straight and seems like requesting access to vanadium was a suggestion to researchers only, I am not so sure how keen will ops be on that but that was the meeting outcome.
For developers such as yourself I have documented the current methods of access to EL logs in wikitech: https://wikitech.wikimedia.org/wiki/EventLogging#Troubleshooting_events_comi...
I understand that these are not ideal but I think they should be sufficient, at this time this is what we can offer you. If these methods of access are not sufficient please let us know and drive the work with Kevin.
On Mon, Jan 12, 2015 at 8:01 AM, Nuria Ruiz nuria@wikimedia.org wrote:
It seems a bit silly for me to ask for sudo just so I can occasionally
look at some logs, but I'll go ahead and do that. Much agree, it is unfortunate that is the state of affairs.
I don't remember that being the outcome of our meeting. I don't even
have access to stat1002, much less vanadium. I only >have access to stat1003. I'll try asking for access to those servers as well, but ideally I would like for all the developers on my >team to have access to an EventLogging error log for the cluster.
mmmm.. that is why I think we should always have meeting notes, if someone that was also on that meeting can chime in it will be great. I might be misremembering this.
Thank you,
Nuria
On Sun, Jan 11, 2015 at 8:51 PM, Ori Livneh ori@wikimedia.org wrote:
On Sun, Jan 11, 2015 at 7:54 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
On Sun, Jan 11, 2015 at 5:28 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Kaldari:
Looks like I've lost permission to view those logs on Beta Labs
again. Any chance you could fix them? I think it will be best for you to ask for sudo on the labs machine in question, which is what I did in order not to run into these issues, folks in the labs channel should be able to help you.
It seems a bit silly for me to ask for sudo just so I can occasionally look at some logs, but I'll go ahead and do that.
I agree that this does not make sense. Can you describe what would be ideal, from your perspective? How would you like to access this information?
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hey Ryan,
I want to make sure we address your needs as best we can so I am following up on this.
Access permisions What's the machine and the log you're trying to access (please be specific, I am not a developer so assume I know very little). I'll pass this on to Ops so they can have a look at why permissions changed. This shouldn't happen.
Piping Events that fail validation. It's in our backlog, it's relatively high priority, but not high enough for us to have tasked it out yet or committed it to a sprint. That'll happen in February.
On Fri, Jan 9, 2015 at 5:49 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Looks like I've lost permission to view those logs on Beta Labs again. Any chance you could fix them? Also, was any progress ever made on piping the live cluster errors into a dedicated log with easy access? I know it sounds like it wouldn't be that useful, but we have actually had cases where server-side EventLogging was failing on en.wiki, but working on Beta Labs and locally. It would also be useful for catching obscure failures that only happen for edge cases.
Kaldari
On Wed, Jan 7, 2015 at 1:27 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Ah, sorry, I was looking on the wrong server (deployment-bastion). Thanks!
On Wed, Jan 7, 2015 at 1:21 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Ahem.... they are there:
nuria@deployment-eventlogging02:/var/log/upstart$ ls eventlogging_*log eventlogging_processor-client-side-events.log eventlogging_processor-server-side-events.log
On Wed, Jan 7, 2015 at 12:57 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
It seems the EventLogging logs have disappeared from /var/log/upstart/ on Beta Labs (deployment-bastion). Does anyone know where they are now?
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Jan 13, 2015 at 2:50 PM, Kevin Leduc kevin@wikimedia.org wrote:
Access permisions What's the machine and the log you're trying to access (please be specific, I am not a developer so assume I know very little). I'll pass this on to Ops so they can have a look at why permissions changed. This shouldn't happen.
The logs are /var/log/upstart/eventlogging_processor-client-side-events.log and /var/log/upstart/eventlogging_processor-server-side-events.log on deployment-eventlogging02. I have now been given sudo on this machine, so it isn't an issue anymore for me personally, but it would be great if other developers could access these logs for debugging purposes.
Piping Events that fail validation. It's in our backlog, it's relatively high priority, but not high enough for us to have tasked it out yet or committed it to a sprint. That'll happen in February.
That sounds great. Thanks for the update!
Kaldari
Kevin:
The logs in question have always required sudo. I just changed permits before by hand so they were visible to all users.
Permits are set up by the puppet setup of EL which is identical in beta labs and production.
Thanks,
Nuria
On Jan 13, 2015, at 2:58 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
On Tue, Jan 13, 2015 at 2:50 PM, Kevin Leduc kevin@wikimedia.org wrote: Access permisions What's the machine and the log you're trying to access (please be specific, I am not a developer so assume I know very little). I'll pass this on to Ops so they can have a look at why permissions changed. This shouldn't happen.
The logs are /var/log/upstart/eventlogging_processor-client-side-events.log and /var/log/upstart/eventlogging_processor-server-side-events.log on deployment-eventlogging02. I have now been given sudo on this machine, so it isn't an issue anymore for me personally, but it would be great if other developers could access these logs for debugging purposes.
Piping Events that fail validation. It's in our backlog, it's relatively high priority, but not high enough for us to have tasked it out yet or committed it to a sprint. That'll happen in February.
That sounds great. Thanks for the update!
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
+1 on the piping logs into some nice interface. Would solve a lot of the problems we have been looped in on.
On Tuesday, January 13, 2015, Nuria nuria@wikimedia.org wrote:
Kevin:
The logs in question have always required sudo. I just changed permits before by hand so they were visible to all users.
Permits are set up by the puppet setup of EL which is identical in beta labs and production.
Thanks,
Nuria
On Jan 13, 2015, at 2:58 PM, Ryan Kaldari <rkaldari@wikimedia.org javascript:_e(%7B%7D,'cvml','rkaldari@wikimedia.org');> wrote:
On Tue, Jan 13, 2015 at 2:50 PM, Kevin Leduc <kevin@wikimedia.org javascript:_e(%7B%7D,'cvml','kevin@wikimedia.org');> wrote:
Access permisions What's the machine and the log you're trying to access (please be specific, I am not a developer so assume I know very little). I'll pass this on to Ops so they can have a look at why permissions changed. This shouldn't happen.
The logs are /var/log/upstart/eventlogging_processor-client-side-events.log and /var/log/upstart/eventlogging_processor-server-side-events.log on deployment-eventlogging02. I have now been given sudo on this machine, so it isn't an issue anymore for me personally, but it would be great if other developers could access these logs for debugging purposes.
Piping Events that fail validation. It's in our backlog, it's relatively high priority, but not high enough for us to have tasked it out yet or committed it to a sprint. That'll happen in February.
That sounds great. Thanks for the update!
Kaldari
Analytics mailing list Analytics@lists.wikimedia.org javascript:_e(%7B%7D,'cvml','Analytics@lists.wikimedia.org'); https://lists.wikimedia.org/mailman/listinfo/analytics