Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I asked Andrew to generate a list of potential files.
-Toby
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
/a/eventlogging/archive/*
/a/squid/archive api arabic-banner bannerImpressions blog edits edits-geocoded glam_nara mobile mobile-geocoded sampled sampled-geocoded sopa teahouse zero
Tell me what not to prune, and a reason why it shouldn’t be pruned.
:)
On Oct 30, 2014, at 3:32 PM, Toby Negrin tnegrin@wikimedia.org wrote:
I asked Andrew to generate a list of potential files.
-Toby
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote: Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Also, I'm not clear on the significance of the EL archive directory. Can
you remind me/direct me to documentation? Well, the logs just record the incoming pipeline of events, we have used them to troubleshoot operational issues in the past but the bulk of data analysis in EL happens from data stored on database.
Some info here: https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hey guys,
Sorry for the late response, but I'm still not sure what lives in */a/eventlogging/archive/**
Will deleting from there affect what logs we have stored in the DB? Is this an intermediate log storage place, a canonical one, etc.?
What will we no longer be able to do after it is pruned?
-Aaron
On Thu, Oct 30, 2014 at 2:35 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Also, I'm not clear on the significance of the EL archive directory.
Can you remind me/direct me to documentation? Well, the logs just record the incoming pipeline of events, we have used them to troubleshoot operational issues in the past but the bulk of data analysis in EL happens from data stored on database.
Some info here: https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
No, database records are not affected. It should not impact your work with EL in any way as your findings come from the records in the database.
The logs are used mainly for operational purposes by the dev team as maintainers of the system.
On Nov 4, 2014, at 11:55 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Hey guys,
Sorry for the late response, but I'm still not sure what lives in /a/eventlogging/archive/*
Will deleting from there affect what logs we have stored in the DB? Is this an intermediate log storage place, a canonical one, etc.?
What will we no longer be able to do after it is pruned?
-Aaron
On Thu, Oct 30, 2014 at 2:35 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
Well, the logs just record the incoming pipeline of events, we have used them to troubleshoot operational issues in the past but the bulk of data analysis in EL happens from data stored on database.
Some info here: https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Nov 4, 2014, at 2:08 PM, Nuria nuria@wikimedia.org wrote:
No, database records are not affected. It should not impact your work with EL in any way as your findings come from the records in the database. The logs are used mainly for operational purposes by the dev team as maintainers of the system.
this obviously means that any missing, corrupted or invalid data from the log DB can only be recovered within the data retention window, which seems reasonable to me.
On Nov 4, 2014, at 11:55 AM, Aaron Halfaker <ahalfaker@wikimedia.org mailto:ahalfaker@wikimedia.org> wrote:
Hey guys,
Sorry for the late response, but I'm still not sure what lives in /a/eventlogging/archive/*
Will deleting from there affect what logs we have stored in the DB? Is this an intermediate log storage place, a canonical one, etc.?
What will we no longer be able to do after it is pruned?
-Aaron
On Thu, Oct 30, 2014 at 2:35 PM, Nuria Ruiz <nuria@wikimedia.org mailto:nuria@wikimedia.org> wrote:
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
Well, the logs just record the incoming pipeline of events, we have used them to troubleshoot operational issues in the past but the bulk of data analysis in EL happens from data stored on database.
Some info here: https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker <ahalfaker@wikimedia.org mailto:ahalfaker@wikimedia.org> wrote: Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz <nuria@wikimedia.org mailto:nuria@wikimedia.org> wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I didn't find that obvious. Thanks Dario. I agree that a 90 day data retention window would be long enough to deal with any corrupt/invalid data.
On Tue, Nov 4, 2014 at 11:12 PM, Dario Taraborelli < dtaraborelli@wikimedia.org> wrote:
On Nov 4, 2014, at 2:08 PM, Nuria nuria@wikimedia.org wrote:
No, database records are not affected. It should not impact your work with EL in any way as your findings come from the records in the database. The logs are used mainly for operational purposes by the dev team as maintainers of the system.
this obviously means that any missing, corrupted or invalid data from the log DB can only be recovered within the data retention window, which seems reasonable to me.
On Nov 4, 2014, at 11:55 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Hey guys,
Sorry for the late response, but I'm still not sure what lives in */a/eventlogging/archive/**
Will deleting from there affect what logs we have stored in the DB? Is this an intermediate log storage place, a canonical one, etc.?
What will we no longer be able to do after it is pruned?
-Aaron
On Thu, Oct 30, 2014 at 2:35 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Also, I'm not clear on the significance of the EL archive directory.
Can you remind me/direct me to documentation? Well, the logs just record the incoming pipeline of events, we have used them to troubleshoot operational issues in the past but the bulk of data analysis in EL happens from data stored on database.
Some info here: https://wikitech.wikimedia.org/wiki/EventLogging#Data_storage
On Thu, Oct 30, 2014 at 12:22 PM, Aaron Halfaker <ahalfaker@wikimedia.org
wrote:
Nuria, can you specify which logs will be trimmed.
Also, I'm not clear on the significance of the EL archive directory. Can you remind me/direct me to documentation?
On Thu, Oct 30, 2014 at 12:36 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Hello,
To comply with our privacy policy we are going to purge logs in 1002 that are older than 90 days. Please let us know whether this is an issue. We hope to have these changes done by the end of next week.
A concrete example:
Logs in, for example, the eventlogging archiving directory:
@stat1002:/a/eventlogging/archive$
will be restricted to the last 90 days.
Thanks,
Nuria
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics