[Labs-l] IRC logs are gone

[Petr Bena]

I found out exactly what happened, though I don't know who did it. The
whole logs folder was recursively deleted 4 days ago, interesting on this
is that the bot is only creating the log folders when the channel config is
loaded and log folder doesn't exist. Because the bot was running and all
channels were loaded, it was attempting to flush its own cache to disk but
it wasn't able to do so because the path of target file wasn't valid (when
you write to a file, it doesn't automatically create the folders). That's
why we are missing logs for so many days - the bot simply wasn't producing
any, backup is running every day but there was nothing to backup, and
yesterday when I inserted new module for labs tools, the bot needed to
reboot, and that was the time when folders were created and logging started
to continue.


On Sat, Dec 15, 2012 at 1:45 AM, Petr Bena <benapetr at gmail.com> wrote:

> of course that's not the way I prefer we need to find out what is
> happening, it may be even some my mistake or some weird bug, there is a
> script in cron which generates the .gz files, that one is running as root
> as far as I know, because it creates the backups too and sets some
> permissions on them, it's cronned on bots-1 but I didn't change it for
> weeks and it never happened, so i don't think it has something to do with
> that...
>
>
>
> On Sat, Dec 15, 2012 at 1:38 AM, Jeremy Baron <jeremy at tuxmachine.com>wrote:
>
>> On Fri, Dec 14, 2012 at 7:12 PM, Jeremy Baron <jeremy at tuxmachine.com>
>> wrote:
>> > On Fri, Dec 14, 2012 at 6:55 PM, Petr Bena <benapetr at gmail.com> wrote:
>> >> Hi, I don't know what happened, but someone removed recursively all
>> data in
>> >> ~wm-bot/logs
>> >
>> > Wasn't that accident already announced?
>>
>> erm, whooops. seems I misinterpreted
>> http://lists.wikimedia.org/pipermail/labs-l/2012-December/000553.html
>> and that prior deletion was narrower than you've just noticed?
>>
>> >> I may have a backup but because this removal was done yesterday or
>> earlier
>> >> and apparently happens every day, it's a problem, because even the
>> complete
>> >> logs archive is almost empty now.
>> >
>> > Every day?? elaborate?
>>
>> Well, I'm still not too clear on this point but if this really is
>> happening repeatedly (cron'd??) then we need to track down what's
>> doing it. We can't be needing to manually restore backups multiple
>> times per week. (and we'd probably have incremental data loss between
>> last backup and deletion too)
>>
>> -Jeremy
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20121215/06edb93b/attachment.html>