Hi,
because of this bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=55690
wm-bot is unable to write logs to file storage. Unfortunatelly since 10 minutes ago, the mysql storage broke as well, I have no idea why but I am afraid that only fix that comes to my mind now involves bot restart which isn't possible because all logs that couldn't be written are cached in operating memory (so if I restarted wm-bot we would loose 2 - 3 days of logs of all channels that are in RAM now).
So, if someone wondered what is up with public channel logging, this is a reason why there are logs missing, not only on file storage at bots.wmflabs.org/~wm-bot/logs but also sql at http://tools.wmflabs.org/wm-bot/logs/
I hope wmf will soon start selling "I love gluster" t-shirts :-) I would get one
On Tue, Oct 15, 2013 at 4:15 PM, Petr Bena benapetr@gmail.com wrote:
I hope wmf will soon start selling "I love gluster" t-shirts :-) I would get one
I don't know much about gluster or its terminology but maybe we can make something as good as http://www.zazzle.com/2_lgtm_tee_shirt-235862924096653439 ?
On Tue, Oct 15, 2013 at 9:15 AM, Petr Bena benapetr@gmail.com wrote:
Hi,
because of this bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=55690
wm-bot is unable to write logs to file storage. Unfortunatelly since 10 minutes ago, the mysql storage broke as well, I have no idea why but I am afraid that only fix that comes to my mind now involves bot restart which isn't possible because all logs that couldn't be written are cached in operating memory (so if I restarted wm-bot we would loose 2 - 3 days of logs of all channels that are in RAM now).
So, if someone wondered what is up with public channel logging, this is a reason why there are logs missing, not only on file storage at bots.wmflabs.org/~wm-bot/logs but also sql at http://tools.wmflabs.org/wm-bot/logs/
I hope wmf will soon start selling "I love gluster" t-shirts :-) I would get one
It's possible to switch to NFS, you know ;).
- Ryan
not that it would actually worked better :P I would be most happy to switch to local /dev/vdb storage which never had any problems, but for that wm-bot would need to be on own project
On Tue, Oct 15, 2013 at 6:46 PM, Ryan Lane rlane32@gmail.com wrote:
On Tue, Oct 15, 2013 at 9:15 AM, Petr Bena benapetr@gmail.com wrote:
Hi,
because of this bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=55690
wm-bot is unable to write logs to file storage. Unfortunatelly since 10 minutes ago, the mysql storage broke as well, I have no idea why but I am afraid that only fix that comes to my mind now involves bot restart which isn't possible because all logs that couldn't be written are cached in operating memory (so if I restarted wm-bot we would loose 2 - 3 days of logs of all channels that are in RAM now).
So, if someone wondered what is up with public channel logging, this is a reason why there are logs missing, not only on file storage at bots.wmflabs.org/~wm-bot/logs but also sql at http://tools.wmflabs.org/wm-bot/logs/
I hope wmf will soon start selling "I love gluster" t-shirts :-) I would get one
It's possible to switch to NFS, you know ;).
- Ryan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Oct 16, 2013 at 1:45 AM, Petr Bena benapetr@gmail.com wrote:
not that it would actually worked better :P I would be most happy to switch to local /dev/vdb storage which never had any problems, but for that wm-bot would need to be on own project
That's not a good idea. Performance would be better (which you probably don't need), but at the cost of decreased ability to recover. /dev/vdb goes away with an instance. If a compute node fails and your instance is on it, your data is gone.
- Ryan
right now it is configured like that and I love it. Thanks to that wm-bot never crashes because of nfs / gluster outages, its system files are on local fs which is backed up to gluster every day, in fact it contains just binaries and config files, nothing what can't be recovered easily. Local fs doesn't just bring better performance, but especially perfect availability - it's never offline, while nfs and gluster are having outages all the time...
On Wed, Oct 16, 2013 at 6:00 PM, Ryan Lane rlane32@gmail.com wrote:
On Wed, Oct 16, 2013 at 1:45 AM, Petr Bena benapetr@gmail.com wrote:
not that it would actually worked better :P I would be most happy to switch to local /dev/vdb storage which never had any problems, but for that wm-bot would need to be on own project
That's not a good idea. Performance would be better (which you probably don't need), but at the cost of decreased ability to recover. /dev/vdb goes away with an instance. If a compute node fails and your instance is on it, your data is gone.
- Ryan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org