Today ORES in production was sending out unreasonable amount of timeout
errors. Causing icinga to scream and 14% failure rate on average for ORES
review tool jobs. It turned out that ores workers are logging too much
causing the nodes to run out of disk space. [1] I suspect we had similar
issue in our labs nodes.
I made changes for prod and labs and deployed it today. You can find more
details in the phab card
[1]:
https://phabricator.wikimedia.org/T146581
Cheers
Best