Hey, It seems there is some sort of back pressure on the ORES service right now causing to send out timeout and overload errors which made icinga scream at #wikimedia-ai several times today. If you're running the requests, please slow down a little.
We Increased the capacity for now [1] (Thanks to Andrew Bogott). That brought back everything to the normal state. Sorry for any inconvenience.
I forgot some links to show: 1- Scoring requests: https://grafana.wikimedia.org/dashboard/db/ores?panelId=1&fullscreen&... 2- Response time: https://grafana.wikimedia.org/dashboard/db/ores?panelId=15&fullscreen&am... 3- uwsgi busy workers: https://grafana.wikimedia.org/dashboard/db/ores?panelId=13&fullscreen&am... 4- icinga screams: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-ai/20161016.txt
Best
On Mon, Oct 17, 2016 at 12:39 AM Amir Ladsgroup ladsgroup@gmail.com wrote:
Hey, It seems there is some sort of back pressure on the ORES service right now causing to send out timeout and overload errors which made icinga scream at #wikimedia-ai several times today. If you're running the requests, please slow down a little.
We Increased the capacity for now [1] (Thanks to Andrew Bogott). That brought back everything to the normal state. Sorry for any inconvenience.