Hello Wikidata,

We have had some ups and downs with the WDQS service in our CODFW datacenter today, but we believe the service is stabilized.

We have opened an incident report for the issue[0] . I'm still filling it out, but I should have more information within the next 24 hours.

Please accept our apologies for any inconvenience this may have caused.

Best,

Brian King SRE, Search Platform Team Wikimedia Foundation IRC: inflatador

[0] https://wikitech.wikimedia.org/wiki/Incidents/2023-05-23_wdqs_CODFW_5xx_erro...

On May 23, 2023, at 9:17 AM, Guillaume Lederrey glederrey@wikimedia.org wrote:

Hello all!

We are currently experiencing issues with the codfw WDQS cluster, with a sharp drop in successful queries [1]. We suspect that the cluster is overloaded by some expensive queries, but we are still tracking them down. This should affect only traffic routed to the codfw cluster [2].

We will post an update as soon as we know more.

Good luck!

Guillaume

[1] https://grafana.wikimedia.org/d/l-3CMlN4z/wdqs-uptime-slo?orgId=1&var-Da... https://grafana.wikimedia.org/d/l-3CMlN4z/wdqs-uptime-slo?orgId=1&var-Datacenter=codfw&var-SLO_Threshold=.95&var-Period_Days=90d&from=now-6h&to=now [2] https://wikitech.wikimedia.org/wiki/Global_traffic_routing https://wikitech.wikimedia.org/wiki/Global_traffic_routing

-- Guillaume Lederrey (he/him) Engineering Manager Wikimedia Foundation https://wikimediafoundation.org/_______________________________________________ Wikidata mailing list -- wikidata@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/mes... To unsubscribe send an email to wikidata-leave@lists.wikimedia.org