Hi,
SQID uses a somewhat challenging SPARQL query to refresh its statistical data for the current usage of classes [1]. This is done once per hour, with one retry after 60sec if the first attempt times out. In the past, timeouts have been common, but it usually worked after a while.
Since a few days, however, the query always times out. In spite of the 48 attempts throughout each day, the query did not succeed once since 8/30/2016, 8:12:28 PM [2].
Possible explanations: * WDQS experiences more load now (every day, every hour). * The query got slower since for some reason the overall number of P31 statements increased in a sudden way (or for some reason crossed some threshold). * There have been technical changes to WDQS that reduce performance.
I don't have statistics on the success rate of the problematic query in past weeks, so I cannot say if the timeout rate had increased before the current week.
Does anybody have further information or obsevations that could help to clarify what is going on? We can rewrite our software to use simpler queries if this one fails now, but it seems like a step backwards.
Best regards,
Markus
[1] Here is the query:
SELECT ?cl ?clLabel ?c WHERE { { SELECT ?cl (count(*) as ?c) WHERE { ?i wdt:P31 ?cl } GROUP BY ?cl } SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . } }
[2] https://tools.wmflabs.org/sqid/#/status
Hello!
I just had a quick look at some of our graphs [1][2]. It does look like we've had a slight increase in number of queries per second and correlated higher IO. I'm not sure at this point if this is related to the timeout you see. I'll dig a bit deeper...
Thanks for the patience!
[1] https://grafana.wikimedia.org/dashboard/db/wikidata-query-service [2] https://ganglia.wikimedia.org/latest/?r=month&c=Wikidata+Query+Service+e...
On Sun, Sep 4, 2016 at 2:26 PM, Markus Kroetzsch markus.kroetzsch@tu-dresden.de wrote:
Hi,
SQID uses a somewhat challenging SPARQL query to refresh its statistical data for the current usage of classes [1]. This is done once per hour, with one retry after 60sec if the first attempt times out. In the past, timeouts have been common, but it usually worked after a while.
Since a few days, however, the query always times out. In spite of the 48 attempts throughout each day, the query did not succeed once since 8/30/2016, 8:12:28 PM [2].
Possible explanations:
- WDQS experiences more load now (every day, every hour).
- The query got slower since for some reason the overall number of P31
statements increased in a sudden way (or for some reason crossed some threshold).
- There have been technical changes to WDQS that reduce performance.
I don't have statistics on the success rate of the problematic query in past weeks, so I cannot say if the timeout rate had increased before the current week.
Does anybody have further information or obsevations that could help to clarify what is going on? We can rewrite our software to use simpler queries if this one fails now, but it seems like a step backwards.
Best regards,
Markus
[1] Here is the query:
SELECT ?cl ?clLabel ?c WHERE { { SELECT ?cl (count(*) as ?c) WHERE { ?i wdt:P31 ?cl } GROUP BY ?cl } SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . } }
[2] https://tools.wmflabs.org/sqid/#/status
-- Prof. Dr. Markus Kroetzsch Knowledge-Based Systems Group Faculty of Computer Science TU Dresden +49 351 463 38486 https://iccl.inf.tu-dresden.de/web/KBS/en
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Thanks. Meanwhile, I have also implemented a fall-back query for SQID that does not use the aggregate but only finds distinct results instead. This enables us to update at least the labels automatically from SPARQL (downside: I no longer have reliable statistics on whether the problem is fixed).
Markus
On 05.09.2016 11:27, Guillaume Lederrey wrote:
Hello!
I just had a quick look at some of our graphs [1][2]. It does look like we've had a slight increase in number of queries per second and correlated higher IO. I'm not sure at this point if this is related to the timeout you see. I'll dig a bit deeper...
Thanks for the patience!
[1] https://grafana.wikimedia.org/dashboard/db/wikidata-query-service [2] https://ganglia.wikimedia.org/latest/?r=month&c=Wikidata+Query+Service+e...
On Sun, Sep 4, 2016 at 2:26 PM, Markus Kroetzsch markus.kroetzsch@tu-dresden.de wrote:
Hi,
SQID uses a somewhat challenging SPARQL query to refresh its statistical data for the current usage of classes [1]. This is done once per hour, with one retry after 60sec if the first attempt times out. In the past, timeouts have been common, but it usually worked after a while.
Since a few days, however, the query always times out. In spite of the 48 attempts throughout each day, the query did not succeed once since 8/30/2016, 8:12:28 PM [2].
Possible explanations:
- WDQS experiences more load now (every day, every hour).
- The query got slower since for some reason the overall number of P31
statements increased in a sudden way (or for some reason crossed some threshold).
- There have been technical changes to WDQS that reduce performance.
I don't have statistics on the success rate of the problematic query in past weeks, so I cannot say if the timeout rate had increased before the current week.
Does anybody have further information or obsevations that could help to clarify what is going on? We can rewrite our software to use simpler queries if this one fails now, but it seems like a step backwards.
Best regards,
Markus
[1] Here is the query:
SELECT ?cl ?clLabel ?c WHERE { { SELECT ?cl (count(*) as ?c) WHERE { ?i wdt:P31 ?cl } GROUP BY ?cl } SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . } }
[2] https://tools.wmflabs.org/sqid/#/status
-- Prof. Dr. Markus Kroetzsch Knowledge-Based Systems Group Faculty of Computer Science TU Dresden +49 351 463 38486 https://iccl.inf.tu-dresden.de/web/KBS/en
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Your query without Label works pretty well
SELECT ?cl (count(*) as ?c) WHERE { ?i wdt:P31 ?cl } GROUP BY ?cl }
I bet wikibase:label has to be reimplemented in some other way to prove efficient...
On Mon, Sep 5, 2016 at 11:42 AM, Markus Kroetzsch < markus.kroetzsch@tu-dresden.de> wrote:
Thanks. Meanwhile, I have also implemented a fall-back query for SQID that does not use the aggregate but only finds distinct results instead. This enables us to update at least the labels automatically from SPARQL (downside: I no longer have reliable statistics on whether the problem is fixed).
Markus
On 05.09.2016 11:27, Guillaume Lederrey wrote:
Hello!
I just had a quick look at some of our graphs [1][2]. It does look like we've had a slight increase in number of queries per second and correlated higher IO. I'm not sure at this point if this is related to the timeout you see. I'll dig a bit deeper...
Thanks for the patience!
[1] https://grafana.wikimedia.org/dashboard/db/wikidata-query-service [2] https://ganglia.wikimedia.org/latest/?r=month&c=Wikidata+Que ry+Service+eqiad
On Sun, Sep 4, 2016 at 2:26 PM, Markus Kroetzsch markus.kroetzsch@tu-dresden.de wrote:
Hi,
SQID uses a somewhat challenging SPARQL query to refresh its statistical data for the current usage of classes [1]. This is done once per hour, with one retry after 60sec if the first attempt times out. In the past, timeouts have been common, but it usually worked after a while.
Since a few days, however, the query always times out. In spite of the 48 attempts throughout each day, the query did not succeed once since 8/30/2016, 8:12:28 PM [2].
Possible explanations:
- WDQS experiences more load now (every day, every hour).
- The query got slower since for some reason the overall number of P31
statements increased in a sudden way (or for some reason crossed some threshold).
- There have been technical changes to WDQS that reduce performance.
I don't have statistics on the success rate of the problematic query in past weeks, so I cannot say if the timeout rate had increased before the current week.
Does anybody have further information or obsevations that could help to clarify what is going on? We can rewrite our software to use simpler queries if this one fails now, but it seems like a step backwards.
Best regards,
Markus
[1] Here is the query:
SELECT ?cl ?clLabel ?c WHERE { { SELECT ?cl (count(*) as ?c) WHERE { ?i wdt:P31 ?cl } GROUP BY ?cl } SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . } }
[2] https://tools.wmflabs.org/sqid/#/status
-- Prof. Dr. Markus Kroetzsch Knowledge-Based Systems Group Faculty of Computer Science TU Dresden +49 351 463 38486 https://iccl.inf.tu-dresden.de/web/KBS/en
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
On 07.09.2016 03:05, Stas Malyshev wrote:
Hi!
I bet wikibase:label has to be reimplemented in some other way to prove efficient...
Yes, label service may be inefficient sometimes. I'll look into how it can be improved.
However, the query without the counting but with the labels included also works. Probably we need to do two queries instead of one ...
Markus
FWIW, I also noticed the SERVICE label being much slower than using rdfs:label. Not sure if that's a recent development, but I switched to avoid timeouts.
On Wed, Sep 7, 2016 at 7:36 AM Markus Kroetzsch < markus.kroetzsch@tu-dresden.de> wrote:
On 07.09.2016 03:05, Stas Malyshev wrote:
Hi!
I bet wikibase:label has to be reimplemented in some other way to prove efficient...
Yes, label service may be inefficient sometimes. I'll look into how it can be improved.
However, the query without the counting but with the labels included also works. Probably we need to do two queries instead of one ...
Markus
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata