Hello all!
Here is (at last!) an update on what we are doing to protect the stability of Wikidata Query Service.
For 4 years we have been offering to Wikidata users the Query Service, a powerful tool that allows anyone to query the content of Wikidata, without any identification needed. This means that anyone can use the service using a script and make heavy or very frequent requests. However, this freedom has led to the service being overloaded by a too big amount of queries, causing the issues or lag that you may have noticed.
A reminder about the context:
We have had a number of incidents where the public WDQS endpoint was overloaded by bot traffic. We don't think that any of that activity was intentionally malicious, but rather that the bot authors most probably don't understand the cost of their queries and the impact they have on our infrastructure. We've recently seen more distributed bots, coming from multiple IPs from cloud providers. This kind of pattern makes it harder and harder to filter or throttle an individual bot. The impact has ranged from increased update lag to full service interruption.
What we have been doing:
While we would love to allow anyone to run any query they want at any time, we're not able to sustain that load, and we need to be more aggressive in how we throttle clients. We want to be fair to our users and allow everyone to use the service productively. We also want the service to be available to the casual user and provide up-to-date access to the live Wikidata data. And while we would love to throttle only abusive bots, to be able to do that we need to be able to identify them.
We have two main means of identifying bots:
1) their user agent and IP address 2) the pattern of their queries
Identifying patterns in queries is done manually, by a person inspecting the logs. It takes time and can only be done after the fact. We can only start our identification process once the service is already overloaded. This is not going to scale.
IP addresses are starting to be problematic. We see bots running on cloud providers and running their workloads on multiple instances, with multiple IP addresses.
We are left with user agents. But here, we have a problem again. To block only abusive bots, we would need those bots to use a clearly identifiable user agent, so that we can throttle or block them and contact the author to work together on a solution. It is unlikely that an intentionally abusive bot will voluntarily provide a way to be blocked. So we need to be more aggressive about bots which are using a generic user agent. We are not blocking those, but we are limiting the number of requests coming from generic user agents. This is a large bucket, with a lot of bots that are in this same category of "generic user agent". Sadly, this is also the bucket that contains many small bots that generate only a very reasonable load. And so we are also impacting the bots that play fair.
At the moment, if your bot is affected by our restrictions, configure a custom user agent that identifies you; this should be sufficient to give you enough bandwidth. If you are still running into issues, please contact us; we'll find a solution together.
What's coming next:
First, it is unlikely that we will be able to remove the current restrictions in the short term. We're sorry for that, but the alternative - service being unresponsive or severely lagged for everyone - is worse.
We are exploring a number of alternatives. Adding authentication to the service, and allowing higher quotas to bots that authenticate. Creating an asynchronous queue, which could allow running more expensive queries, but with longer deadlines. And we are in the process of hiring another engineer to work on these ideas.
Thanks for your patience!
WDQS Team