WDQS updates have stopped - Wikidata - lists.wikimedia.org

List overview All Threads
Download

WDQS updates have stopped

Re: [Wikidata] [Wiki-research-l]...

whitepaper on the Belgian museums'...

James Heald

17 Nov 2015 17 Nov '15

6:39 p.m.

Any idea what's going on with the SPARQL service ? Usually the data gets updated every minute or two, but it's over 11 hours now. -- James.

Reply

Show replies by date

Lydia Pintscher

18 Nov 18 Nov

3:13 p.m.

On Tue, Nov 17, 2015 at 7:39 PM, James Heald <j.heald(a)ucl.ac.uk> wrote:

Any idea what's going on with the SPARQL service ? Usually the data gets updated every minute or two, but it's over 11 hours now.

My best guess looking at things right now is that SuccuBot is making a huge number of edits and the updater for the query service might not be able to handle that yet. Stas: Could you have a look? Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Reply

Lydia Pintscher

3:24 p.m.

On Wed, Nov 18, 2015 at 4:13 PM, Lydia Pintscher <lydia.pintscher(a)wikimedia.de> wrote:

On Tue, Nov 17, 2015 at 7:39 PM, James Heald <j.heald(a)ucl.ac.uk> wrote:

Any idea what's going on with the SPARQL service ? Usually the data gets updated every minute or two, but it's over 11 hours now.

My best guess looking at things right now is that SuccuBot is making a huge number of edits and the updater for the query service might not be able to handle that yet. Stas: Could you have a look?

Update: Succu stopped the bot so we can see if that helps. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Reply

Stas Malyshev

3:44 p.m.

Hi!

Usually the data gets updated every minute or two, but it's over 11 hours now.

My best guess looking at things right now is that SuccuBot is making a huge number of edits and the updater for the query service might not be able to handle that yet. Stas: Could you have a look?

Yes, looks like there's a large volume of updates, so the service is several hours behind, but it seems to be catching up now. What's the bot is doing? -- Stas Malyshev smalyshev(a)wikimedia.org

Reply

Lydia Pintscher

3:47 p.m.

On Wed, Nov 18, 2015 at 4:44 PM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Yes, looks like there's a large volume of updates, so the service is several hours behind, but it seems to be catching up now. What's the bot is doing?

https://www.wikidata.org/wiki/Special:Contributions/SuccuBot Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Reply

Stas Malyshev

3:58 p.m.

Hi!

Yes, looks like there's a large volume of updates, so the service is several hours behind, but it seems to be catching up now. What's the bot is doing?

https://www.wikidata.org/wiki/Special:Contributions/SuccuBot

Last set of edits seems suspect to me - e.g. adding copies of en label to a bunch of species as ru label without it even having ruwiki entry. I'm not sure it's a good thing, but yes, that would generate quite a big load on updating especially that it seems to be adding hundreds (if not thousands) labels per minute. I've also added a note on Succu's talk page to discuss it. -- Stas Malyshev smalyshev(a)wikimedia.org

Reply

Andra Waagmeester

6:03 p.m.

How do you do add "hunderds (if not thousands)" items per minute? We typically see speeds of 20-30 items per minute with our bot account. For our purposes it would be convenient if that number can be increased. On Wed, Nov 18, 2015 at 4:58 PM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hi!

Yes, looks like there's a large volume of updates, so the service is several hours behind, but it seems to be catching up now. What's the bot is doing?

https://www.wikidata.org/wiki/Special:Contributions/SuccuBot

Last set of edits seems suspect to me - e.g. adding copies of en label to a bunch of species as ru label without it even having ruwiki entry. I'm not sure it's a good thing, but yes, that would generate quite a big load on updating especially that it seems to be adding hundreds (if not thousands) labels per minute. I've also added a note on Succu's talk page to discuss it. -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Lydia Pintscher

6:31 p.m.

On Wed, Nov 18, 2015 at 7:03 PM, Andra Waagmeester <andra(a)micelio.be> wrote:

How do you do add "hunderds (if not thousands)" items per minute? We typically see speeds of 20-30 items per minute with our bot account. For our purposes it would be convenient if that number can be increased.

That rate is preferred for technical (dispatching to Wikipedia and co and updating of the query service) and social (ability to review changes) reason. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Reply

Federico Leva (Nemo)

6:40 p.m.

Andra Waagmeester, 18/11/2015 19:03:

How do you do add "hunderds (if not thousands)" items per minute?

Usually 1) concurrency, 2) low latency. Nemo

Reply

Markus Krötzsch

8:30 p.m.

On 18.11.2015 19:40, Federico Leva (Nemo) wrote:

Andra Waagmeester, 18/11/2015 19:03:

How do you do add "hunderds (if not thousands)" items per minute?

Usually 1) concurrency, 2) low latency.

In fact, it is not hard to get this. I guess Andra is getting speeds of 20-30 items because their bot framework is throttling the speed on purpose. If I don't throttle WDTK, I can easily do well over 100 edits per minute in a single thread (I did not try the maximum ;-). Already a few minutes of fast editing might push up the median dispatch lag sufficiently for a bot to stop/wait. While the slow edit rate is a rough guess (not a strict rule), respecting the dispatch stats is mandatory for Wikidata bots, so things will eventually slow down (or your bot be blocked ;-). See [1]. Markus [1] https://www.wikidata.org/wiki/Wikidata:Bots

Reply

Stas Malyshev

19 Nov 19 Nov

12:48 a.m.

Hi! The service have caught up now, but in the near future I would like to ask to make the bots throttle the edits a bit, for now. In the meantime, I'll look into speeding up the update process further, but given the nature of the database it may still be possible to temporarily overload it with large enough update stream. So keeping bot updates under 10 per second would be nice (this is somewhat arbitrary from back-of-the envelope calculation, so don't take the exact figure *too* seriously). Note that this should not be too hard a limit - that allows to update every single record now in Wikidata in about 2 weeks, which seems to be OK for most tasks. But it is a limitation and as I said, I'll work to eventually get rid of it. Thanks, -- Stas Malyshev smalyshev(a)wikimedia.org

Reply

Gerard Meijssen

6:57 a.m.

Hoi, So in essence, WDQ and WDQS have the same problem of keeping up with the database. How do the two compare? Thanks, GerardM On 19 November 2015 at 01:48, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hi! The service have caught up now, but in the near future I would like to ask to make the bots throttle the edits a bit, for now. In the meantime, I'll look into speeding up the update process further, but given the nature of the database it may still be possible to temporarily overload it with large enough update stream. So keeping bot updates under 10 per second would be nice (this is somewhat arbitrary from back-of-the envelope calculation, so don't take the exact figure *too* seriously). Note that this should not be too hard a limit - that allows to update every single record now in Wikidata in about 2 weeks, which seems to be OK for most tasks. But it is a limitation and as I said, I'll work to eventually get rid of it. Thanks, -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Tom Morris

7:55 a.m.

So, the page that Markus points to describes heeding the replication lag limit as a recommendation. Since running a bot is a privilege, not a right, why isn't the "recommendation" a requirement instead of a recommendation? Tom On Wed, Nov 18, 2015 at 3:30 PM, Markus Krötzsch < markus(a)semantic-mediawiki.org> wrote:

On 18.11.2015 19:40, Federico Leva (Nemo) wrote:

Andra Waagmeester, 18/11/2015 19:03:

How do you do add "hunderds (if not thousands)" items per minute?

Usually 1) concurrency, 2) low latency.

In fact, it is not hard to get this. I guess Andra is getting speeds of 20-30 items because their bot framework is throttling the speed on purpose. If I don't throttle WDTK, I can easily do well over 100 edits per minute in a single thread (I did not try the maximum ;-). Already a few minutes of fast editing might push up the median dispatch lag sufficiently for a bot to stop/wait. While the slow edit rate is a rough guess (not a strict rule), respecting the dispatch stats is mandatory for Wikidata bots, so things will eventually slow down (or your bot be blocked ;-). See [1]. Markus [1] https://www.wikidata.org/wiki/Wikidata:Bots _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Gerard Meijssen

9:40 a.m.

Hoi, Because once it is a requirement and not a recommendation, it will be impossible to reverse this. The insidious creep of more rules and requirements will make Wikidata increasingly less of a wiki. Arguably most of the edits done by bot are of a higher quality than those done by hand. It is for the people maintaining the SPARQL environment to ensure that it is up to the job as it does not affect Wikidata itself. I think each of these argument holds its own. Together they are hopefully potent enough to prevent such silliness. Thanks, GerardM On 19 November 2015 at 08:55, Tom Morris <tfmorris(a)gmail.com> wrote:

So, the page that Markus points to describes heeding the replication lag limit as a recommendation. Since running a bot is a privilege, not a right, why isn't the "recommendation" a requirement instead of a recommendation? Tom On Wed, Nov 18, 2015 at 3:30 PM, Markus Krötzsch < markus(a)semantic-mediawiki.org> wrote:

On 18.11.2015 19:40, Federico Leva (Nemo) wrote:

Andra Waagmeester, 18/11/2015 19:03:

How do you do add "hunderds (if not thousands)" items per minute?

Usually 1) concurrency, 2) low latency.

In fact, it is not hard to get this. I guess Andra is getting speeds of 20-30 items because their bot framework is throttling the speed on purpose. If I don't throttle WDTK, I can easily do well over 100 edits per minute in a single thread (I did not try the maximum ;-). Already a few minutes of fast editing might push up the median dispatch lag sufficiently for a bot to stop/wait. While the slow edit rate is a rough guess (not a strict rule), respecting the dispatch stats is mandatory for Wikidata bots, so things will eventually slow down (or your bot be blocked ;-). See [1]. Markus [1] https://www.wikidata.org/wiki/Wikidata:Bots _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Markus Krötzsch

10:06 a.m.

On 19.11.2015 10:40, Gerard Meijssen wrote:

Hoi, Because once it is a requirement and not a recommendation, it will be impossible to reverse this. The insidious creep of more rules and requirements will make Wikidata increasingly less of a wiki. Arguably most of the edits done by bot are of a higher quality than those done by hand. It is for the people maintaining the SPARQL environment to ensure that it is up to the job as it does not affect Wikidata itself. I think each of these argument holds its own. Together they are hopefully potent enough to prevent such silliness.

Maybe it would not be that bad. I actually think that many bots right now are slower than they could be because they are afraid to overload the site. If bots would check the lag, they could operate close to the maximum load that the site can currently handle, which is probably more than most bots are doing now. The "requirement" vs. "recommendation" thing is maybe not so relevant, since bot rules (mandatory or not) are currently not enforced in any strong way. Basically, the whole system is based on mutual trust and this is how it should stay. Markus

Reply

Gerard Meijssen

10:40 a.m.

Hoi, <grin> Markus we agree. </grin> Given that the lag of updates is measurable, it would be good to have an algorithm that allows bots to negotiate their speed and thereby maximise throughput. When such an algorithm is in bot environments as pywiki, any and all pywiki bots can safely go wild and do the good they are known for. <grin> Thanks, GerardM On 19 November 2015 at 11:06, Markus Krötzsch <markus(a)semantic-mediawiki.org

wrote:

On 19.11.2015 10:40, Gerard Meijssen wrote:

Hoi, Because once it is a requirement and not a recommendation, it will be impossible to reverse this. The insidious creep of more rules and requirements will make Wikidata increasingly less of a wiki. Arguably most of the edits done by bot are of a higher quality than those done by hand. It is for the people maintaining the SPARQL environment to ensure that it is up to the job as it does not affect Wikidata itself. I think each of these argument holds its own. Together they are hopefully potent enough to prevent such silliness.

Maybe it would not be that bad. I actually think that many bots right now are slower than they could be because they are afraid to overload the site. If bots would check the lag, they could operate close to the maximum load that the site can currently handle, which is probably more than most bots are doing now. The "requirement" vs. "recommendation" thing is maybe not so relevant, since bot rules (mandatory or not) are currently not enforced in any strong way. Basically, the whole system is based on mutual trust and this is how it should stay. Markus _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Lukas Benedix

12:10 p.m.

Is there any evidence, that the quality of bot edits is higher than edits by humans? LB

Hoi, Because once it is a requirement and not a recommendation, it will be impossible to reverse this. The insidious creep of more rules and requirements will make Wikidata increasingly less of a wiki. Arguably most of the edits done by bot are of a higher quality than those done by hand. It is for the people maintaining the SPARQL environment to ensure that it is up to the job as it does not affect Wikidata itself. I think each of these argument holds its own. Together they are hopefully potent enough to prevent such silliness. Thanks, GerardM On 19 November 2015 at 08:55, Tom Morris <tfmorris(a)gmail.com> wrote:

So, the page that Markus points to describes heeding the replication lag limit as a recommendation. Since running a bot is a privilege, not a right, why isn't the "recommendation" a requirement instead of a recommendation? Tom On Wed, Nov 18, 2015 at 3:30 PM, Markus Krötzsch < markus(a)semantic-mediawiki.org> wrote:

On 18.11.2015 19:40, Federico Leva (Nemo) wrote:

Andra Waagmeester, 18/11/2015 19:03: > How do you do add "hunderds (if not thousands)" items per minute? > Usually 1) concurrency, 2) low latency.

In fact, it is not hard to get this. I guess Andra is getting speeds of 20-30 items because their bot framework is throttling the speed on purpose. If I don't throttle WDTK, I can easily do well over 100 edits per minute in a single thread (I did not try the maximum ;-). Already a few minutes of fast editing might push up the median dispatch lag sufficiently for a bot to stop/wait. While the slow edit rate is a rough guess (not a strict rule), respecting the dispatch stats is mandatory for Wikidata bots, so things will eventually slow down (or your bot be blocked ;-). See [1]. Markus [1] https://www.wikidata.org/wiki/Wikidata:Bots _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Lydia Pintscher

1:53 p.m.

On Thu, Nov 19, 2015 at 1:10 PM, Lukas Benedix <benedix(a)zedat.fu-berlin.de> wrote:

Is there any evidence, that the quality of bot edits is higher than edits by humans?

Let's please not treat bot edits as something magical that comes down from the sky. Bots have a human operating them. If that person knows what they are doing it is good - if not not. The difference is just the scale and impact a person can have. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Reply

3104

days inactive

3106

days old

wikidata@lists.wikimedia.org

Manage subscription

17 comments

9 participants

tags (0)

participants (9)

Andra Waagmeester
Federico Leva (Nemo)
Gerard Meijssen
James Heald
Lukas Benedix
Lydia Pintscher
Markus Krötzsch
Stas Malyshev
Tom Morris