Slow response and incomplete result for RecentChange API in wikidata.org

List overview All Threads
Download

newer

older

Identify problems with requesting...

In case of creation of an item...

Wenjie Song

13 Nov 2018 13 Nov '18

10:45 a.m.

Dear WikiMedia, We are using Action API to get recent change updates from www.wikidata.org:

...

https://www.wikidata.org/w/api.php?action=query&format=json&continu… <https://www.wikidata.org/w/api.php?action=query&format=jsonfm&continue=&list=recentchanges&rclimit=500&rcnamespace=0%7C2%7C4%7C6%7C10%7C14%7C120%7C122%7C828&rcprop=comment%7Cflags%7Cids%7Cloginfo%7Coresscores%7Csizes%7Ctags%7Ctimestamp%7Ctitle%7Cuser>

But we've found that this API become very slow recently, the response time of one API request is between 15s ~ 25s. Sometimes the API returns an empty recent change list: [image: SHgNOsVRJPz.png] Other times the length of the response recent change list is always between 35~37, even if we set the rclimit parameter to 500. Is there any changes related to www.wikidata.org recently? Sincerely, Wenjie -- Best Regards, Wenjie

Attachments:

attachment.htm (text/html — 3.0 KB)
SHgNOsVRJPz.png (image/png — 42.5 KB)

Show replies by date

Thomas Arrow

14 Nov 14 Nov

10:24 a.m.

New subject: Slow response and incomplete result for RecentChange API in wikidata.org

Hi, Thanks for the report! We've just opened a ticket about this: https://phabricator.wikimedia.org/T209459#4745495 where you can follow along for updates. You'll see that the problem appears to be related to the `oresscores` parameter. If you omit this then you get a timely response. Best, Tom

Lydia Pintscher

2 Dec 2 Dec

7:56 p.m.

New subject: Slow response and incomplete result for RecentChange API in wikidata.org

On Wed, Nov 14, 2018 at 9:46 AM Wenjie Song <wenjies(a)google.com> wrote:

...

Dear WikiMedia, We are using Action API to get recent change updates from www.wikidata.org :

Hi Wenjie, Can you please check and let us know if you are still experiencing the problem? Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Stas Malyshev

8:46 p.m.

New subject: Slow response and incomplete result for RecentChange API in wikidata.org

Hi!

...

Can you please check and let us know if you are still experiencing the problem?

We have a task https://phabricator.wikimedia.org/T202764 which I suspect describes the same issue. It is still open, and even though WDQS is running on Kafka in production and thus is not affected by it, I see it every time I run it on Labs (where Kafka stream is not available). So I think the issues with RC API on wikidata are still alive. There's also a parallel issue of https://phabricator.wikimedia.org/T207718 with RDF fetching, which also still happens. -- Stas Malyshev smalyshev(a)wikimedia.org

Amir Sarabadani

3 Dec 3 Dec

2:38 p.m.

New subject: Slow response and incomplete result for RecentChange API in wikidata.org

Also given that it uses oresscores, we recently fixed some performance issues caused by it. Do you still have issues with it? Best On Sun, 2 Dec 2018 at 20:47, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

...

Hi!

Can you please check and let us know if you are still experiencing the problem?

-- Amir Sarabadani Software Engineer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Tel. (030) 219 158 26-0 http://wikimedia.de Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen Wissens frei teilhaben kann. Helfen Sie uns dabei! http://spenden.wikimedia.de/ Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Stas Malyshev

7:30 p.m.

New subject: Slow response and incomplete result for RecentChange API in wikidata.org

Hi!

...

Also given that it uses oresscores, we recently fixed some performance issues caused by it. Do you still have issues with it?

Yes, the issues I have listed still happen. My API calls do not use ORES. E.g. see: https://logstash.wikimedia.org/goto/63db4ce68fb5da3cdc7828150de10c59 -- Stas Malyshev smalyshev(a)wikimedia.org

1970

days inactive

1990

days old

wikidata@lists.wikimedia.org

Manage subscription

5 comments

5 participants

tags (0)

participants (5)

Amir Sarabadani
Lydia Pintscher
Stas Malyshev
Thomas Arrow
Wenjie Song