Hi, For one of my projects, I need to be able to keep the most up to date version of wikipedia html pages for a few languages like en, zh, de, es, fr etc. So this is done currently in two steps, 1. Listen to changes on stream API documented here <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams> and then extract the page titles. 2. For each of the titles, get the latest HTML using the Wikipedia REST api <https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_title__title_> and persist the HTML. I understand that in order to avoid the 429 (Too many requests error), we need to make sure we limit the api request to 1 per second. Just wanted to check if we can make requests to different languages like en.wikipedia.org, fr.wikipedia.org etc in parallel or do those requests also need to be done in serial manner (1 per second), in order to not hit HTTP 429 error. Please let me know if you need more information. -- Regards, Aadithya -- Sent from my iPad3 _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api

Thank you for the quick response, Michael. I was making close to 10 requests per second previously. But would hit the HTTP 429 errors frequently. In the etiquette document here <https://www.mediawiki.org/wiki/API:Etiquette>, it suggested we make requests in serial manner rather than parallel. Hence started making requests in serial manner and one request per second, as I did not want to abuse the API. But as you can imagine it takes up a lot of time, especially when trying to expand to multiple languages. Also, I send a valid User-Agent header as described here <https://meta.wikimedia.org/wiki/User-Agent_policy>. What do you think could be other reasons why I hit the HTTP 429 error? Is there a cap on total number of requests per day/week etc.? On Wed, May 8, 2019 at 10:43 AM Michael Holloway < mholloway(a)wikimedia.org&gt; wrote: > Hi Aadithya, > > According to the information on the top of the REST API docs page > <https://wikimedia.org/api/rest_v1/>, you should in general be able > to make up to 200 read requests per second to the REST API without any > trouble. As far as I know, that information is accurate. Are you hitting > 429s at a lower request rate than that? > > To answer your question, sending requests in parallel to multiple > language subdomains should not be a problem so long as your overall request > rate remains lower than ~200/s. > > On Tue, May 7, 2019 at 8:27 PM Aadithya C Udupa < > udupa.adithya(a)gmail.com&gt; wrote: > >> Hi, >> For one of my projects, I need to be able to keep the most up to date >> version of wikipedia html pages for a few languages like en, zh, de, es, fr >> etc. So this is done currently in two steps, >> 1. Listen to changes on stream API documented here >> <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams> and >> then extract the page titles. >> 2. For each of the titles, get the latest HTML using the Wikipedia >> REST api >> <https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_title__title_> and >> persist the HTML. >> >> I understand that in order to avoid the 429 (Too many requests >> error), we need to make sure we limit the api request to 1 per second. Just >> wanted to check if we can make requests to different languages like >> en.wikipedia.org, fr.wikipedia.org etc in parallel or do those >> requests also need to be done in serial manner (1 per second), in order to >> not hit HTTP 429 error. >> >> Please let me know if you need more information. >> >> >> -- >> Regards, >> Aadithya >> -- >> Sent from my iPad3 >> _______________________________________________ >> Mediawiki-api mailing list >> Mediawiki-api(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-api >> > > > -- > Michael Holloway > Software Engineer, Reading Infrastructure > _______________________________________________ > Mediawiki-api mailing list > Mediawiki-api(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mediawiki-api > -- Regards, Aadithya _______________________________________________ Mediawiki-api mailing list Mediawiki-api(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api