Hello,
I'm currently in the last stages of finishing a bot that will make templates, userpages, talkpages on all Wikimedia wikis, this is done by the Wikimedia API but I have a problem now and kind of need advice.
The bot will do the following:
bs.wiktionary.org User:GlobalEditBot --> check if the page exist creating (+244). --> creating the page Watchlist added --> Adding it to my watchlist Retrieving watchlist for wiktionary:bs --> confirming its on it.
And it trows back this error:
Creating page [[wiktionary:bs:Korisnik:GlobalEditBot]] via API Unknown Error. API Error code:ratelimited Information:You've exceeded your rate limit. Please wait some time and try again
What is the policy by Wikimedia for the API use, how much delay should I use between the wiki's? Currently its 3 sec between every wiki's but that makes me hit the limit. What is the adviced time?
Best,
Dirt Diver
On Fri, Jul 1, 2011 at 2:23 PM, Dirt Diver info@dirtdiver.org wrote:
What is the policy by Wikimedia for the API use, how much delay should I use between the wiki's? Currently its 3 sec between every wiki's but that makes me hit the limit. What is the adviced time?
Rate limits affect edits, rollbacks and moves only. They are applied both in the UI and in the API (although you probably have to try pretty hard to hit them in the UI).
On Wikimedia wikis, the rate limit for editing is 8 edits per minute for anonymous (logged-out) and new users. So waiting 3 seconds between edits will result in almost 20 edits per minute, that's way too much. If you're operating a bot, you should get an account for it and ask for a bot flag. That way you'll be exempt from rate limits and the bot's edits will be hidden from Special:RecentChanges if you pass &bot=1 in the action=edit call.
Roan Kattouw (Catrope)
Roan Kattouw wrote:
On Wikimedia wikis, the rate limit for editing is 8 edits per minute for anonymous (logged-out) and new users. So waiting 3 seconds between edits will result in almost 20 edits per minute, that's way too much. If you're operating a bot, you should get an account for it and ask for a bot flag. That way you'll be exempt from rate limits and the bot's edits will be hidden from Special:RecentChanges if you pass &bot=1 in the action=edit call.
Roan Kattouw (Catrope)
I think his problem is that he wants to do that in several wikis (eg. create user pages on all wikis), so that would require a global bot flag. http://meta.wikimedia.org/wiki/Global_bots#Global_bots
However, that isn't allowed by the policy: "bot must only maintain interlanguage links or fix double-redirects"
That could be changed by community consensus, but i don't think it would be easy.
On Fri, Jul 1, 2011 at 5:59 PM, Platonides Platonides@gmail.com wrote:
Roan Kattouw wrote: I think his problem is that he wants to do that in several wikis (eg. create user pages on all wikis), so that would require a global bot flag. http://meta.wikimedia.org/wiki/Global_bots#Global_bots
However, that isn't allowed by the policy: "bot must only maintain interlanguage links or fix double-redirects"
That could be changed by community consensus, but i don't think it would be easy.
It's a moot point anyway, Abigor's bot was globally blocked: http://meta.wikimedia.org/w/index.php?title=Steward_requests%2FGlobal&action=historysubmit&diff=2693470&oldid=2693415
Baseball Bugs reports that the Make Your Own feature in WikiLove doesn't work for him, and he gets an API error consistently.
I asked him to go to the following URL to make sure that it returns a result for him: http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil...
Instead of getting a result from the API, he gets a 403 error: "The website declined to show this webpage / HTTP 403"
What could possibly be the reason for this? Is there an IP blacklist for the API or something?
Ryan Kaldari
----- Original Message -----
From: "Ryan Kaldari" rkaldari@wikimedia.org
Baseball Bugs reports that the Make Your Own feature in WikiLove doesn't work for him, and he gets an API error consistently.
I asked him to go to the following URL to make sure that it returns a result for him: http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil...
Instead of getting a result from the API, he gets a 403 error: "The website declined to show this webpage / HTTP 403"
FWIW, I get an XML file here, which Firefox is nice enough to show the source of, because there's no XSL associated with it. Road Runner Tampa Bay, IPv4; FF 4.0.1; SuSE 11.3.
Cheers, -- jra
Ryan Kaldari rkaldari@wikimedia.org wrote:
Baseball Bugs reports that the Make Your Own feature in WikiLove doesn't work for him, and he gets an API error consistently.
I asked him to go to the following URL to make sure that it returns a result for him: http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil...
Instead of getting a result from the API, he gets a 403 error: "The website declined to show this webpage / HTTP 403"
What could possibly be the reason for this? Is there an IP blacklist for the API or something?
Missing User-Agent header?
Tim
On Sun, Jul 3, 2011 at 3:21 AM, Ryan Kaldari rkaldari@wikimedia.org wrote:
Instead of getting a result from the API, he gets a 403 error: "The website declined to show this webpage / HTTP 403"
Is he using IE6? I think we still haven't deployed the proper fixes for the IE6 extension issue yet, although they did go into the 1.17 release.
Roan
No, he's using IE8 and it does send his User Agent header: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)
Are there any other reasons why someone would be rejected from the API? Getting imageinfo doesn't require a token, so I'm at a loss as to what is going on.
Ryan Kaldari
On 7/3/11 3:15 AM, Roan Kattouw wrote:
On Sun, Jul 3, 2011 at 3:21 AM, Ryan Kaldarirkaldari@wikimedia.org wrote:
Instead of getting a result from the API, he gets a 403 error: "The website declined to show this webpage / HTTP 403"
Is he using IE6? I think we still haven't deployed the proper fixes for the IE6 extension issue yet, although they did go into the 1.17 release.
Roan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Ryan Kaldari rkaldari@wikimedia.org wrote:
No, he's using IE8 and it does send his User Agent header: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)
Are there any other reasons why someone would be rejected from the API? Getting imageinfo doesn't require a token, so I'm at a loss as to what is going on. [...]
It seems to be related to the User-Agent header:
| [tim@passepartout ~]$ wget --user-agent="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)" 'http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil...' | --2011-07-03 19:28:45-- http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil... | Resolving en.wikipedia.org... 91.198.174.232 | Connecting to en.wikipedia.org|91.198.174.232|:80... connected. | HTTP request sent, awaiting response... 403 Forbidden | 2011-07-03 19:28:45 ERROR 403: Forbidden.
| [tim@passepartout ~]$ wget --user-agent="abc" 'http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil... 19:28:47-- http://en.wikipedia.org/w/api.php?action=query&format=xml&titles=Fil... | Resolving en.wikipedia.org... 91.198.174.232 | Connecting to en.wikipedia.org|91.198.174.232|:80... connected. | HTTP request sent, awaiting response... 200 OK | Length: 295 [text/xml] | Saving to: `api.php?action=query&format=xml&titles=File:Trophy.png&prop=imageinfo'
| 100%[===================================================================================================================>] 295 --.-K/s in 0.001s
| 2011-07-03 19:28:48 (386 KB/s) - `api.php?action=query&format=xml&titles=File:Trophy.png&prop=imageinfo' saved [295/295]
| [tim@passepartout ~]$
Tim
On Sun, Jul 3, 2011 at 9:13 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
No, he's using IE8 and it does send his User Agent header: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)
Are there any other reasons why someone would be rejected from the API?
Ah yes, the IE6 detection code just looks for "MSIE" in the User-Agent header :D
This'll all be fixed when I deploy the new code for dealing with the IE6 issue. I will probably do this on Thursday.
Roan Kattouw (Catrope)
That means that the Make Your Own feature in WikiLove and some of the features in UploadWizard are broken for almost half our users. Is there any way to deploy that fix sooner than Thursday? Alternately, could we do a live fix by changing "MSIE" to "MSIE 6" on the cluster? (I have no idea if that is totally insane or a reasonable suggestion.)
Ryan Kaldari
On 7/3/11 12:46 PM, Roan Kattouw wrote:
On Sun, Jul 3, 2011 at 9:13 PM, Ryan Kaldarirkaldari@wikimedia.org wrote:
No, he's using IE8 and it does send his User Agent header: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)
Are there any other reasons why someone would be rejected from the API?
Ah yes, the IE6 detection code just looks for "MSIE" in the User-Agent header :D
This'll all be fixed when I deploy the new code for dealing with the IE6 issue. I will probably do this on Thursday.
Roan Kattouw (Catrope)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sun, Jul 3, 2011 at 10:09 PM, Ryan Kaldari rkaldari@wikimedia.org wrote:
That means that the Make Your Own feature in WikiLove and some of the features in UploadWizard are broken for almost half our users. Is there any way to deploy that fix sooner than Thursday? Alternately, could we do a live fix by changing "MSIE" to "MSIE 6" on the cluster? (I have no idea if that is totally insane or a reasonable suggestion.)
If someone else (Tim?) were to do it, it could be done sooner. It shouldn't be done today (Sunday) or tomorrow (Independence Day) because lots of people have those days off. Given the way Tim's timezone and schedule correlates with SF office hours, I guess the earliest time he could do it is Wednesday morning his time, i.e. Tuesday afternoon PDT. I will be traveling starting Tuesday morning PDT and won't be able to do this (either due to lack of reliable internet access, or lack of sleep) until Thursday morning PDT.
Roan Kattouw (Catrope)
If I am up to date here, the fix that exists in trunk for the API-handling code is still not going to help us. It merely excludes highly improbable "extensions" like ".jpg&foo=bar&quux=blarg".
But, what if your query arguments *legitimately* end with an extension like "http://en.wikipedia.org/w/api.php?action=doSomething&page=File:Something..." ? You can't depend on query argument order.
The only solution is to encode our queryargs differently.
So, in the last deploy I deployed a workaround for this in UploadWizard. At the last stage before firing the AJAX query I convert any '.' in the query data to '%2E'.
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/90649
This won't trigger any bizarre bugs in IE6 and your code (if properly written) should never know anything happened. We could advise other consumers of our API to do the same.
BTW, this is a fairly tricky fix as it relies on certain quirks of jQuery 1.3, which is what we have deployed. Also, it was easier for me to deploy this fix for UW because all its ajax calls go through an API object.
In jQuery 1.4 you can fix this in a more standardized way, with ajax filters.
Of course it would be better if we just fixed the XSS prevention somehow, but I don't see how that's possible given the constraints.
On 7/3/11 1:09 PM, Ryan Kaldari wrote:
That means that the Make Your Own feature in WikiLove and some of the features in UploadWizard are broken for almost half our users. Is there any way to deploy that fix sooner than Thursday? Alternately, could we do a live fix by changing "MSIE" to "MSIE 6" on the cluster? (I have no idea if that is totally insane or a reasonable suggestion.)
Ryan Kaldari
On 7/3/11 12:46 PM, Roan Kattouw wrote:
On Sun, Jul 3, 2011 at 9:13 PM, Ryan Kaldarirkaldari@wikimedia.org wrote:
No, he's using IE8 and it does send his User Agent header: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; .NET4.0C)
Are there any other reasons why someone would be rejected from the API?
Ah yes, the IE6 detection code just looks for "MSIE" in the User-Agent header :D
This'll all be fixed when I deploy the new code for dealing with the IE6 issue. I will probably do this on Thursday.
Roan Kattouw (Catrope)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sun, Jul 3, 2011 at 10:27 PM, Neil Kandalgaonkar neilk@wikimedia.org wrote:
If I am up to date here, the fix that exists in trunk for the API-handling code is still not going to help us. It merely excludes highly improbable "extensions" like ".jpg&foo=bar&quux=blarg".
But, what if your query arguments *legitimately* end with an extension like "http://en.wikipedia.org/w/api.php?action=doSomething&page=File:Something..." ? You can't depend on query argument order.
That's riht.
The only solution is to encode our queryargs differently.
So, in the last deploy I deployed a workaround for this in UploadWizard. At the last stage before firing the AJAX query I convert any '.' in the query data to '%2E'.
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/90649
This won't trigger any bizarre bugs in IE6 and your code (if properly written) should never know anything happened. We could advise other consumers of our API to do the same.
I guess that could work.
jQuery 1.3, which is what we have deployed.
ORLY?
$.fn.jquery
"1.4.2"
Also, it was easier for me to deploy this fix for UW because all its ajax calls go through an API object.
In jQuery 1.4 you can fix this in a more standardized way, with ajax filters.
Well as I just pointed out, we have 1.4.2 on the cluster, and 1.6.x in trunk IIRC. So let's just do that then.
Roan Kattouw (Catrope)
On 7/3/11 2:55 PM, Roan Kattouw wrote:
Well as I just pointed out, we have 1.4.2 on the cluster, and 1.6.x in trunk IIRC. So let's just do that then.
Er, sorry, my TARDIS was set to 2009 or something.
I got the version numbers wrong in my post, but the basic premise stands. You can fix it with ajax prefilters in jQuery >1.5, and in 1.4 you must stringify the data attribute of the AJAX call and s/./%2e/g; before sending it.
Tim has fixed the issue on the live wikis. I imagine he just pushed up the fix that was already on trunk, so we probably still need to fix it for the case mentioned by Neil.
Ryan Kaldari
On 7/3/11 11:41 PM, Neil Kandalgaonkar wrote:
On 7/3/11 2:55 PM, Roan Kattouw wrote:
Well as I just pointed out, we have 1.4.2 on the cluster, and 1.6.x in trunk IIRC. So let's just do that then.
Er, sorry, my TARDIS was set to 2009 or something.
I got the version numbers wrong in my post, but the basic premise stands. You can fix it with ajax prefilters in jQuery>1.5, and in 1.4 you must stringify the data attribute of the AJAX call and s/./%2e/g; before sending it.
----- Original Message -----
From: "Ryan Kaldari" rkaldari@wikimedia.org
That means that the Make Your Own feature in WikiLove and some of the features in UploadWizard are broken for almost half our users. Is there any way to deploy that fix sooner than Thursday? Alternately, could we do a live fix by changing "MSIE" to "MSIE 6" on the cluster? (I have no idea if that is totally insane or a reasonable suggestion.)
So that we have a good feeling for impact, given Roun's reply
----- Original Message -----
From: "Ryan Kaldari" rkaldari@wikimedia.org
That means that the Make Your Own feature in WikiLove and some of the features in UploadWizard are broken for almost half our users. Is there any way to deploy that fix sooner than Thursday? Alternately, could we do a live fix by changing "MSIE" to "MSIE 6" on the cluster? (I have no idea if that is totally insane or a reasonable suggestion.)
So that there's a clear idea of impact:
"half the users *who use those features*" are impacted?
or
"half of WPs users, who are using IE != 6, but not all of them are using those features"?
Cheers, -- jra
On 7/3/11 8:31 PM, Jay Ashworth wrote:
So that there's a clear idea of impact: "half the users *who use those features*" are impacted?
or
"half of WPs users, who are using IE != 6, but not all of them are using those features"?
About 40% of the users of Wikipedia are using Internet Explorer. For all of those users, unless I am misunderstanding the problem, some of our API functions are currently disabled. We have a few deployed extensions that make use of these API functions specifically. The only two I know about personally are WikiLove and UploadWizard. It sounds like UploadWizard is already working around this problem according to Neil, so the impact may be limited to WikiLove and individual/bot API use. Within WikiLove, the problem seems to be limited to the "make your own" feature. There may be other extensions that are affected as well, but I haven't checked.
Ryan Kaldari
wikitech-l@lists.wikimedia.org