As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html), the default continuation mode for action=query requests to api.php will be changing to be easier for new coders to use correctly.
*The date is now set:* we intend to merge the change to ride the deployment train at the end of June. That should be 1.26wmf12, to be deployed to test wikis on June 30, non-Wikipedias on July 1, and Wikipedias on July 2.
If your bot or script is receiving the warning about this upcoming change (as seen here https://www.mediawiki.org/w/api.php?action=query&list=allpages, for example), it's time to fix your code!
- The simple solution is to simply include the "rawcontinue" parameter with your request to continue receiving the raw continuation data ( example https://www.mediawiki.org/w/api.php?action=query&list=allpages&rawcontinue=1). No other code changes should be necessary. - Or you could update your code to use the simplified continuation documented at https://www.mediawiki.org/wiki/API:Query#Continuing_queries (example https://www.mediawiki.org/w/api.php?action=query&list=allpages&continue=), which is much easier for clients to implement correctly.
Either of the above solutions may be tested immediately, you'll know it works because you stop seeing the warning.
I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
AAlertBot AboHeidiBot AbshirBot Acebot Ameenbot ArnauBot Beau.bot Begemot-Bot BeneBot* BeriBot BOT-Superzerocool CalakBot CamelBot CandalBot CategorizationBot CatWatchBot ClueBot_III ClueBot_NG CobainBot CorenSearchBot Cyberbot_I Cyberbot_II DanmicholoBot DeltaQuadBot Dexbot Dibot EdinBot ElphiBot ErfgoedBot Faebot Fatemibot FawikiPatroller HAL HasteurBot HerculeBot Hexabot HRoestBot IluvatarBot Invadibot Irclogbot Irfan-bot Jimmy-abot JYBot Krdbot Legobot Lowercase_sigmabot_III MahdiBot MalarzBOT MastiBot Merge_bot NaggoBot NasirkhanBot NirvanaBot Obaid-bot PatruBOT PBot Phe-bot Rezabot RMCD_bot Shuaib-bot SineBot SteinsplitterBot SvickBOT TaxonBot Theo's_Little_Bot W2Bot WLE-SpainBot Xqbot YaCBot ZedlikBot ZkBot
It means all compat bots will break and since pywikibot team won't support compat and won't change the code to support the new API, We are planning to deprecate it [1]
So please compat bot operators, do update your codes to core, read [2]
[1]: https://phabricator.wikimedia.org/T99365 [2]: https://www.mediawiki.org/wiki/Manual:Pywikibot/Compat_deprecation
Best
On Wed, Jun 3, 2015 at 1:13 AM Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html), the default continuation mode for action=query requests to api.php will be changing to be easier for new coders to use correctly.
*The date is now set:* we intend to merge the change to ride the deployment train at the end of June. That should be 1.26wmf12, to be deployed to test wikis on June 30, non-Wikipedias on July 1, and Wikipedias on July 2.
If your bot or script is receiving the warning about this upcoming change (as seen here https://www.mediawiki.org/w/api.php?action=query&list=allpages, for example), it's time to fix your code!
- The simple solution is to simply include the "rawcontinue" parameter
with your request to continue receiving the raw continuation data ( example < https://www.mediawiki.org/w/api.php?action=query&list=allpages&rawco...
).
No other code changes should be necessary.
- Or you could update your code to use the simplified continuation
documented at https://www.mediawiki.org/wiki/API:Query#Continuing_queries (example < https://www.mediawiki.org/w/api.php?action=query&list=allpages&conti...
),
which is much easier for clients to implement correctly.
Either of the above solutions may be tested immediately, you'll know it works because you stop seeing the warning.
I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23-29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
AAlertBot AboHeidiBot AbshirBot Acebot Ameenbot ArnauBot Beau.bot Begemot-Bot BeneBot* BeriBot BOT-Superzerocool CalakBot CamelBot CandalBot CategorizationBot CatWatchBot ClueBot_III ClueBot_NG CobainBot CorenSearchBot Cyberbot_I Cyberbot_II DanmicholoBot DeltaQuadBot Dexbot Dibot EdinBot ElphiBot ErfgoedBot Faebot Fatemibot FawikiPatroller HAL HasteurBot HerculeBot Hexabot HRoestBot IluvatarBot Invadibot Irclogbot Irfan-bot Jimmy-abot JYBot Krdbot Legobot Lowercase_sigmabot_III MahdiBot MalarzBOT MastiBot Merge_bot NaggoBot NasirkhanBot NirvanaBot Obaid-bot PatruBOT PBot Phe-bot Rezabot RMCD_bot Shuaib-bot SineBot SteinsplitterBot SvickBOT TaxonBot Theo's_Little_Bot W2Bot WLE-SpainBot Xqbot YaCBot ZedlikBot ZkBot
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am wondering why we need such changes in the app---
Best :-) Steinsplitter
Date: Tue, 2 Jun 2015 16:42:47 -0400 From: bjorsch@wikimedia.org To: wikitech-l@lists.wikimedia.org; mediawiki-api-announce@lists.wikimedia.org Subject: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html), the default continuation mode for action=query requests to api.php will be changing to be easier for new coders to use correctly.
*The date is now set:* we intend to merge the change to ride the deployment train at the end of June. That should be 1.26wmf12, to be deployed to test wikis on June 30, non-Wikipedias on July 1, and Wikipedias on July 2.
If your bot or script is receiving the warning about this upcoming change (as seen here https://www.mediawiki.org/w/api.php?action=query&list=allpages, for example), it's time to fix your code!
- The simple solution is to simply include the "rawcontinue" parameter
with your request to continue receiving the raw continuation data ( example https://www.mediawiki.org/w/api.php?action=query&list=allpages&rawcontinue=1). No other code changes should be necessary.
- Or you could update your code to use the simplified continuation
documented at https://www.mediawiki.org/wiki/API:Query#Continuing_queries (example https://www.mediawiki.org/w/api.php?action=query&list=allpages&continue=), which is much easier for clients to implement correctly.
Either of the above solutions may be tested immediately, you'll know it works because you stop seeing the warning.
I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
AAlertBot AboHeidiBot AbshirBot Acebot Ameenbot ArnauBot Beau.bot Begemot-Bot BeneBot* BeriBot BOT-Superzerocool CalakBot CamelBot CandalBot CategorizationBot CatWatchBot ClueBot_III ClueBot_NG CobainBot CorenSearchBot Cyberbot_I Cyberbot_II DanmicholoBot DeltaQuadBot Dexbot Dibot EdinBot ElphiBot ErfgoedBot Faebot Fatemibot FawikiPatroller HAL HasteurBot HerculeBot Hexabot HRoestBot IluvatarBot Invadibot Irclogbot Irfan-bot Jimmy-abot JYBot Krdbot Legobot Lowercase_sigmabot_III MahdiBot MalarzBOT MastiBot Merge_bot NaggoBot NasirkhanBot NirvanaBot Obaid-bot PatruBOT PBot Phe-bot Rezabot RMCD_bot Shuaib-bot SineBot SteinsplitterBot SvickBOT TaxonBot Theo's_Little_Bot W2Bot WLE-SpainBot Xqbot YaCBot ZedlikBot ZkBot
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Steinsplitter, just to be clear, you meant "API", not "app", correct?
On Wed, Jun 3, 2015 at 3:49 AM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am wondering why we need such changes in the app---
Best :-) Steinsplitter
Date: Tue, 2 Jun 2015 16:42:47 -0400 From: bjorsch@wikimedia.org To: wikitech-l@lists.wikimedia.org;
mediawiki-api-announce@lists.wikimedia.org
Subject: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for
action=query will change at the end of this month
As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html
),
the default continuation mode for action=query requests to api.php will
be
changing to be easier for new coders to use correctly.
*The date is now set:* we intend to merge the change to ride the
deployment
train at the end of June. That should be 1.26wmf12, to be deployed to
test
wikis on June 30, non-Wikipedias on July 1, and Wikipedias on July 2.
If your bot or script is receiving the warning about this upcoming change (as seen here https://www.mediawiki.org/w/api.php?action=query&list=allpages, for example), it's time to fix your code!
- The simple solution is to simply include the "rawcontinue" parameter
with your request to continue receiving the raw continuation data ( example <
https://www.mediawiki.org/w/api.php?action=query&list=allpages&rawco...
). No other code changes should be necessary.
- Or you could update your code to use the simplified continuation
documented at
https://www.mediawiki.org/wiki/API:Query#Continuing_queries
(example <
https://www.mediawiki.org/w/api.php?action=query&list=allpages&conti...
), which is much easier for clients to implement correctly.
Either of the above solutions may be tested immediately, you'll know it works because you stop seeing the warning.
I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
AAlertBot AboHeidiBot AbshirBot Acebot Ameenbot ArnauBot Beau.bot Begemot-Bot BeneBot* BeriBot BOT-Superzerocool CalakBot CamelBot CandalBot CategorizationBot CatWatchBot ClueBot_III ClueBot_NG CobainBot CorenSearchBot Cyberbot_I Cyberbot_II DanmicholoBot DeltaQuadBot Dexbot Dibot EdinBot ElphiBot ErfgoedBot Faebot Fatemibot FawikiPatroller HAL HasteurBot HerculeBot Hexabot HRoestBot IluvatarBot Invadibot Irclogbot Irfan-bot Jimmy-abot JYBot Krdbot Legobot Lowercase_sigmabot_III MahdiBot MalarzBOT MastiBot Merge_bot NaggoBot NasirkhanBot NirvanaBot Obaid-bot PatruBOT PBot Phe-bot Rezabot RMCD_bot Shuaib-bot SineBot SteinsplitterBot SvickBOT TaxonBot Theo's_Little_Bot W2Bot WLE-SpainBot Xqbot YaCBot ZedlikBot ZkBot
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
yes.
Date: Wed, 3 Jun 2015 04:25:10 -0700 From: abaso@wikimedia.org To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
Steinsplitter, just to be clear, you meant "API", not "app", correct?
On Wed, Jun 3, 2015 at 3:49 AM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am wondering why we need such changes in the app---
Best :-) Steinsplitter
Date: Tue, 2 Jun 2015 16:42:47 -0400 From: bjorsch@wikimedia.org To: wikitech-l@lists.wikimedia.org;
mediawiki-api-announce@lists.wikimedia.org
Subject: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for
action=query will change at the end of this month
As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html
),
the default continuation mode for action=query requests to api.php will
be
changing to be easier for new coders to use correctly.
*The date is now set:* we intend to merge the change to ride the
deployment
train at the end of June. That should be 1.26wmf12, to be deployed to
test
wikis on June 30, non-Wikipedias on July 1, and Wikipedias on July 2.
If your bot or script is receiving the warning about this upcoming change (as seen here https://www.mediawiki.org/w/api.php?action=query&list=allpages, for example), it's time to fix your code!
- The simple solution is to simply include the "rawcontinue" parameter
with your request to continue receiving the raw continuation data ( example <
https://www.mediawiki.org/w/api.php?action=query&list=allpages&rawco...
). No other code changes should be necessary.
- Or you could update your code to use the simplified continuation
documented at
https://www.mediawiki.org/wiki/API:Query#Continuing_queries
(example <
https://www.mediawiki.org/w/api.php?action=query&list=allpages&conti...
), which is much easier for clients to implement correctly.
Either of the above solutions may be tested immediately, you'll know it works because you stop seeing the warning.
I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
AAlertBot AboHeidiBot AbshirBot Acebot Ameenbot ArnauBot Beau.bot Begemot-Bot BeneBot* BeriBot BOT-Superzerocool CalakBot CamelBot CandalBot CategorizationBot CatWatchBot ClueBot_III ClueBot_NG CobainBot CorenSearchBot Cyberbot_I Cyberbot_II DanmicholoBot DeltaQuadBot Dexbot Dibot EdinBot ElphiBot ErfgoedBot Faebot Fatemibot FawikiPatroller HAL HasteurBot HerculeBot Hexabot HRoestBot IluvatarBot Invadibot Irclogbot Irfan-bot Jimmy-abot JYBot Krdbot Legobot Lowercase_sigmabot_III MahdiBot MalarzBOT MastiBot Merge_bot NaggoBot NasirkhanBot NirvanaBot Obaid-bot PatruBOT PBot Phe-bot Rezabot RMCD_bot Shuaib-bot SineBot SteinsplitterBot SvickBOT TaxonBot Theo's_Little_Bot W2Bot WLE-SpainBot Xqbot YaCBot ZedlikBot ZkBot
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jun 3, 2015 at 6:49 AM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am wondering why we need such changes in the [API]
We usually do. In this case, however, the advantages of changing the default for new API users seems to outweigh the disadvantages of a well-announced breaking change with a simple parameter to request backwards-compatible output.
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with between 1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok).
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
That's not impossible, but I wouldn't count on it.
Hi,
Brad Jorsch (Anomie) schreef op 3-6-2015 om 18:43:
<not an announcement>
Could you please stop using the announce list for chatter? Probably better to put the announce list on moderation.
Maarten
Out of curiosity, this may also break all versions of huggle that are older than 3.15, which was released sometime around November 2014.
I strongly recommend you to upgrade if you are using such an ancient version. I don't know what is going to happen, but it may break. Unless someone patches the legacy huggle version, it will probably remain defunct.
On Wed, Jun 3, 2015 at 11:23 PM, Maarten Dammers maarten@mdammers.nl wrote:
Hi,
Brad Jorsch (Anomie) schreef op 3-6-2015 om 18:43:
<not an announcement>
Could you please stop using the announce list for chatter? Probably better to put the announce list on moderation.
Maarten
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
* 3.1.5 not 3.15, sorry for confusion :P
On Thu, Jun 4, 2015 at 3:19 PM, Petr Bena benapetr@gmail.com wrote:
Out of curiosity, this may also break all versions of huggle that are older than 3.15, which was released sometime around November 2014.
I strongly recommend you to upgrade if you are using such an ancient version. I don't know what is going to happen, but it may break. Unless someone patches the legacy huggle version, it will probably remain defunct.
On Wed, Jun 3, 2015 at 11:23 PM, Maarten Dammers maarten@mdammers.nl wrote:
Hi,
Brad Jorsch (Anomie) schreef op 3-6-2015 om 18:43:
<not an announcement>
Could you please stop using the announce list for chatter? Probably better to put the announce list on moderation.
Maarten
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Would it make sense to introduce per-user-agent API black list? This way we could prevent accidental usage of the broken tools.
On Thu, Jun 4, 2015 at 4:19 PM, Petr Bena benapetr@gmail.com wrote:
- 3.1.5 not 3.15, sorry for confusion :P
On Thu, Jun 4, 2015 at 3:19 PM, Petr Bena benapetr@gmail.com wrote:
Out of curiosity, this may also break all versions of huggle that are older than 3.15, which was released sometime around November 2014.
I strongly recommend you to upgrade if you are using such an ancient version. I don't know what is going to happen, but it may break. Unless someone patches the legacy huggle version, it will probably remain defunct.
On Wed, Jun 3, 2015 at 11:23 PM, Maarten Dammers maarten@mdammers.nl
wrote:
Hi,
Brad Jorsch (Anomie) schreef op 3-6-2015 om 18:43:
<not an announcement>
Could you please stop using the announce list for chatter? Probably
better
to put the announce list on moderation.
Maarten
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) <bjorsch@wikimedia.org
wrote:
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with between 1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok).
Legal recommended we only post the list of bots, not the human accounts. These are:
AHbot AsuraBot Autobot BattyBot Bibcode_Bot Bottuzzu ChenzwBot Cydebot DickensBot DrTrigonBot DSisyphBot DumbBOT DYKHousekeepingBot DYKUpdateBot FBot GiftBot GrantsBot HangsnaBot HangsnaBot2 ImageRemovalBot InceptionBot JackBot JBot Jimmy-bot Kenrick95Bot KrBot KrinkleBot LivingBot MalafayaBot MaraBot MauroBot MBHbot Mr.Z-bot NowCommons-Sichtbot Olafbot PereBot PseudoBot QianBot Rainbot Reports_bot RFF-Bot Salebot Sanjeev_bot SemperBlottoBot SergoBot SHBot Steenthbot TurkászBot UWCTransferBot VlsergeyBot VriuBot YiFeiBot Yobot ZacheBot Zlobot
Note this list is still from May 23–29; a bot appearing in this list may have been updated since then.
On Wed, 2015-06-10 at 15:01 -0400, Brad Jorsch (Anomie) wrote:
Legal recommended we only post the list of bots, not the human accounts. These are:
AHbot AsuraBot
[...]
Note this list is still from May 23–29; a bot appearing in this list may have been updated since then.
As there are seven days left, does someone have time / capacity to provide an updated list of bots that likely haven't seen updates yet?
Also, in [1], Sitic came up with a query of de.wp *gadgets* affected: https://de.wikipedia.org/w/index.php?title=Spezial:Suche&limit=100&o...
Has there been on-wiki outreach to *gadget* maintainers?
Thanks in advance! andre
[1] https://de.wikipedia.org/wiki/Wikipedia:Bots/Notizen#Hinweis_an_alle_Bot-Bet...
On Wed, Jun 24, 2015 at 2:43 AM, Andre Klapper aklapper@wikimedia.org wrote:
As there are seven days left, does someone have time / capacity to provide an updated list of bots that likely haven't seen updates yet?
Also, in [1], Sitic came up with a query of de.wp *gadgets* affected: https://de.wikipedia.org/w/index.php?title=Spezial:Suche&limit=100&o...
Has there been on-wiki outreach to *gadget* maintainers?
Thanks in advance! andre
[1] https://de.wikipedia.org/wiki/Wikipedia:Bots/Notizen#Hinweis_an_alle_Bot-Bet...
Did the change go live? Did the wikis fall over?
Is there any ongoing analysis of the number of user agents (bots / gadgets / etc) still not specifying continue method?
Is there a common approach for gadgets to identify themselves to the API - e.g. setting a custom user-agent?
-- John Vandenberg
On Mon, Jul 6, 2015 at 9:02 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
Did the change go live?
Yes!
Did the wikis fall over?
If they did, no one told me about it or complained about it in any of the places I looked. Not that I personally looked in extremely many places, but I'd have expected it to make it to this mailing list or Phabricator if something important broke.
Is there any ongoing analysis of the number of user agents (bots / gadgets / etc) still not specifying continue method?
The log warnings I used before the change were removed as part of the change, since it doesn't seem particularly interesting anymore: we can't tell server-side whether a client is actually broken or just isn't using continuation at all.
I can tell you that the gzip-compressed api-feature-usage log files were around 2.5G on June 1 (just before the final communication push), 1.1G on July 1 (just before the change), and only 46M on July 4 (after the change with the logging for this removed).
Is there a common approach for gadgets to identify themselves to the
API - e.g. setting a custom user-agent?
Clients that cannot set the User-Agent header (such as gadgets and other scripts running in a browser) may use the Api-User-Agent header instead. If present, this header is prepended to the standard User-Agent header for purposes such as the api-feature-usage log and Special:ApiFeatureUsage.
On Tue, Jul 7, 2015 at 4:20 PM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
Is there a common approach for gadgets to identify themselves to the
API - e.g. setting a custom user-agent?
Clients that cannot set the User-Agent header (such as gadgets and other scripts running in a browser) may use the Api-User-Agent header instead. If present, this header is prepended to the standard User-Agent header for purposes such as the api-feature-usage log and Special:ApiFeatureUsage.
Only very few do so unfortunately.. We should see if we can figure out some strategy to encourage that a bit more... Add a 'user-agent' option into the JS mediawiki.Api lib for instance ? log if people then not use it ? Might be a bit 'loud'. Other ideas ?
DJ
Date: Tue, 7 Jul 2015 10:20:33 -0400 From: bjorsch@wikimedia.org
On Mon, Jul 6, 2015 at 9:02 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
Did the change go live?
Yes!
Did the wikis fall over?
If they did, no one told me about it or complained about it in any of the places I looked. Not that I personally looked in extremely many places, but I'd have expected it to make it to this mailing list or Phabricator if something important broke.
You can't expect the average user to go to Phabricator or post on this mailing list. If you are lucky, they complain on the respective talk pages like [1] or some other place.
I think, there is be quite a substantial amount of programs around which still use the old interface and are broken now. Waiting for someone to notice and fix...
Marco
[1] https://commons.wikimedia.org/w/index.php?title=MediaWiki_talk%3AGadget-Cat-...
Is there any ongoing analysis of the number of user agents (bots / gadgets / etc) still not specifying continue method?
The log warnings I used before the change were removed as part of the change, since it doesn't seem particularly interesting anymore: we can't tell server-side whether a client is actually broken or just isn't using continuation at all.
I can tell you that the gzip-compressed api-feature-usage log files were around 2.5G on June 1 (just before the final communication push), 1.1G on July 1 (just before the change), and only 46M on July 4 (after the change with the logging for this removed).
Is there a common approach for gadgets to identify themselves to the
API - e.g. setting a custom user-agent?
Clients that cannot set the User-Agent header (such as gadgets and other scripts running in a browser) may use the Api-User-Agent header instead. If present, this header is prepended to the standard User-Agent header for purposes such as the api-feature-usage log and Special:ApiFeatureUsage.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
All this unnecessary changes is just making wikicoders live hard. *sigh*
From: maic23@live.de To: wikitech-l@lists.wikimedia.org Date: Wed, 8 Jul 2015 08:17:41 +0200 Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
Date: Tue, 7 Jul 2015 10:20:33 -0400 From: bjorsch@wikimedia.org
On Mon, Jul 6, 2015 at 9:02 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
Did the change go live?
Yes!
Did the wikis fall over?
If they did, no one told me about it or complained about it in any of the places I looked. Not that I personally looked in extremely many places, but I'd have expected it to make it to this mailing list or Phabricator if something important broke.
You can't expect the average user to go to Phabricator or post on this mailing list. If you are lucky, they complain on the respective talk pages like [1] or some other place.
I think, there is be quite a substantial amount of programs around which still use the old interface and are broken now. Waiting for someone to notice and fix...
Marco
[1] https://commons.wikimedia.org/w/index.php?title=MediaWiki_talk%3AGadget-Cat-...
Is there any ongoing analysis of the number of user agents (bots / gadgets / etc) still not specifying continue method?
The log warnings I used before the change were removed as part of the change, since it doesn't seem particularly interesting anymore: we can't tell server-side whether a client is actually broken or just isn't using continuation at all.
I can tell you that the gzip-compressed api-feature-usage log files were around 2.5G on June 1 (just before the final communication push), 1.1G on July 1 (just before the change), and only 46M on July 4 (after the change with the logging for this removed).
Is there a common approach for gadgets to identify themselves to the
API - e.g. setting a custom user-agent?
Clients that cannot set the User-Agent header (such as gadgets and other scripts running in a browser) may use the Api-User-Agent header instead. If present, this header is prepended to the standard User-Agent header for purposes such as the api-feature-usage log and Special:ApiFeatureUsage.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I run GrantsBot, which is listed here.
I've updated all GrantsBot API requests to use rawcontinue=1. But as I read through this thread, it's not clear to me that that's the problem. I can't find a single instance in my code where I'm actually continuing a query. Does this breaking change only apply be an issue if you were using querycontinue in the first place?
I'm sure this has been covered elsewhere or I'm missing something obvious, but I'd be grateful for specific confirmation :)
Best, Jonathan
On Wed, Jun 10, 2015 at 12:01 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org
wrote:
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with
between
1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok).
Legal recommended we only post the list of bots, not the human accounts. These are:
AHbot AsuraBot Autobot BattyBot Bibcode_Bot Bottuzzu ChenzwBot Cydebot DickensBot DrTrigonBot DSisyphBot DumbBOT DYKHousekeepingBot DYKUpdateBot FBot GiftBot GrantsBot HangsnaBot HangsnaBot2 ImageRemovalBot InceptionBot JackBot JBot Jimmy-bot Kenrick95Bot KrBot KrinkleBot LivingBot MalafayaBot MaraBot MauroBot MBHbot Mr.Z-bot NowCommons-Sichtbot Olafbot PereBot PseudoBot QianBot Rainbot Reports_bot RFF-Bot Salebot Sanjeev_bot SemperBlottoBot SergoBot SHBot Steenthbot TurkászBot UWCTransferBot VlsergeyBot VriuBot YiFeiBot Yobot ZacheBot Zlobot
Note this list is still from May 23–29; a bot appearing in this list may have been updated since then.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Jonathan, in theory, if you never use query continuation, you don't need to do anything with your code - you just need to tell the community that the bot is ok to use the new continuation system. This way noone will block the bot just in case. It is much better than to use "rawcontinue", because that flag will keep telling us someone needs the old system.
On Mon, Jun 29, 2015 at 8:01 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I run GrantsBot, which is listed here.
I've updated all GrantsBot API requests to use rawcontinue=1. But as I read through this thread, it's not clear to me that that's the problem. I can't find a single instance in my code where I'm actually continuing a query. Does this breaking change only apply be an issue if you were using querycontinue in the first place?
I'm sure this has been covered elsewhere or I'm missing something obvious, but I'd be grateful for specific confirmation :)
Best, Jonathan
On Wed, Jun 10, 2015 at 12:01 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Wed, Jun 3, 2015 at 12:43 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org
wrote:
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg <jayvdb@gmail.com
wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with
between
1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB
if
that's not fixed (someone please check!), or the like. For comparison,
in
the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost
but
not entirely sure it would be ok).
Legal recommended we only post the list of bots, not the human accounts. These are:
AHbot AsuraBot Autobot BattyBot Bibcode_Bot Bottuzzu ChenzwBot Cydebot DickensBot DrTrigonBot DSisyphBot DumbBOT DYKHousekeepingBot DYKUpdateBot FBot GiftBot GrantsBot HangsnaBot HangsnaBot2 ImageRemovalBot InceptionBot JackBot JBot Jimmy-bot Kenrick95Bot KrBot KrinkleBot LivingBot MalafayaBot MaraBot MauroBot MBHbot Mr.Z-bot NowCommons-Sichtbot Olafbot PereBot PseudoBot QianBot Rainbot Reports_bot RFF-Bot Salebot Sanjeev_bot SemperBlottoBot SergoBot SHBot Steenthbot TurkászBot UWCTransferBot VlsergeyBot VriuBot YiFeiBot Yobot ZacheBot Zlobot
Note this list is still from May 23–29; a bot appearing in this list may have been updated since then.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- Jonathan T. Morgan Senior Design Researcher Wikimedia Foundation User:Jmorgan (WMF) https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Mon, Jun 29, 2015 at 1:53 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
It is much better than to use "rawcontinue", because that flag will keep telling us someone needs the old system.
I doubt we'll ever get rid of rawcontinue. There's practically no code to maintain there (it's just a one-line function[1] and a trivial if-block to call it[2]) so it's highly unlikely to be worth even minimal effort.
[1]: https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%... [2]: https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%...
Thanks Yuri and Brad!
On Mon, Jun 29, 2015 at 11:10 AM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Mon, Jun 29, 2015 at 1:53 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
It is much better than to use "rawcontinue", because that flag will keep telling us someone needs the old system.
I doubt we'll ever get rid of rawcontinue. There's practically no code to maintain there (it's just a one-line function[1] and a trivial if-block to call it[2]) so it's highly unlikely to be worth even minimal effort.
[1]:
https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%... [2]:
https://git.wikimedia.org/blob/mediawiki%2Fcore.git/26288853/includes%2Fapi%...
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hi Brad,
Brad Jorsch (Anomie) schreef op 2-6-2015 om 22:42:
As has been announced several times (most recently at https://lists.wikimedia.org/pipermail/wikitech-l/2015-April/081559.html), the default continuation mode for action=query requests to api.php will be changing to be easier for new coders to use correctly.
... CategorizationBot ... ErfgoedBot
Pulled the plug on both bots. You won't see them in your logs again. Some background at https://commons.wikimedia.org/wiki/Commons:Village_pump#Pulled_the_plug_on_C...
Maarten
wikitech-l@lists.wikimedia.org