I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will keep the logic as before.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie... On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Yuri Astrakhan wrote:
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them.
I feel that former MediaWiki Web API maintainers should actively pay attention to which mailing lists they're posting to. ;-) I doubt you intended to send this message to mediawiki-api-announce.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at https://www.mediawiki.org/wiki/Special:Permalink/1663950
I briefly looked at this proposal and I'm confused why you feel it's necessary to write yet another MediaWiki API library/framework in Python. Don't we already have mwlib, python-wikitools, and a half-dozen others?
If the intention is for this to be an abstraction layer between existing Python frameworks and the MediaWiki API, that seems even sillier to me as we've already created two layers of abstraction (MediaWiki PHP --> api.php and api.php --> Python framework). I don't see how a third layer of abstraction, even a "minimal" one, would help anything.
Or put another way: nearly every Python library has a login action that communicates with api.php. And obviously api.php has a login action that communicates with SpecialUserLogin.php and the rest of the MediaWiki authentication infrastructure. What would be solved here by implementing your proposal?
MZMcBride
Le 03/06/2015 15:43, MZMcBride a écrit :
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at https://www.mediawiki.org/wiki/Special:Permalink/1663950
I briefly looked at this proposal and I'm confused why you feel it's necessary to write yet another MediaWiki API library/framework in Python. Don't we already have mwlib, python-wikitools, and a half-dozen others?
Because https://xkcd.com/927/
More seriously, that follow up a workshop during the Lyon Hackathon.
Task: https://phabricator.wikimedia.org/T97950 Minutes: https://phabricator.wikimedia.org/P675
I think one of the outcome is you can now install the framework from pypi https://pypi.python.org/pypi/pywikibot
The low level classes that are robust will be extracted to a standalone lib which would let others build other softwares on top of it.
Or to say it otherwise, instead of reinventing the wheel it will be made available to everyone easily.
First, kudos to the API team for going the extra mile and reaching out to the community to guide them through this process. This doesn't effect the apps at the moment, but it's good to know you guys are thinking about clients and how API changes affect them.
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without making any changes?
On the other hand, if we don't want to support the old behavior, would it be better to simply return an error (e.g. HTTP 400) instead of breaking clients in a less explicit way? For example, as a client, I would prefer my code failed faster (bad request) instead of failing more-or-less silently.
Cheers,
Brian
On Wed, Jun 3, 2015 at 7:50 AM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will keep the logic as before.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at
https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie... On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
- mediawiki-api-annnounce
Sorry, didn't mean to CC the api announcement list. Seems like my message got bounced anyway.
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
First, kudos to the API team for going the extra mile and reaching out to the community to guide them through this process. This doesn't effect the apps at the moment, but it's good to know you guys are thinking about clients and how API changes affect them.
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without making any changes?
On the other hand, if we don't want to support the old behavior, would it be better to simply return an error (e.g. HTTP 400) instead of breaking clients in a less explicit way? For example, as a client, I would prefer my code failed faster (bad request) instead of failing more-or-less silently.
Cheers,
Brian
On Wed, Jun 3, 2015 at 7:50 AM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will keep the logic as before.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at
https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie... On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning
more
than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who
is,
please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without making any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
I would like to point out that it might be a good idea add &formatversion=1 for anyone who wants to lock the current formatting in place.
On Wed, Jun 3, 2015 at 8:13 PM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without making any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate at once.
Now it looks as API: we are deprecating what we promised to deprecated long ago - ok pywikipedia compat: did not handle the deprecation of API before, and are not going to fix copy-pasted in tens of places (not one place, it's never that simple) query builders to support "rawcontinue", we announce compat as discontinued together with the old style API. API deprecation was not coordinated with client library deprecation - not ok
If there is one year gap between two deprecations - ok, bot writers can choose either compat or core, and their bots can still work. Most users don't use APi directly so it should be the problem of coordination between API and clients developers.
Message: 1 Date: Wed, 3 Jun 2015 19:08:24 +0300 From: Yuri Astrakhan yastrakhan@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CALOOOkhzCUf14Tf= P0uQw7ZXAmwb9qAxs4kgvbJAvSsMQY82GA@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
MZ: >I feel that former MediaWiki Web API maintainers should actively pay attention to which mailing lists they're posting to. ;-) I doubt you intended to send this message to mediawiki-api-announce. Mz, I don't think I ever spent much time maintaining it :)) But yes, good point, reply all is evil at times :)
MZ: re why minimalistic lib - for most apis out there, people tend to write "reference implementation" - code that explains to would-be lib authors how to use API. It sets up prefered usage patterns and guidelines. We never had that for the api. This resulted in a large number of API libs that vary in how they use api. Pywikibot is a powerful platform, but is too big for many usecases (as discussed in Lyon). Hence the need.
SW: >Most operator are volunteers and don't have time to change the code every month because there is a change in the api.
- I might agree about every month, but we are talking about a feature that
has been out for 2 years...
The old one was imho perfectly.
- Most API users imho would laugh at this statement. See the roadmap
https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap which came as a result of analysing numerous bugs & wishes filed against the api, such as returning {} instead of [] for empty values, inconsistencies, the silly '*' value for text, and many other problems that have accumulated during the years. API might work for you, but it does not support multilingual error messaging, it overcomplicates the process of writing javascript clients, it does not easily cache. In short, lots of problems.
Was the new api coded by WMF or by volunteers?
- I wrote the original API as a volunteer (took about a year in 2005/6). I
recently coded the continuation as a volunteer, and Brad has improved it as a WMF employee. I later became a full time WMF employee as well, but my project was Zero, not API. As a volunteer, over 2 years ago I wrote a huge API improvement roadmap https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmapthat Brad picked up and greatly extended, and is driving it forward with the amazing improvements. From what I know, Brad has now been officially moved into position of working on the API. In other words, that employee vs volunteer line is very fuzzy. No point of splitting it that way.
TLDR;
In short, we need to make improvements if we want to move forward in turning API into a platform, with flexible JavaScript-driven interfaces such as Visual Editor. To allow creative uses that go beyond AWB, that support complex interactions, and not just the way for bots to make edits. Unfortunately, that means that despite our best efforts, very infrequently, some bots must be updated.
If the bot author cannot add a simple one line change "rawcontinue=1" once in two years because of their busy personal live, I don't think that person should be making automatic edits to Wikipedia - because sometimes bots make mistakes that require substantially more time involvement. I would not trust wikipedia edits to a bot runner under such circumstances. If the bot runner is not a programmer, they should get the latest version of their bot. If there is no up-to-date code because noone is maintaining it, it again should not be accessing wikipedia - we sometimes discover security bugs that require fixing, or because bot calls wiki too often, or other changes in content structure - e.g. introduction of WikiData for interwiki links required all interwiki bots to be updated.
Wikipedia is a living, developing ecosystem, and it does require updates to all parties accessing it. People accessing wikipedia from the older browsers discover that they no longer can do everything they used to many years ago - because we now use newer browser features, and fallback into basic mode for older browsers. Please participate in the evolution of the project.
On Wed, Jun 3, 2015 at 5:22 PM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho
perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the
technical
aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live.
-- Steinsplitter
Date: Wed, 3 Jun 2015 14:50:48 +0300 From: yastrakhan@wikimedia.org To: wikitech-l@lists.wikimedia.org CC: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation
mode
for action=query will change at the end of this month
I feel that bot operators should actively pay attention to the
technical
aspects of the community and the mailing lists. So, the bot operator
who
never updates their software, doesn't pay attention to the
announcements,
and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just
for
the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will
keep
the logic as before.
Lastly, I again would like to promote the idea discussed at the
hackathon
-- a client side minimalistic library that bigger frameworks like
pywikibot
rely on, and that is designed in part by the core developers. See the proposal at
https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie...
On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com
wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning
more
than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who
is,
please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the
scale
of bots operators that will be affected when this lands - currently
in
one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Message: 2 Date: Wed, 3 Jun 2015 12:34:12 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSuKej9O0beGt6xco+c+j7uTa0BtZoeCAQs9Cz6XQ2gYwg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 6:49 AM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am
wondering
why we need such changes in the [API]
We usually do. In this case, however, the advantages of changing the default for new API users seems to outweigh the disadvantages of a well-announced breaking change with a simple parameter to request backwards-compatible output.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 3 Date: Wed, 3 Jun 2015 12:43:36 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Cc: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSvprWamR3y+TFfcLg6Jv4hkbtCqbzj6JxfSvZe4v8f4Ow@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with between 1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok).
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
That's not impossible, but I wouldn't count on it.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 4 Date: Wed, 3 Jun 2015 13:13:34 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Cc: MediaWiki API announcements & discussion mediawiki-api@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CAEepRSsfin=praEeth2qQQFtwS_SHgFDbZFpXEd= uqORvn4sJA@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 5 Date: Wed, 3 Jun 2015 20:31:42 +0300 From: Yuri Astrakhan yastrakhan@wikimedia.org To: "MediaWiki API announcements & discussion" mediawiki-api@lists.wikimedia.org Cc: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] [Mediawiki-api] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CALOOOkjcnXoXCSMOQe5z58NR2iw24tf7qPcOkSXWT1qZLt30hQ@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
I would like to point out that it might be a good idea add &formatversion=1 for anyone who wants to lock the current formatting in place.
On Wed, Jun 3, 2015 at 8:13 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which
means
that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and
then
when to re-add it, and when to remove the meta or list modules. I
wouldn't
be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making
format=json&formatversion=2
the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Message: 6 Date: Wed, 3 Jun 2015 11:55:40 -0600 From: Brian Wolff bawolff@gmail.com To: Wikimedia developers wikitech-l@lists.wikimedia.org, mediawiki-api@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CA+oo+DUnkpwL_EsG7DmGQOq5OsSPJmH9FnL+eW_xbaOO6XFjdg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On 6/3/15, Steinsplitter Wiki steinsplitter-wiki@live.com wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should
keep
the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho
perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers.
Most
of them having a job and real live.
-- Steinsplitter
My understanding is that most of the people who were using the original continuation, were using it wrong, causing subtle bugs in their script. Thus the existing implementation was wasting considerable amount of volunteer bot developer time. In the long run this change will hopefully reduce the total amount of time spent by volunteer bot makers chasing weird bugs, at the expense of some short term pain.
Its always a challenge to balance backwards compatibility with fixing things that are causing problems. I think the API team is keenly aware of the frustrations that changes to the api cause, and try to make sure that intentional breakage only happens when the benefits truly outweigh the cons.
-- Bawolff
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
End of Wikitech-l Digest, Vol 143, Issue 8
On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko intracer@gmail.com wrote:
It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate at once.
Now it looks as API: we are deprecating what we promised to deprecated long ago - ok pywikipedia compat: did not handle the deprecation of API before, and are not going to fix copy-pasted in tens of places (not one place, it's never that simple) query builders to support "rawcontinue", we announce compat as discontinued together with the old style API. API deprecation was not coordinated with client library deprecation - not ok
If there is one year gap between two deprecations - ok, bot writers can choose either compat or core, and their bots can still work. Most users don't use APi directly so it should be the problem of coordination between API and clients developers.
On the pywikipedia side, it has been unofficially deprecated for a while, and the intention was to decommission compat as gracefully as possible, with Wikimania as the final killing ground.
The pywikibot developers roughly hashed out a decommissioning plan at the Lyon hackathon, and worked with WMF staff to start doing impact analysis. https://phabricator.wikimedia.org/T101214
Then the MW API continuation breakage was announced post Lyon. Hmm.
The continuation problem in pywikipedia / compat is not so much those 50 or so occurrences where continuation bugs appear to exist, but 1. testing those 50 or so occurrences 2. finding the other less obvious occurrences 3. scripts people have written using compat's query.GetData may also be broken, as it doesnt do continuation.
With a lot of wasted effort, 1 & 2 might be resolved so that 'compat' code works, and someone could create a hack on query.GetData which adds continuation for scripts not yet adapted to do continuation. However the active pywikibot developers (approx. 5 people?) are focused on making core better , including about 10 complex patches under review that improve its query continuation algorithms, and making core more compatible with 'compat' to ease the pain of switching to core.
If anyone submits patches for compat continuation bugs affecting them, they will be reviewed (usually by people familiar with compat) with the presumption that the patch author has tested it, and merged if there are no major problems.
-- John Vandenberg
Maybe someone with enough time and knowledge can fork compat and keep it alive...
Date: Tue, 9 Jun 2015 13:03:03 +1000 From: jayvdb@gmail.com To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko intracer@gmail.com wrote:
It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate at once.
Now it looks as API: we are deprecating what we promised to deprecated long ago - ok pywikipedia compat: did not handle the deprecation of API before, and are not going to fix copy-pasted in tens of places (not one place, it's never that simple) query builders to support "rawcontinue", we announce compat as discontinued together with the old style API. API deprecation was not coordinated with client library deprecation - not ok
If there is one year gap between two deprecations - ok, bot writers can choose either compat or core, and their bots can still work. Most users don't use APi directly so it should be the problem of coordination between API and clients developers.
On the pywikipedia side, it has been unofficially deprecated for a while, and the intention was to decommission compat as gracefully as possible, with Wikimania as the final killing ground.
The pywikibot developers roughly hashed out a decommissioning plan at the Lyon hackathon, and worked with WMF staff to start doing impact analysis. https://phabricator.wikimedia.org/T101214
Then the MW API continuation breakage was announced post Lyon. Hmm.
The continuation problem in pywikipedia / compat is not so much those 50 or so occurrences where continuation bugs appear to exist, but
- testing those 50 or so occurrences
- finding the other less obvious occurrences
- scripts people have written using compat's query.GetData may also
be broken, as it doesnt do continuation.
With a lot of wasted effort, 1 & 2 might be resolved so that 'compat' code works, and someone could create a hack on query.GetData which adds continuation for scripts not yet adapted to do continuation. However the active pywikibot developers (approx. 5 people?) are focused on making core better , including about 10 complex patches under review that improve its query continuation algorithms, and making core more compatible with 'compat' to ease the pain of switching to core.
If anyone submits patches for compat continuation bugs affecting them, they will be reviewed (usually by people familiar with compat) with the presumption that the patch author has tested it, and merged if there are no major problems.
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, 2015-06-09 at 11:32 +0200, Steinsplitter Wiki wrote:
Maybe someone with enough time and knowledge can fork compat and keep it alive...
As far as I understand it, there is no need to fork anything (no disagreement with existing maintainers; no contribution barriers).
Anybody interested could have kept compat (more) alive over the last years. Or could in the future.
andre
Il 09/06/2015 11:32, Steinsplitter Wiki ha scritto:
Maybe someone with enough time and knowledge can fork compat and keep it alive...
Everyone is free to fork it, of course, but in this case it'd only bring more fragmentation to the ecosystem. As John said, patches are always welcome.
On Wed, Jun 3, 2015 at 1:13 PM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the defaults. What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why https://developer.github.com/v3/ people https://www.reddit.com/dev/api version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
That's my point, why *ever* make it a breaking change? It's such a low cost to add a few characters to the URL to specify an API version. Our docs should tell developers what the current version is.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-announce
Maybe we should have a reply-to for api-announce mailings that prevent accidental chatter?
On Thu, Jun 4, 2015 at 12:45 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
On Wed, Jun 3, 2015 at 1:13 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the defaults. What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why https://developer.github.com/v3/ people https://www.reddit.com/dev/api version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
That's my point, why *ever* make it a breaking change? It's such a low cost to add a few characters to the URL to specify an API version. Our docs should tell developers what the current version is.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
On 06/04/2015 09:45 AM, Brian Gerstle wrote:
While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the defaults. What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why https://developer.github.com/v3/ people https://www.reddit.com/dev/api version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
I'd recommend reading https://phabricator.wikimedia.org/T41592, which contains a pretty good rationale of why we currently don't version the API.
-- Legoktm
I know about the phab ticket, but I'm glad you referenced it because I found this:
(Anomie): I think feature flags to *select new behavior* and *a good
deprecation process* will take care of most things that actually need improvement, to the point where we can do per-module versioning on an ad hoc basis rather than trying to force it everywhere.
IOW, why don't we continue w/ this feature flagging approach, which seems like a decent way to version APIs and prevent breaking backwards compatibility?
On Thu, Jun 4, 2015 at 1:05 PM, Legoktm legoktm.wikipedia@gmail.com wrote:
On 06/04/2015 09:45 AM, Brian Gerstle wrote:
While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the
defaults.
What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why https://developer.github.com/v3/ people <
https://www.reddit.com/dev/api%3E
version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
I'd recommend reading https://phabricator.wikimedia.org/T41592, which contains a pretty good rationale of why we currently don't version the API.
-- Legoktm
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jun 4, 2015 at 1:59 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
IOW, why don't we continue w/ this feature flagging approach, which seems like a decent way to version APIs and prevent breaking backwards compatibility?
That's basically what this is, except it's named "rawcontinue" and applies as a boolean to just the continuation format rather than something like "queryversion".
But we'd still have the issue where changing the default version (or making the version parameter non-optional) is necessary here, otherwise the default version would be stuck at "1".
Just found the deprecation process document https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap, specifically:
- If not already present, a request parameter will be added to
specifically request the old behavior. 2. The change will be announced: - A message will be sent to the mediawiki-api-announce http://lists.wikimedia.org/pipermail/mediawiki-api-announce/ mailing list. - Deprecation warnings will be output when neither the select-new-version nor the select-old-version flags are used. Logs will also be made. 3. *After a suitable timeframe, the new version will become the default.* 4. *Any flag to select the new version explicitly may at some point be removed, leading to "unrecognized parameter" warnings.*
My argument is that step #3 is unnecessary and #4 needs clarification in that IMO APIs should only be removed when they are no longer supported, otherwise you're just creating busy work for yourself and the clients.
On Thu, Jun 4, 2015 at 1:59 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
I know about the phab ticket, but I'm glad you referenced it because I found this:
(Anomie): I think feature flags to *select new behavior* and *a good
deprecation process* will take care of most things that actually need improvement, to the point where we can do per-module versioning on an ad hoc basis rather than trying to force it everywhere.
IOW, why don't we continue w/ this feature flagging approach, which seems like a decent way to version APIs and prevent breaking backwards compatibility?
On Thu, Jun 4, 2015 at 1:05 PM, Legoktm legoktm.wikipedia@gmail.com wrote:
On 06/04/2015 09:45 AM, Brian Gerstle wrote:
While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the
defaults.
What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why https://developer.github.com/v3/ people <
https://www.reddit.com/dev/api%3E
version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
I'd recommend reading https://phabricator.wikimedia.org/T41592, which contains a pretty good rationale of why we currently don't version the API.
-- Legoktm
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Sorry, guess I'm also calling #1 into question, since the old behavior should have already been the default or selectable by its own flag/parameter/URL-path-component/content-type/whatever.
On Thu, Jun 4, 2015 at 2:08 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
Just found the deprecation process document https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap, specifically:
- If not already present, a request parameter will be added to
specifically request the old behavior. 2. The change will be announced: - A message will be sent to the mediawiki-api-announce http://lists.wikimedia.org/pipermail/mediawiki-api-announce/ mailing list. - Deprecation warnings will be output when neither the select-new-version nor the select-old-version flags are used. Logs will also be made. 3. *After a suitable timeframe, the new version will become the default.* 4. *Any flag to select the new version explicitly may at some point be removed, leading to "unrecognized parameter" warnings.*
My argument is that step #3 is unnecessary and #4 needs clarification in that IMO APIs should only be removed when they are no longer supported, otherwise you're just creating busy work for yourself and the clients.
On Thu, Jun 4, 2015 at 1:59 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
I know about the phab ticket, but I'm glad you referenced it because I found this:
(Anomie): I think feature flags to *select new behavior* and *a good
deprecation process* will take care of most things that actually need improvement, to the point where we can do per-module versioning on an ad hoc basis rather than trying to force it everywhere.
IOW, why don't we continue w/ this feature flagging approach, which seems like a decent way to version APIs and prevent breaking backwards compatibility?
On Thu, Jun 4, 2015 at 1:05 PM, Legoktm legoktm.wikipedia@gmail.com wrote:
On 06/04/2015 09:45 AM, Brian Gerstle wrote:
While it is (a little bit) nicer for new developers, they'll just
burned
(along with all the other current API users) when you change the
defaults.
What I'm trying to say is, changing the default seems like more work
for
more people with very little benefit. This is why https://developer.github.com/v3/ people <
https://www.reddit.com/dev/api%3E
version https://stripe.com/docs/api#charge_object APIs https://developer.linkedin.com/docs/rest-api.
I'd recommend reading https://phabricator.wikimedia.org/T41592, which contains a pretty good rationale of why we currently don't version the API.
-- Legoktm
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live.
-- Steinsplitter
Date: Wed, 3 Jun 2015 14:50:48 +0300 From: yastrakhan@wikimedia.org To: wikitech-l@lists.wikimedia.org CC: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will keep the logic as before.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like pywikibot rely on, and that is designed in part by the core developers. See the proposal at https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie... On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
MZ: >I feel that former MediaWiki Web API maintainers should actively pay attention to which mailing lists they're posting to. ;-) I doubt you intended to send this message to mediawiki-api-announce. Mz, I don't think I ever spent much time maintaining it :)) But yes, good point, reply all is evil at times :)
MZ: re why minimalistic lib - for most apis out there, people tend to write "reference implementation" - code that explains to would-be lib authors how to use API. It sets up prefered usage patterns and guidelines. We never had that for the api. This resulted in a large number of API libs that vary in how they use api. Pywikibot is a powerful platform, but is too big for many usecases (as discussed in Lyon). Hence the need.
SW: >Most operator are volunteers and don't have time to change the code every month because there is a change in the api. * I might agree about every month, but we are talking about a feature that has been out for 2 years...
The old one was imho perfectly.
* Most API users imho would laugh at this statement. See the roadmap https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap which came as a result of analysing numerous bugs & wishes filed against the api, such as returning {} instead of [] for empty values, inconsistencies, the silly '*' value for text, and many other problems that have accumulated during the years. API might work for you, but it does not support multilingual error messaging, it overcomplicates the process of writing javascript clients, it does not easily cache. In short, lots of problems.
Was the new api coded by WMF or by volunteers?
* I wrote the original API as a volunteer (took about a year in 2005/6). I recently coded the continuation as a volunteer, and Brad has improved it as a WMF employee. I later became a full time WMF employee as well, but my project was Zero, not API. As a volunteer, over 2 years ago I wrote a huge API improvement roadmap https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmapthat Brad picked up and greatly extended, and is driving it forward with the amazing improvements. From what I know, Brad has now been officially moved into position of working on the API. In other words, that employee vs volunteer line is very fuzzy. No point of splitting it that way.
TLDR;
In short, we need to make improvements if we want to move forward in turning API into a platform, with flexible JavaScript-driven interfaces such as Visual Editor. To allow creative uses that go beyond AWB, that support complex interactions, and not just the way for bots to make edits. Unfortunately, that means that despite our best efforts, very infrequently, some bots must be updated.
If the bot author cannot add a simple one line change "rawcontinue=1" once in two years because of their busy personal live, I don't think that person should be making automatic edits to Wikipedia - because sometimes bots make mistakes that require substantially more time involvement. I would not trust wikipedia edits to a bot runner under such circumstances. If the bot runner is not a programmer, they should get the latest version of their bot. If there is no up-to-date code because noone is maintaining it, it again should not be accessing wikipedia - we sometimes discover security bugs that require fixing, or because bot calls wiki too often, or other changes in content structure - e.g. introduction of WikiData for interwiki links required all interwiki bots to be updated.
Wikipedia is a living, developing ecosystem, and it does require updates to all parties accessing it. People accessing wikipedia from the older browsers discover that they no longer can do everything they used to many years ago - because we now use newer browser features, and fallback into basic mode for older browsers. Please participate in the evolution of the project.
On Wed, Jun 3, 2015 at 5:22 PM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live.
-- Steinsplitter
Date: Wed, 3 Jun 2015 14:50:48 +0300 From: yastrakhan@wikimedia.org To: wikitech-l@lists.wikimedia.org CC: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode
for action=query will change at the end of this month
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists. So, the bot operator who never updates their software, doesn't pay attention to the announcements, and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just for the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will
keep
the logic as before.
Lastly, I again would like to promote the idea discussed at the hackathon -- a client side minimalistic library that bigger frameworks like
pywikibot
rely on, and that is designed in part by the core developers. See the proposal at
https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie...
On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning
more
than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who
is,
please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 6/3/15, Steinsplitter Wiki steinsplitter-wiki@live.com wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live.
-- Steinsplitter
My understanding is that most of the people who were using the original continuation, were using it wrong, causing subtle bugs in their script. Thus the existing implementation was wasting considerable amount of volunteer bot developer time. In the long run this change will hopefully reduce the total amount of time spent by volunteer bot makers chasing weird bugs, at the expense of some short term pain.
Its always a challenge to balance backwards compatibility with fixing things that are causing problems. I think the API team is keenly aware of the frustrations that changes to the api cause, and try to make sure that intentional breakage only happens when the benefits truly outweigh the cons.
-- Bawolff
On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
Should someone contact those bots on their talk page to notify the owners or do we hope everyone reads this mailing list?
Also, this change will affect not only bots but also every piece of software or tool that was written or published prior to the change and relies on the API to fetch data. So there are two issues with that: * The maintainer of the tool has no time/interest to compile and publish new releases of versions which were considered stable. This means the tool will die if the source code is not available or no one wants to take over. * Previously published programs that were considered stable and are executed after July 1st. The user may not be aware that this tool is no longer stable and requires an update. As far as I understood this API change: The results from the API will just be somewhat different and not produce an error. So the software will not crash but may produce weird behavior instead. Is there any solution to this?
Thanks, Marco
On Wed, Jun 17, 2015 at 9:15 PM, Marco maic23@live.de wrote:
On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning more than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who is, please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
Should someone contact those bots on their talk page to notify the owners or do we hope everyone reads this mailing list?
I see Whatamidoing (WMF) has already done this for many bots.
I saw one of the problematic bots says they are using AutoWikiBrowser. maybe the bot is using an old version of AWB?
Also, this change will affect not only bots but also every piece of software or tool that was written or published prior to the change and relies on the API to fetch data. So there are two issues with that:
- The maintainer of the tool has no time/interest to compile and publish new
releases of versions which were considered stable. This means the tool will die if the source code is not available or no one wants to take over.
- Previously published programs that were considered stable and are executed
after July 1st. The user may not be aware that this tool is no longer stable and requires an update. As far as I understood this API change: The results from the API will just be somewhat different and not produce an error. So the software will not crash but may produce weird behavior instead. Is there any solution to this?
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?
On Wed, Jun 17, 2015 at 10:13 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?
I saw at least one major bot had all of its actual continuation uses fixed after the first notification, but did not bother to fix cases where it was receiving the warning but was never going to continue the query. Others may well have done the same.
For example, if a client wants to get the current content for enwiki's Main Page, it might hit https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvpro.... That returns continuation (and the warning) in case the client would want further revisions, but the client here doesn't. Similar cases could be made for fetching the most recent protection log entry of a page, the imageinfo for the current version of an image, and so on.
The current plan, as shown in https://gerrit.wikimedia.org/r/#/c/160223/, is to output a different warning for some period of time after the changeover. People investigating the sudden failure of their bot/script/gadget would hopefully see that.
There is no such a thing like "properly break". That's like if someone was happily sad. Come on
On Thu, Jun 18, 2015 at 4:09 PM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Wed, Jun 17, 2015 at 10:13 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?
I saw at least one major bot had all of its actual continuation uses fixed after the first notification, but did not bother to fix cases where it was receiving the warning but was never going to continue the query. Others may well have done the same.
For example, if a client wants to get the current content for enwiki's Main Page, it might hit https://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvpro.... That returns continuation (and the warning) in case the client would want further revisions, but the client here doesn't. Similar cases could be made for fetching the most recent protection log entry of a page, the imageinfo for the current version of an image, and so on.
The current plan, as shown in https://gerrit.wikimedia.org/r/#/c/160223/, is to output a different warning for some period of time after the changeover. People investigating the sudden failure of their bot/script/gadget would hopefully see that.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jun 18, 2015 at 12:13 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?
Not 'all', please, but perhaps all queries where there is actual continuation data like is used now for the existing warning. I would really to avoid the API failing at all, ever, for "?action=query&meta=siteinfo" - that is how clients get the API generator version, and only when we have the API generator version do we know what features are supported by the API.
Or just not flick the switch on July 1, and only default to the new continuation mode for formatversion=2.
I'm reluctant to interject again, but as a client of the API I feel the need to speak up. Let's work through an example: Clients A, B, & C are using API 1 when the Server adds API 2:
1. Current, "change the default" approach: 1. All clients start receiving warnings as *extra payload data* (how does this even work for API's w/o a response payload?) 2. Clients B & C start using API 2 3. Server changes API 1 to behave like API 2 4. Client A is broken, so switches to API 2 5. Server removes API 2 (becomes the new default) 6. Clients A, B, & C have to switch to API 1 2. Possible alternative: 1. Clients get an email (as they currently do) about the new API being available 2. Clients A & B migrate to API 2 3. Since there's only 1 client left (and/or the deadline has passed) Server removes API 1, *explicitly breaking* Client C 4. Someone notices Client C is broken and moves it to API 2 (or whatever the currently-supported API is)
Am I missing something? What are your thoughts on 1 vs. 2?
On Thu, Jun 18, 2015 at 10:17 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
On Thu, Jun 18, 2015 at 12:13 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jayvdb@gmail.com wrote:
The API currently emits a warning if a query continuation mode isnt selected.
I guess on July 1 the API could emit an error, and not return any query data. Then the data isnt going to cause weird behaviour - it will break, properly.
Hmm, this is actually an interesting idea - would it make sense to error
on
missing "continue" or "rawcontinue" for all action=query for about a
month
or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?
Not 'all', please, but perhaps all queries where there is actual continuation data like is used now for the existing warning. I would really to avoid the API failing at all, ever, for "?action=query&meta=siteinfo" - that is how clients get the API generator version, and only when we have the API generator version do we know what features are supported by the API.
Or just not flick the switch on July 1, and only default to the new continuation mode for formatversion=2.
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
- Current, "change the default" approach:
does this even work for API's w/o a response payload?)
- All clients start receiving warnings as *extra payload data* (how
What module in api.php doesn't have a response payload? action=opensearch with format=json is the only example I can think of.
5. Server removes API 2 (becomes the new default) 6. Clients A, B, & C have to switch to API 1
You're making false assumptions here. For example, with this continuation change, explicitly supplying '&continue=' in the initial query is not going to suddenly stop working. Nor is '&rawcontinue=', for that matter.
The same goes for a hypothetical future change to make '&formatversion=2' the default for format=json. We wouldn't suddenly make clients have to supply '&formatversion=1' to get the formatversion=2 behavior, or make supplying a formatversion be an error or warning.
Brad Jorsch (Anomie) wrote:
On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
- Current, "change the default" approach:
- All clients start receiving warnings as *extra payload data* (how does this even work for API's w/o a response payload?)
What module in api.php doesn't have a response payload? action=opensearch with format=json is the only example I can think of.
I would think any request to api.php using format=none would lack a response payload? However I'm not totally sure that I'm reading and using the term response payload in the same way that you two might be.
MZMcBride
On Thu, Jun 18, 2015 at 11:33 AM, MZMcBride z@mzmcbride.com wrote:
Brad Jorsch (Anomie) wrote:
On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
- Current, "change the default" approach:
- All clients start receiving warnings as *extra payload data* (how does this even work for API's w/o a response payload?)
What module in api.php doesn't have a response payload? action=opensearch with format=json is the only example I can think of.
I would think any request to api.php using format=none would lack a response payload? However I'm not totally sure that I'm reading and using the term response payload in the same way that you two might be.
Haha, true. But if someone is using format=none, they're explicitly not caring to know if their request worked or not. ;)
I guess it comes down to is this: if we're going to continue supporting old behavior, they should be accessible via the same old requests. *This removes the need for existing clients to be updated in the first place*. If we eventually want to delete the old code keeping the old behavior separated from the new will make it clear & explicit what's being dropped and what to use instead. For example, dropping formatversion=1 means clients need to use formatversion=2 (or whatever other supported versions).
Lastly, changing the default behavior to make things sane for new developers is, IMO, a bad trade-off because they'll eventually get tripped by us pulling the rug out from under their feet by *breaking backwards compatibility stable APIs*. Those sorts of changes should be reserved for experimental or even beta APIs. Continuing queries seems like a stable—and pervasive—part of the API.
On Thu, Jun 18, 2015 at 11:40 AM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Thu, Jun 18, 2015 at 11:33 AM, MZMcBride z@mzmcbride.com wrote:
Brad Jorsch (Anomie) wrote:
On Thu, Jun 18, 2015 at 10:37 AM, Brian Gerstle <bgerstle@wikimedia.org
wrote:
- Current, "change the default" approach:
- All clients start receiving warnings as *extra payload data* (how does this even work for API's w/o a response payload?)
What module in api.php doesn't have a response payload?
action=opensearch
with format=json is the only example I can think of.
I would think any request to api.php using format=none would lack a response payload? However I'm not totally sure that I'm reading and using the term response payload in the same way that you two might be.
Haha, true. But if someone is using format=none, they're explicitly not caring to know if their request worked or not. ;)
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
I guess it comes down to is this: if we're going to continue supporting old behavior, they should be accessible via the same old requests. *This removes the need for existing clients to be updated in the first place*. If we eventually want to delete the old code keeping the old behavior separated from the new will make it clear & explicit what's being dropped and what to use instead. ...
Lastly, changing the default behavior to make things sane for new developers is, IMO, a bad trade-off
That seems the crux of it. Because the MediaWiki API isn't versioned (and there are good reasons for it), the recommended "best practices" form of an API request evolves, something like
api.php?continue=& formatversion=2 &utf8= &*your_actual_params*
and over time the best practices boilerplate at the front gets longer unless we change a default and break old clients. Examples and documentation should show best practices; T103015 is to use formatversion=2 and "https" in all example API requests. (We should have used &continue= in examples for the last year, it's too late now.)
The above is actually a real URL and shows three different approaches: 1. As the e-mail subject says we're going to make continue= the default in a few weeks so you won't need to add it (but clients MUST add rawcontinue= to get the old behavior). 2. formatversion=2 is the future but won't be the default for a while. 3. If you request formatversion=2 then results default to utf8, so you don't need to specify utf8. (Note formatversion=2 only applies to format=json or php.)
Which approach to take is a judgement call, I'm interject-opinion-reluctant :)
because they'll eventually get tripped by us pulling the rug out from under their feet by *breaking backwards compatibility stable APIs*.
Or, over time the best practices boilerplate endlessly expands: responselayout=clean &reporterrors=schema &facebookoverlordmode= ... Does that make our API "user-hostile"? IMO it just makes it wordy.
Those sorts of changes should be reserved for experimental or even beta APIs. Continuing queries seems like a stable—and pervasive—part of the API.
Cheers,
Before playing around with api serious bugs should be fixed... I see xxXx unresolved bugs on pahricator. It is frustrating....
Date: Thu, 18 Jun 2015 14:56:25 -0700 From: spage@wikimedia.org To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month
On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
I guess it comes down to is this: if we're going to continue supporting old behavior, they should be accessible via the same old requests. *This removes the need for existing clients to be updated in the first place*. If we eventually want to delete the old code keeping the old behavior separated from the new will make it clear & explicit what's being dropped and what to use instead. ...
Lastly, changing the default behavior to make things sane for new developers is, IMO, a bad trade-off
That seems the crux of it. Because the MediaWiki API isn't versioned (and there are good reasons for it), the recommended "best practices" form of an API request evolves, something like
api.php?continue=& formatversion=2 &utf8= &*your_actual_params*
and over time the best practices boilerplate at the front gets longer unless we change a default and break old clients. Examples and documentation should show best practices; T103015 is to use formatversion=2 and "https" in all example API requests. (We should have used &continue= in examples for the last year, it's too late now.)
The above is actually a real URL and shows three different approaches:
- As the e-mail subject says we're going to make continue= the default in
a few weeks so you won't need to add it (but clients MUST add rawcontinue= to get the old behavior). 2. formatversion=2 is the future but won't be the default for a while. 3. If you request formatversion=2 then results default to utf8, so you don't need to specify utf8. (Note formatversion=2 only applies to format=json or php.)
Which approach to take is a judgement call, I'm interject-opinion-reluctant :)
because they'll eventually get tripped by us pulling the rug out from under their feet by *breaking backwards compatibility stable APIs*.
Or, over time the best practices boilerplate endlessly expands: responselayout=clean &reporterrors=schema &facebookoverlordmode= ... Does that make our API "user-hostile"? IMO it just makes it wordy.
Those sorts of changes should be reserved for experimental or even beta APIs. Continuing queries seems like a stable—and pervasive—part of the API.
Cheers,
=S Page WMF Tech writer _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Fri, Jun 19, 2015 at 7:56 AM, S Page spage@wikimedia.org wrote:
On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
I guess it comes down to is this: if we're going to continue supporting old behavior, they should be accessible via the same old requests. *This removes the need for existing clients to be updated in the first place*. If we eventually want to delete the old code keeping the old behavior separated from the new will make it clear & explicit what's being dropped and what to use instead. ...
Lastly, changing the default behavior to make things sane for new developers is, IMO, a bad trade-off
That seems the crux of it. Because the MediaWiki API isn't versioned (and there are good reasons for it), the recommended "best practices" form of an API request evolves, something like
api.php?continue=& formatversion=2 &utf8= &*your_actual_params*
Why doesnt formatversion=2 assume continue= and utf8= ?
-- John Vandenberg
On Fri, Jun 19, 2015 at 5:42 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
Why doesnt formatversion=2 assume continue= and utf8= ?
It does assume utf8= (there's an ascii= to request the old mode in combination with formatversion=2).
It doesn't assume continue= because that belongs to ApiQuery, not ApiFormatJson or ApiFormatPhp.
wikitech-l@lists.wikimedia.org