It would be nice if MediaWiki API _AND_ pywikipedia bot do not deprecate at once.
Now it looks as API: we are deprecating what we promised to deprecated long ago - ok pywikipedia compat: did not handle the deprecation of API before, and are not going to fix copy-pasted in tens of places (not one place, it's never that simple) query builders to support "rawcontinue", we announce compat as discontinued together with the old style API. API deprecation was not coordinated with client library deprecation - not ok
If there is one year gap between two deprecations - ok, bot writers can choose either compat or core, and their bots can still work. Most users don't use APi directly so it should be the problem of coordination between API and clients developers.
Message: 1 Date: Wed, 3 Jun 2015 19:08:24 +0300 From: Yuri Astrakhan yastrakhan@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CALOOOkhzCUf14Tf= P0uQw7ZXAmwb9qAxs4kgvbJAvSsMQY82GA@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
MZ: >I feel that former MediaWiki Web API maintainers should actively pay attention to which mailing lists they're posting to. ;-) I doubt you intended to send this message to mediawiki-api-announce. Mz, I don't think I ever spent much time maintaining it :)) But yes, good point, reply all is evil at times :)
MZ: re why minimalistic lib - for most apis out there, people tend to write "reference implementation" - code that explains to would-be lib authors how to use API. It sets up prefered usage patterns and guidelines. We never had that for the api. This resulted in a large number of API libs that vary in how they use api. Pywikibot is a powerful platform, but is too big for many usecases (as discussed in Lyon). Hence the need.
SW: >Most operator are volunteers and don't have time to change the code every month because there is a change in the api.
- I might agree about every month, but we are talking about a feature that
has been out for 2 years...
The old one was imho perfectly.
- Most API users imho would laugh at this statement. See the roadmap
https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap which came as a result of analysing numerous bugs & wishes filed against the api, such as returning {} instead of [] for empty values, inconsistencies, the silly '*' value for text, and many other problems that have accumulated during the years. API might work for you, but it does not support multilingual error messaging, it overcomplicates the process of writing javascript clients, it does not easily cache. In short, lots of problems.
Was the new api coded by WMF or by volunteers?
- I wrote the original API as a volunteer (took about a year in 2005/6). I
recently coded the continuation as a volunteer, and Brad has improved it as a WMF employee. I later became a full time WMF employee as well, but my project was Zero, not API. As a volunteer, over 2 years ago I wrote a huge API improvement roadmap https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmapthat Brad picked up and greatly extended, and is driving it forward with the amazing improvements. From what I know, Brad has now been officially moved into position of working on the API. In other words, that employee vs volunteer line is very fuzzy. No point of splitting it that way.
TLDR;
In short, we need to make improvements if we want to move forward in turning API into a platform, with flexible JavaScript-driven interfaces such as Visual Editor. To allow creative uses that go beyond AWB, that support complex interactions, and not just the way for bots to make edits. Unfortunately, that means that despite our best efforts, very infrequently, some bots must be updated.
If the bot author cannot add a simple one line change "rawcontinue=1" once in two years because of their busy personal live, I don't think that person should be making automatic edits to Wikipedia - because sometimes bots make mistakes that require substantially more time involvement. I would not trust wikipedia edits to a bot runner under such circumstances. If the bot runner is not a programmer, they should get the latest version of their bot. If there is no up-to-date code because noone is maintaining it, it again should not be accessing wikipedia - we sometimes discover security bugs that require fixing, or because bot calls wiki too often, or other changes in content structure - e.g. introduction of WikiData for interwiki links required all interwiki bots to be updated.
Wikipedia is a living, developing ecosystem, and it does require updates to all parties accessing it. People accessing wikipedia from the older browsers discover that they no longer can do everything they used to many years ago - because we now use newer browser features, and fallback into basic mode for older browsers. Please participate in the evolution of the project.
On Wed, Jun 3, 2015 at 5:22 PM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho
perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the
technical
aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live.
-- Steinsplitter
Date: Wed, 3 Jun 2015 14:50:48 +0300 From: yastrakhan@wikimedia.org To: wikitech-l@lists.wikimedia.org CC: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation
mode
for action=query will change at the end of this month
I feel that bot operators should actively pay attention to the
technical
aspects of the community and the mailing lists. So, the bot operator
who
never updates their software, doesn't pay attention to the
announcements,
and ignores api warnings should be blocked after the deadline. Bot operators do not operate in a vacuum, and should never run bots just
for
the sake of running them. Community should always be able to find and communicate with the bot operators. Obviously we should not make sudden changes (except in the security/breaking matters), and try to make the process as easy as possible. The rawcontinue param is exactly that, simply adding it will
keep
the logic as before.
Lastly, I again would like to promote the idea discussed at the
hackathon
-- a client side minimalistic library that bigger frameworks like
pywikibot
rely on, and that is designed in part by the core developers. See the proposal at
https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Clie...
On Jun 3, 2015 2:29 PM, "John Mark Vandenberg" jayvdb@gmail.com
wrote:
On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
... I've compiled a list of bots that have hit the deprecation warning
more
than 10000 times over the course of the week May 23–29. If you are responsible for any of these bots, please fix them. If you know who
is,
please make sure they've seen this notification. Thanks.
Thank you Brad for doing impact analysis and providing a list of the 71 bots with more than 10,000 problems per week. We can try to solve those by working with the bot operators.
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the
scale
of bots operators that will be affected when this lands - currently
in
one months time.
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
-- John Vandenberg
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Message: 2 Date: Wed, 3 Jun 2015 12:34:12 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSuKej9O0beGt6xco+c+j7uTa0BtZoeCAQs9Cz6XQ2gYwg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 6:49 AM, Steinsplitter Wiki < steinsplitter-wiki@live.com> wrote:
I haven't followed this discussion, however i am wondering why api is not keep backward compatible. This will break a lot of stuff and i am
wondering
why we need such changes in the [API]
We usually do. In this case, however, the advantages of changing the default for new API users seems to outweigh the disadvantages of a well-announced breaking change with a simple parameter to request backwards-compatible output.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 3 Date: Wed, 3 Jun 2015 12:43:36 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Cc: mediawiki-api-announce@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSvprWamR3y+TFfcLg6Jv4hkbtCqbzj6JxfSvZe4v8f4Ow@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg jayvdb@gmail.com wrote:
If possible, could you compile a list of bots affected at a lower threshold - maybe 1,000. That will give us a better idea of the scale of bots operators that will be affected when this lands - currently in one months time.
I already have the list of *accounts* affected: there are 510 with between 1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out.
I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok).
Will the deploy date be moved back if the impact doesnt diminish by bots being fixed?
That's not impossible, but I wouldn't count on it.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 4 Date: Wed, 3 Jun 2015 13:13:34 -0400 From: "Brad Jorsch (Anomie)" bjorsch@wikimedia.org To: Wikimedia developers wikitech-l@lists.wikimedia.org Cc: MediaWiki API announcements & discussion mediawiki-api@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CAEepRSsfin=praEeth2qQQFtwS_SHgFDbZFpXEd= uqORvn4sJA@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Message: 5 Date: Wed, 3 Jun 2015 20:31:42 +0300 From: Yuri Astrakhan yastrakhan@wikimedia.org To: "MediaWiki API announcements & discussion" mediawiki-api@lists.wikimedia.org Cc: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] [Mediawiki-api] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CALOOOkjcnXoXCSMOQe5z58NR2iw24tf7qPcOkSXWT1qZLt30hQ@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
I would like to point out that it might be a good idea add &formatversion=1 for anyone who wants to lock the current formatting in place.
On Wed, Jun 3, 2015 at 8:13 PM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle bgerstle@wikimedia.org wrote:
My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without
making
any changes?
The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which
means
that the default shouldn't be something user-hostile.
The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and
then
when to re-add it, and when to remove the meta or list modules. I
wouldn't
be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators.
The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works.
Yes, some of the same could be said for making
format=json&formatversion=2
the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation
Mediawiki-api mailing list Mediawiki-api@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-api
Message: 6 Date: Wed, 3 Jun 2015 11:55:40 -0600 From: Brian Wolff bawolff@gmail.com To: Wikimedia developers wikitech-l@lists.wikimedia.org, mediawiki-api@lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CA+oo+DUnkpwL_EsG7DmGQOq5OsSPJmH9FnL+eW_xbaOO6XFjdg@mail.gmail.com> Content-Type: text/plain; charset=UTF-8
On 6/3/15, Steinsplitter Wiki steinsplitter-wiki@live.com wrote:
Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should
keep
the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho
perfectly.
Was the new api coded by WMF or by volunteers?
I feel that bot operators should actively pay attention to the technical aspects of the community and the mailing lists.
Sorry, i disagree. Bot operators are volunteers and not payed staffers.
Most
of them having a job and real live.
-- Steinsplitter
My understanding is that most of the people who were using the original continuation, were using it wrong, causing subtle bugs in their script. Thus the existing implementation was wasting considerable amount of volunteer bot developer time. In the long run this change will hopefully reduce the total amount of time spent by volunteer bot makers chasing weird bugs, at the expense of some short term pain.
Its always a challenge to balance backwards compatibility with fixing things that are causing problems. I think the API team is keenly aware of the frustrations that changes to the api cause, and try to make sure that intentional breakage only happens when the benefits truly outweigh the cons.
-- Bawolff
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
End of Wikitech-l Digest, Vol 143, Issue 8