Hi lovely pywikibot folks. I'd like to clean up our deprecated code since we have some that's six years old....
% grep -RIoh "since='[^']+'" ./* | sort | head -n 5 since='20150206' since='20150515' since='20150615' since='20150617' since='20150619'
% grep -RIoh "since='[^']+'" ./* | wc -l 130
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
Thanks! -Damian
Hi Damian,
On 4/21/21 2:32 PM, Damian Johnson wrote:
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
I'm not aware of Pywikibot having such a policy, but I think it would be a good idea to have one. MediaWiki has a stable interface policy[1] which defines what parts are stable to build on top of and which are considered internal and then a process on how to deprecate and make changes to what's supposed to be stable.
One of the things I worked on for MediaWiki's deprecation process is developing codesearch[2] which makes it pretty straightforward for developers to see what methods/functions are practically being used and see what use cases are. I think something like that would be valuable for Pywikibot as well, but code for most bots/scripts is really all over the place. Something like Toolhub[3] would help with this too.
[1] https://www.mediawiki.org/wiki/Stable_interface_policy [2] https://codesearch.wmcloud.org/search/ [3] https://meta.wikimedia.org/wiki/Toolhub
HTH, -- Legoktm
Hi folks,
I'd like to bring this thread back to life since in the months passed *a lot* of deprecated code has been removed, some of which was quite "recent". I totally understand that projects move forward, and so does Python, I totally understand the developer's whish to use new language features and I don't believe we should keep deprecated code for decades, as it often happens in wikis. However, given that the vast majority of Pywikibot users are busy volunteers and that many projects depend on (mostly unsupervised) pywikibot code for critical maintenance work, I believe we need some kind of pedictibility on deprecation.
As a user, I would like to understand: 1. when and how can a function become deprecated 2. when and how can a parameter become deprecated 3. how long will I still be able to use a deprecated function or parameter 4. what Python versions will Pywikibot support.
For 3, I propose to maintain a compatibility for *at least* 3 years. This roughly matches the Debian lifecycle, as the longest-maintained non-LTS release of major Linux distributions.
For 4, I propose to support a Python version (e.g. 3.5) untill all of the following are true (that is, the longest period between them): - two years after official support has ended (e.g. 7 years after launch) AND - one year after Toolforge moved to a newer Python version. AND - the percentage of users of a version goes under 5%
This should allow both Toolforge and independent users ample time to update their code without surprises, even when using LTS releases.
What do you think? I would love to see feedback from both developers and users on these questions and possible answers. Even if you don't agree with these proposals, please make your own, so that we can hopefully agree on some rules - any deadlines would be better than none.
Regards, Strainu
În vin., 30 apr. 2021 la 00:33, Kunal Mehta legoktm@debian.org a scris:
Hi Damian,
On 4/21/21 2:32 PM, Damian Johnson wrote:
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
I'm not aware of Pywikibot having such a policy, but I think it would be a good idea to have one. MediaWiki has a stable interface policy[1] which defines what parts are stable to build on top of and which are considered internal and then a process on how to deprecate and make changes to what's supposed to be stable.
One of the things I worked on for MediaWiki's deprecation process is developing codesearch[2] which makes it pretty straightforward for developers to see what methods/functions are practically being used and see what use cases are. I think something like that would be valuable for Pywikibot as well, but code for most bots/scripts is really all over the place. Something like Toolhub[3] would help with this too.
[1] https://www.mediawiki.org/wiki/Stable_interface_policy [2] https://codesearch.wmcloud.org/search/ [3] https://meta.wikimedia.org/wiki/Toolhub
HTH, -- Legoktm
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Hi guys,
I'm with Strainu on this one. Let's take an example: Someone decided that handleArg didn't look pretty enough and renamed it in January this year. So far so good, but 4 months later the redirect was removed [1]. Why was this redirect removed? Don't you realize dozens of robots use Pywikibot as a library? For me alone this broke over 40 jobs.
I appreciate time and effort being spend on improving the framework, but here you derailed.
Maarten
[1] https://gerrit.wikimedia.org/r/c/pywikibot/core/+/689513/3/pywikibot/pagegen...
On 13-11-2021 13:12, Strainu wrote:
Hi folks,
I'd like to bring this thread back to life since in the months passed *a lot* of deprecated code has been removed, some of which was quite "recent". I totally understand that projects move forward, and so does Python, I totally understand the developer's whish to use new language features and I don't believe we should keep deprecated code for decades, as it often happens in wikis. However, given that the vast majority of Pywikibot users are busy volunteers and that many projects depend on (mostly unsupervised) pywikibot code for critical maintenance work, I believe we need some kind of pedictibility on deprecation.
As a user, I would like to understand:
- when and how can a function become deprecated
- when and how can a parameter become deprecated
- how long will I still be able to use a deprecated function or parameter
- what Python versions will Pywikibot support.
For 3, I propose to maintain a compatibility for *at least* 3 years. This roughly matches the Debian lifecycle, as the longest-maintained non-LTS release of major Linux distributions.
For 4, I propose to support a Python version (e.g. 3.5) untill all of the following are true (that is, the longest period between them):
- two years after official support has ended (e.g. 7 years after launch) AND
- one year after Toolforge moved to a newer Python version. AND
- the percentage of users of a version goes under 5%
This should allow both Toolforge and independent users ample time to update their code without surprises, even when using LTS releases.
What do you think? I would love to see feedback from both developers and users on these questions and possible answers. Even if you don't agree with these proposals, please make your own, so that we can hopefully agree on some rules - any deadlines would be better than none.
Regards, Strainu
În vin., 30 apr. 2021 la 00:33, Kunal Mehta legoktm@debian.org a scris:
Hi Damian,
On 4/21/21 2:32 PM, Damian Johnson wrote:
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
I'm not aware of Pywikibot having such a policy, but I think it would be a good idea to have one. MediaWiki has a stable interface policy[1] which defines what parts are stable to build on top of and which are considered internal and then a process on how to deprecate and make changes to what's supposed to be stable.
One of the things I worked on for MediaWiki's deprecation process is developing codesearch[2] which makes it pretty straightforward for developers to see what methods/functions are practically being used and see what use cases are. I think something like that would be valuable for Pywikibot as well, but code for most bots/scripts is really all over the place. Something like Toolhub[3] would help with this too.
[1] https://www.mediawiki.org/wiki/Stable_interface_policy [2] https://codesearch.wmcloud.org/search/ [3] https://meta.wikimedia.org/wiki/Toolhub
HTH, -- Legoktm
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list -- pywikibot@lists.wikimedia.org To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Hi Strainu,
Thanks for bringing this up. And sorry to hear that many have had negative experiences with recent removals of deprecated code (though I had nothing to do with it).
The Pywikibot project is barely maintained; a small community of interested folks are pushing it forward, and most code gets merged without review. I don't think we can claim, in good conscience, that there are folks who still keep a full architectural view of all elements of this project (as is the case with MediaWiki and many other tools we use daily).
The point is: if the framework is too difficult to execute, it won't happen. So, although I like your proposed framework, I find it unrealistic to happen. It would assume someone is pulling data on Python version usage, and they would be involved in code review in such a way to ensure that code incompatible with legacy Python versions is not introduced too soon.
I would encourage us to come up with a framework that is easier to maintain, and mostly automatable. Maybe we should just use the "7 year" rule, make sure that we have a CI pipeline that tries to build Pywikibot for all Python version that are currently within that 7 years, and at the top of the CI pipelines' code we have a comment that educates the users who may decide to edit it when the 7 year period ends for each existing version of Python. This way, incompatible changes would just fail on CI, and the enforcement of rules is quite simple, and breaking them by accident (such as by someone unknowingly modifying the CI pipeline) is less likely. This was just an idea, and may not be the right way, but I hope it shows what I mean by simplifying matters.
Similarly, maybe we should find a way to create a CI pipeline that would check for "@deprecated" methods, and cross-checks it with a reference JSON file in which each deprecated function is listed along with its deprecation date. If someone tries to remove the function before that date + 3 months, the CI would fail. If someone tries to add "@deprecated" without updating the JSON file, the CI would fail. This way, we have one single source of truth about when things were marked as deprecated (the JSON file) and an easy mechanism to track when deprecated methods are getting removed. Again, just a raw idea.
On Sat, Nov 13, 2021 at 7:13 AM Strainu strainu10@gmail.com wrote:
Hi folks,
I'd like to bring this thread back to life since in the months passed *a lot* of deprecated code has been removed, some of which was quite "recent". I totally understand that projects move forward, and so does Python, I totally understand the developer's whish to use new language features and I don't believe we should keep deprecated code for decades, as it often happens in wikis. However, given that the vast majority of Pywikibot users are busy volunteers and that many projects depend on (mostly unsupervised) pywikibot code for critical maintenance work, I believe we need some kind of pedictibility on deprecation.
As a user, I would like to understand:
- when and how can a function become deprecated
- when and how can a parameter become deprecated
- how long will I still be able to use a deprecated function or parameter
- what Python versions will Pywikibot support.
For 3, I propose to maintain a compatibility for *at least* 3 years. This roughly matches the Debian lifecycle, as the longest-maintained non-LTS release of major Linux distributions.
For 4, I propose to support a Python version (e.g. 3.5) untill all of the following are true (that is, the longest period between them):
- two years after official support has ended (e.g. 7 years after launch)
AND
- one year after Toolforge moved to a newer Python version. AND
- the percentage of users of a version goes under 5%
This should allow both Toolforge and independent users ample time to update their code without surprises, even when using LTS releases.
What do you think? I would love to see feedback from both developers and users on these questions and possible answers. Even if you don't agree with these proposals, please make your own, so that we can hopefully agree on some rules - any deadlines would be better than none.
Regards, Strainu
În vin., 30 apr. 2021 la 00:33, Kunal Mehta legoktm@debian.org a scris:
Hi Damian,
On 4/21/21 2:32 PM, Damian Johnson wrote:
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
I'm not aware of Pywikibot having such a policy, but I think it would be a good idea to have one. MediaWiki has a stable interface policy[1] which defines what parts are stable to build on top of and which are considered internal and then a process on how to deprecate and make changes to what's supposed to be stable.
One of the things I worked on for MediaWiki's deprecation process is developing codesearch[2] which makes it pretty straightforward for developers to see what methods/functions are practically being used and see what use cases are. I think something like that would be valuable for Pywikibot as well, but code for most bots/scripts is really all over the place. Something like Toolhub[3] would help with this too.
[1] https://www.mediawiki.org/wiki/Stable_interface_policy [2] https://codesearch.wmcloud.org/search/ [3] https://meta.wikimedia.org/wiki/Toolhub
HTH, -- Legoktm
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list -- pywikibot@lists.wikimedia.org To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Hi all. Just a brief note that this is where I documented our deprecation policy. Adjustments welcome.
https://www.mediawiki.org/wiki/Manual:Pywikibot/Development/Guidelines#Depre... https://phabricator.wikimedia.org/T281202
Hi Huji,
Thanks for your message, it's very thoughtful and raises many valid points. The three year limit for deprecation should be trivial to enforce using git hooks and I can certainly help put it in place.
Now for the Python support. I totally support setting up checks and tests, although I don't consider them critical. If incompatible code does pass through review (it has happened before during GSOC) I feel confident the community will catch that in a few days or weeks. If we can have automation, all the better.
Also, I don't have any bias against a single nimerical limit myself. The algorithm I proposed was based on the issues raised in the latest deprecation discussion ( https://phabricator.wikimedia.org/T286867 ). If all these can be handled in a single limit, then that is clearly preferable and I support it. Note that the limit might well be larger than 7 years in that case.
Regards, Strainu
Pe sâmbătă, 13 noiembrie 2021, Huji Lee huji.huji@gmail.com a scris:
Hi Strainu, Thanks for bringing this up. And sorry to hear that many have had
negative experiences with recent removals of deprecated code (though I had nothing to do with it).
The Pywikibot project is barely maintained; a small community of
interested folks are pushing it forward, and most code gets merged without review. I don't think we can claim, in good conscience, that there are folks who still keep a full architectural view of all elements of this project (as is the case with MediaWiki and many other tools we use daily).
The point is: if the framework is too difficult to execute, it won't
happen. So, although I like your proposed framework, I find it unrealistic to happen. It would assume someone is pulling data on Python version usage, and they would be involved in code review in such a way to ensure that code incompatible with legacy Python versions is not introduced too soon.
I would encourage us to come up with a framework that is easier to
maintain, and mostly automatable. Maybe we should just use the "7 year" rule, make sure that we have a CI pipeline that tries to build Pywikibot for all Python version that are currently within that 7 years, and at the top of the CI pipelines' code we have a comment that educates the users who may decide to edit it when the 7 year period ends for each existing version of Python. This way, incompatible changes would just fail on CI, and the enforcement of rules is quite simple, and breaking them by accident (such as by someone unknowingly modifying the CI pipeline) is less likely. This was just an idea, and may not be the right way, but I hope it shows what I mean by simplifying matters.
Similarly, maybe we should find a way to create a CI pipeline that would
check for "@deprecated" methods, and cross-checks it with a reference JSON file in which each deprecated function is listed along with its deprecation date. If someone tries to remove the function before that date + 3 months, the CI would fail. If someone tries to add "@deprecated" without updating the JSON file, the CI would fail. This way, we have one single source of truth about when things were marked as deprecated (the JSON file) and an easy mechanism to track when deprecated methods are getting removed. Again, just a raw idea.
On Sat, Nov 13, 2021 at 7:13 AM Strainu strainu10@gmail.com wrote:
Hi folks,
I'd like to bring this thread back to life since in the months passed *a lot* of deprecated code has been removed, some of which was quite "recent". I totally understand that projects move forward, and so does Python, I totally understand the developer's whish to use new language features and I don't believe we should keep deprecated code for decades, as it often happens in wikis. However, given that the vast majority of Pywikibot users are busy volunteers and that many projects depend on (mostly unsupervised) pywikibot code for critical maintenance work, I believe we need some kind of pedictibility on deprecation.
As a user, I would like to understand:
- when and how can a function become deprecated
- when and how can a parameter become deprecated
- how long will I still be able to use a deprecated function or
parameter
- what Python versions will Pywikibot support.
For 3, I propose to maintain a compatibility for *at least* 3 years. This roughly matches the Debian lifecycle, as the longest-maintained non-LTS release of major Linux distributions.
For 4, I propose to support a Python version (e.g. 3.5) untill all of the following are true (that is, the longest period between them):
- two years after official support has ended (e.g. 7 years after
launch) AND
- one year after Toolforge moved to a newer Python version. AND
- the percentage of users of a version goes under 5%
This should allow both Toolforge and independent users ample time to update their code without surprises, even when using LTS releases.
What do you think? I would love to see feedback from both developers and users on these questions and possible answers. Even if you don't agree with these proposals, please make your own, so that we can hopefully agree on some rules - any deadlines would be better than none.
Regards, Strainu
În vin., 30 apr. 2021 la 00:33, Kunal Mehta legoktm@debian.org a scris:
Hi Damian,
On 4/21/21 2:32 PM, Damian Johnson wrote:
What is pywikibot's policy regarding code deprecation? Can we remove it after a set duration and, if so, what is it?
I'm not aware of Pywikibot having such a policy, but I think it would
be
a good idea to have one. MediaWiki has a stable interface policy[1] which defines what parts are stable to build on top of and which are considered internal and then a process on how to deprecate and make changes to what's supposed to be stable.
One of the things I worked on for MediaWiki's deprecation process is developing codesearch[2] which makes it pretty straightforward for developers to see what methods/functions are practically being used and see what use cases are. I think something like that would be valuable for Pywikibot as well, but code for most bots/scripts is really all
over
the place. Something like Toolhub[3] would help with this too.
[1] https://www.mediawiki.org/wiki/Stable_interface_policy [2] https://codesearch.wmcloud.org/search/ [3] https://meta.wikimedia.org/wiki/Toolhub
HTH, -- Legoktm
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list -- pywikibot@lists.wikimedia.org To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Hi,
On 11/13/21 11:07 AM, Huji Lee wrote:
The Pywikibot project is barely maintained; a small community of interested folks are pushing it forward, and most code gets merged without review. <snip>
Thanks for pointing this out, my feeling is that this is really the underlying issue here. Keeping deprecated features around for a long time has a cost, and we might not have enough active developers to pay it.
I am very grateful to the people who are keeping Pywikibot going, my bots and other wiki workflows I use are still dependent upon it despite me not having time nor motivation to contribute back to the framework these days.
I am fine with empowering those people to make framework changes they want, and if that means I need to fix my bots more often, so be it, that's the price for me getting to use Pywikibot without contributing back.
In short, in my opinion the people who are doing the work can make the decisions.
<snip> It would assume someone is pulling data on Python version usage, and they would be involved in code review in such a way to ensure that code incompatible with legacy Python versions is not introduced too soon.
It is straightforward for anyone with NDA access to pull Python and Pywikibot version usage from Wikimedia wikis using Turnilo's webrequest data, I've done so in the past when asked[1]. In theory it would be possible for us to have an automated report for it, but I don't know enough about the analytics infrastructure to do that.
[1] https://phabricator.wikimedia.org/T286867#7220020
-- Legoktm