Hi Xqt,

Hi Maarten,

thank you for you feedback. There is the same "redundancy" in some Page methods which calls the corresponding site method and I wouldn’t drop one over the other. Page methods have a higher tier than Site methods which calls api functions and finally the comms functions.

Exactly. I think I'm only using the site object directly in some horrible hacks to work around the lack of structured data on Commons support in Pywikibot.

pagegenerators implements generators and filters for command line options and provides additional page generators which are outside of site methods like multiple site generators and other stuff.

But I don’t see any advantage using the same functionality from pagegenerators module over those site methods who are called by that function. The (now removed) functions had the form

  def this_generator(total=None, site=None)

     if not site: site = Pywikibot.Site()

     site.this(total=total)

There is a disadvantage in that sense that site must not be determined and another disadvantage that most site parameters were not provided with the pg function.

Can you tell me your POV about advantage/disadvantage of those pg function?

There are about 50 generator functions in site.GeneratorMixin but pg had only a few of them.

I made a proposal in https://gerrit.wikimedia.org/r/c/pywikibot/core/+/762132 which would implement all site generators as pagegenerators function. Is this in your sense? Unfortunately this only works with Python 3.7+ and needs some additional work for older Pythons.

Thanks for having a look at this. Interesting idea to create the generators like this. Would fix the example at https://www.mediawiki.org/wiki/Manual:Pywikibot/pagegenerators.py#Calls_from_another_script . Downside is that we won't have any documentation for these generators at https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.html#module-pywikibot.pagegenerators . Maybe middle ground to have a function that gives all the parameters to the site function using **kwargs ? Bit like we do in page at places like https://doc.wikimedia.org/pywikibot/master/_modules/pywikibot/page.html#BasePage.watch . Does make me wonder how to link that with https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html#pywikibot.site._apisite.APISite.watch so we don't have to copy all the parameters. Each generator would be a minimal function with a short description and a link to the underlying function for detailed instructions. Have we ever tried cross-referencing as described at https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects ?

Maarten

Best

xqt

Am 13.02.2022 um 14:40 schrieb Maarten Dammers <maarten@mdammers.nl>:



Hi Xqt,

I don't agree with the removal of the generators that someone decided to deprecate [1]. We always had the principle that pagegenerators is the point of entry to get pages to work on. The pagegenerators are a (thin) layer of abstraction so that you don't have to directly use function in the site objects. With this removal you're breaking one of the very fundamental functions and principles of our framework.

Why did you remove it? Any arguments beside than the incorrect argument that it's redundant?
Was the deprecation and removal discussed somewhere? Can't find it on this list.

Maarten

[1] https://gerrit.wikimedia.org/r/c/pywikibot/core/+/761395/5/pywikibot/pagegenerators.py

On 12-02-2022 13:31, info@gno.de wrote:

Hi all,

a new stable release Pywikibot 7 will be deployed at the end of this month.
There are a lot important points with this release, currently implemented in 
master branch:


Python Plattform:
-----------------

Pywkibot support of Python 3.5.0 - 3.5.2 will be dropped. The minimum plattform
required is Python 3.5.3 which is preinstalled at toolforge. On the other hand
Pywikibot also works with new Python 3.11.0a and *new* supports PyPy 3. Most
used plattform measured by PyPy download statistic is Python 3.8 (T266984). For
developers I highly propose to use Python 3.10 due to "Better error messages"
(https://docs.python.org/3.10/whatsnew/3.10.html#better-error-messages).


Framework Scripts:
------------------

The both scripts generate_family_file and generate_user_files were moved from 
framework root directory to pywikibot/scripts folder. The two scripts shell and
version were moved from scripts folder to pywikibot/scripts. This arrangement
was necessary to implement a code entry point for Pywikibot site-package. 
Possibly you have to change the path environment setting to use these scripts.
But all of them are still available through the pwb.py wrapper script.


Linux:
------

Scripts hash bang was changed from python to python3. This enables to invoke
scripts without preleading interpreter.


PAWS:
-----

Pywikibot 7 is a stable release and becomes the Pywikibot framework base on
PAWS.


Toolforge:
----------

After deployment of Pywikibot 7 this stable release will be available under
/shared/pywikibot/stable and the current master can be accessed under
/shared/pywikibot/core. The pwb.py wrapper script can be used at Tooforge
and the similar script search is functional.
https://doc.wikimedia.org/pywikibot/master/utilities/index.html#module-pwb


MediaWiki:
----------

No changes was made for supporting MediaWiki releases. Pywikibot 7 supports
MediaWiki 1.23-1.37. For current MediaWiki release 1.37 the minimum Pywikibot
version must be 6.6.1. MW 1.19 requires PWB 5 and MW 1.14 requires PWB 4.
(https://www.mediawiki.org/wiki/Manual:Pywikibot/Compatibility)


Pywikibot site-package (PyPi):
------------------------------

Installing Pywikibot as a site package (pip install pywikibot==7.0) comes with
many innovations. pywikibot i18n messages bundle is available. Framework 
scripts listed above can be used with Pywikibot 7.The pwb wrapper script is the
code entry point to invoke these scripts:
pwb [global options] <framework script> [script options|global options]



Pywikibot Tests:
----------------

Pywikibot CI tests were moved from Travis to GitHub action. There are two
actions implemented. The first runs up tp 20 workers in parallel and does the
most tests whereas the second only runs login/logout tests but there are no
parallel task which would lead to failures due to not logged-in test account.


Module Changes:
---------------

Support for API:Redirects was added in site and page module. User.is_locked()
and APISite.is_locked() were added to determine whether a given user or user id
is locked globally. The cached output functionality from compat release was
re-implemented.

families:
Allow family files reside in `families` folder in base_dir by default. This is 
important if Pywikibot is used as a site-package. generate_family_file uses
this new place to save the family file. Wikihow family file was added.

i18n module:
bundles() yields message bundle names or Path objects; known_languages() gives
all languages we have localizations for.

page module:
isStaticRedirect() also considers transcluded __STATICREDIRECT__.
put() and change_category() now has a show_diff parameter. User.is_blocked()
can also detect range blocks.

proofreadpage:
Wikimedia OCR engine is supported.

site module:
APISite.get_globaluserinfo() method was added to retrieve globaluserinfo.
New "maxlimit" property was added to determine the limits for API loads.

textlib module:
textlib comes with to_latin_digits() function as counterpart of
to_local_digits() and case_escape() which gives an escaped regex pattern
depending on 'first-letter' case.

tools module:
intersect_generators was rewritten which makes it running up to 10’000 times 
faster.


Scripts Changes:
----------------

Scripts descriptions can be viewed at
https://doc.wikimedia.org/pywikibot/master/scripts/index.html

ConfigParserBot is provided for several scripts. Any option can be set within
scripts.ini file.
https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.html#pywikibot.bot.ConfigParserBot

add_text scripts provided -create and -createonly options. A CleanBot was added
in category script which can be invoked by clean action option. It removes
redundant grandchildren categories. The fixing_redirect script uses 
concurrent.futures to retrieve redirect targets; this decreases processing time
by 90% if solving moved pages is enabled.


Deprecations:
-------------

Most of the deprecated code parts were removed to start with a new proper and
maintainable code without budensome overhead. The deprecations must be solved
if you have your own code based on Pywikibot before you upgrade to the new
release. Refer https://doc.wikimedia.org/pywikibot/master/changelog.html to see
all changes. Run your scripts and focus on FutureWarnings about deprecated code
parts. Refer the logs for deprecation warnings. Scripts must be run with global
-log option; if your script does not use handle_args() function you can invoke
it with the pwb wrapper script:
pwb.py -log <yourscript> [your options]



Thank you all, comments are welcome

xqt

_______________________________________________
pywikibot mailing list -- pywikibot@lists.wikimedia.org
To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org

_______________________________________________
pywikibot mailing list -- pywikibot@lists.wikimedia.org
To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org

_______________________________________________
pywikibot mailing list -- pywikibot@lists.wikimedia.org
To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org