pywikibot July 2019

pywikibot@lists.wikimedia.org

7 participants
5 discussions

Pywikibot at Wikimania Hackathon
by André Costa 29 Jul '19

29 Jul '19

Hi everyone, Wikimedia Sverige has been running the FindingGLAMs project <https://meta.wikimedia.org/wiki/FindingGLAMs> over the last year which has identified two areas (amongst others) where technical development is needed to unblock GLAM contributions. The first is support for Structured Data on Commons in the major upload tools (of which Pywikibot is a big one), the second is support for uploading Lexicographical data to Wikidata. I’ve seen some Pywikibot activity and discussion on both of these issues but neither seems to have taken off. To try and get either, or both, of these kickstarted I think it would be useful to get some of the active Pywikibot developers together. I believe that even just a joint schematic understanding of how these new features should be implemented would be a good outcome as this would aid actual implementation (as well as subsequent code review). Since Wikimedia Sverige is also hosting Wikimania this year the attached Hackathon <https://wikimania.wikimedia.org/wiki/2019:Hackathon> provides an excellent opportunity for this. If you are attending and interested in joining up for this I’d love to hear from you. Wikimedia Sverige also has a travel budget which can be used for bringing a few selected people working on these areas to the Wikimania Hackathon (this is outside of the scholarship process). Since our budget is limited I’d kindly ask that if you would be interested in receiving some financial support for the travel, please contact me off list and answer the following questions (each answer should be no longer than 100 words): 1. What would you be interested working on? 2. What relevant experience do you have? 3. What is the expected travel cost you would need support with? Related tasks - https://phabricator.wikimedia.org/T223820 - SDC on Commons - https://phabricator.wikimedia.org/T189321 - Lexicographical data - https://phabricator.wikimedia.org/T186200 - Rethink Wikibase data model implementation (may or may not be a blocker to either of the above) Cheers, André Costa (also Lokal_Profil when not wearing my Wikimedia Sverige hat) André Costa | Chief Operating Officer, Wikimedia Sverige | Andre.Costa(a)wikimedia.se | +46 (0)733-964574 Stöd fri kunskap, bli medlem i Wikimedia Sverige. Läs mer på blimedlem.wikimedia.se

1 1

getInterwiki
by masti 22 Jul '19

22 Jul '19

It used to be a getInterwiki method before Wikidata era. What is the easiest way to get them instead of parsing sitelinks list of WD object? masti

2 1

New stable release "3.0.20190722"
by info＠gno.de 22 Jul '19

22 Jul '19

Hi folks, a new stable release "3.0.20190722" was deployed which is available at pypi (as side package without scripts) or from our repository with the tags "3.0.20190430" and "stable". There was a long time between this stable branch and the previous one. The main reason was the implementation of sitelink badges and the implementation of closed wiki access which needed some additional work to get all tests passing. Please note that the last release is always tagged with the "stable" tag. For production systems you should always use this branch instead of master branch because master is under perpetual development (see T217908 at phabricator). PAWS is also based on the stable tagged release. There are currently 96,697 lines of code. The following changes where comming with this change: Important changes which needs operator's attention: * Deprecation warning: support for Python 2 will be dropped in 2020 (T213287). Please update your Python. * deprecate test_family, use wikipedia_family: Site('test', 'wikipedia') instead (T228375, T228300). To have access to test wiki you should instantiate the site object by Site('test', 'wikipedia') instead of Site('test', 'test'). The old method can be used further but can be dropped. * remove the unimplemented "proxy" variable in config.py. You should remove it from user-config.py too. Other changes: * Increase the throttling delay if maxlag >> retry-after (T210606) * deprecate test_family: Site('test', 'test'), use wikipedia_family: Site('test', 'wikipedia') instead (T228375, T228300) * Add "user_agent_description" option in config.py * APISite.fromDBName works for all known dbnames (T225590, 225723, 226960) * Make Family.langs property more robust (T226934) * Remove strategy family * Handle closed_wikis as read-only (T74674) * TokenWallet: login automatically * Add closed_wikis to Family.langs property (T225413) * Redirect 'mo' site code to 'ro' and remove interwiki_replacement_overrides (T225417, T89451) * Add support for badges on Wikibase item sitelinks (T128202) * Remove login.showCaptchaWindow() method * New parameter supplied in suggest_help function for missing dependencies * Remove NonMWAPISite class * Introduce Claim.copy and prevent adding already saved claims (T220131) * Fix create_short_link method after MediaWiki changes (T223865) * Validate proofreadpage.IndexPage contents before saving it * Refactor Link and introduce BaseLink (T66457) * Count skipped pages in BaseBot class * 'actionthrottledtext' is a retryable wikibase error (T192912) * Clear tokens on logout(T222508) * botirc.IRCBot has been dropped * Avoid using outdated browseragents (T222959) * textlib: avoid infinite execution of regex (T222671) * Add CSRF token in sitelogout() api call (T222508) * Refactor WikibasePage.get and overriding methods and improve documentation * Improve title patterns of WikibasePage extensions * Add support for property creation (T160402) For older changes refer HISTORY.rst. Also some changes for scripts where made like: * A lot of commoncat messages got twn support (219094) * script_wui.py was dropped and moved to archive folder (T222759) * default sign was added to welcome.py (T223044) * Quit option was added to disambredir.py (T223048) * Use additional twn messages with checkimages.py (T220178) * redirect.py does no longer fail for RuntimeError retrieving missing redirects (T130911) * Make transferbot.py continue when targetpage cannot be edited by bots or page does not exist (T223816) * Make coordinate_import.py work on a set of Wikidata items (T220806) * Implement -overwrite option in create_categories.py (T220305) * Enable choosing protect level with check_protection_level in protect.py (T225448) * Move categories without leaving a redirect which is the new default behaviour if suppressredirect right is available (T150093) * Namespace identifier can be ommitted for -start option for checkimages.py (T217824) Thanks all contributors for this code changes and all reviewers for their importand code checking and submitting it to the repository. Also thanks for all bug reporters and other persons who gave their valueable comments on phab tasks. Best xqt

1 0

Caching strategy
by Huji Lee 15 Jul '19

15 Jul '19

Hi all, I am working on a bot that fetches a list of anonymous editors on fawiki, uses WHOIS to retrieve more info about that IP, and uses a number of online APIs to check if the IP is a proxy or not.[1] I would like to improve the code by implementing a CIDR cache, so that if I run whois on 8.4.4.8 and determine that its ASN range is 8.4.4.0/24 and then I encounter 8.4.4.9 in the next iteration of my for loop, I would quickly determine this IP also belongs to the same range and skip the WHOIS part for it. The search space would consist of IP ranges like "8.4.4.0 - 8.4.4.25" (these are the beginning and end IP addresses of the 8.4.4.0/24 range). Obviously, we can convert these IPs to Hex to make comparisons easier. Given an IP like 8.4.4.9, we need the object to efficiently determine if it already has an IP range that encompasses this given IP and if so, return the previously cached details for that IP pair. If not, we will store that in cache. The part that I am not fully clear about is the following: how can I avoid having to loop through every range in the cache? Is there a way to create a hash function that checks two inequality comparisons efficiently? Thanks! Huji [1] https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/findproxy…

4 4

Re: [pywikibot] Caching strategy
by Huji Lee 12 Jul '19

12 Jul '19

@Lucas: cidr-trie was exactly what I wanted; thank you! @Strainu: In our use case (fawiki), the number of distinct IPs that make an edit in each day is not too large (usually a few hundred). Therefore, the memory intensity of using a CIDR-trie is minimal. The code at [1] has been updated to include caching in the way that I had desired. Thanks again, Huji [1] https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/findproxy… ----------- I'm very curious if you can run at Wikipedia scale with such a trie in memory on a normal computer (e.g. with only tens of GiB of memory). Please let us know if you actually get this into production (or just submit the script for inclusion in the framework, it sounds really useful) Strainu Pe vineri, 12 iulie 2019, Lucas Werkmeister <mail at lucaswerkmeister.de <https://lists.wikimedia.org/mailman/listinfo/pywikibot>> a scris: >* You probably want to use a trie <https://en.wikipedia.org/wiki/Trie <https://en.wikipedia.org/wiki/Trie>> for *>* this – I found several available Python implementations, but I don’t know *>* what their advantages or disadvantages are, so I’ll just list them in *>* alphabetical order: *>>* - cidr-tree <https://github.com/Figglewatts/cidr-trie <https://github.com/Figglewatts/cidr-trie>> *>* - py-radix <https://github.com/Figglewatts/cidr-trie <https://github.com/Figglewatts/cidr-trie>> *>* - pysubnettree <https://github.com/zeek/pysubnettree <https://github.com/zeek/pysubnettree>> *>* - pytricia <https://github.com/jsommers/pytricia <https://github.com/jsommers/pytricia>> *>>* Cheers, *>* Lucas *>* On 12.07.19 04:43, Huji Lee wrote: *>

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot July 2019