pywikibot

pywikibot@lists.wikimedia.org

3 participants
6830 discussions

new contributor(s)
by Robert Morgan 17 Oct '19

17 Oct '19

Hi all, Per the development section on the wiki, I'm emailing this list to introduce myself. I'm part of a 3rd year undergraduate computer science program course on Open Source software. We have broken up into teams are 2-4 of us are looking into how we can help with pywikibot. We're starting with T184115<https://phabricator.wikimedia.org/T184115> (let us know if there's something more valuable some total n00bs can do). Have a nice day! Morgan11235

2 1

Prepare your small tasks to mentor a new contributor! (Google Code-in early heads-up)
by Andre Klapper 11 Sep '19

11 Sep '19

(Note: This is only an early heads-up, to be prepared. Google Code-in has NOT been announced yet, but last year, GCI mentors asked for more time in advance to identify tasks to mentor. Here you are. :) * You have small, self-contained bugs you'd like to see fixed? * Your documentation needs specific improvements? * Your user interface has some smaller design issues? * Your Outreachy/Summer of Code project welcomes small tweaks? * You'd enjoy helping someone port your template to Lua? * Your gadget code uses some deprecated API calls? * You have tasks in mind that welcome some research? Google Code-in (GCI) is an annual contest for 13-17 year old students. GCI 2019 has not yet been announced but usually takes place from late October to December. It is not only about coding: We also need tasks about design, docs, outreach/research, QA. Read https://www.mediawiki.org/wiki/Google_Code-in/Mentors , add your name to the mentors table, and start tagging tasks in Wikimedia Phabricator by adding the #gci-2019 project tag. We will need MANY mentors and MANY tasks, otherwise we cannot make it. Last year, 199 students successfully worked on 765 tasks supported by 39 mentors. For some achievements from the last round, see https://wikimediafoundation.org/news/2019/02/20/partnerships-make-it-possib… Note that "beginner tasks" (e.g. "Set up Vagrant") and generic tasks are very welcome (like "Choose and replace 2 uses of Linker::link() from the list in T223010" style). We also have more than 400 unassigned open #good-first-bug tasks: https://phabricator.wikimedia.org/maniphest/query/3YnDUWYJfXSo/#R Can and would you mentor some of these tasks in your area? Please take a moment to find / update [Phabricator etc.] tasks in your project(s) which would take an experienced contributor 2-3 hours. Read https://www.mediawiki.org/wiki/Google_Code-in/Mentors , ask if you have any questions, and add your name to https://www.mediawiki.org/wiki/Google_Code-in/2019#List_of_Wikimedia_mentors Thanks (as we will not be able to run this without your help), andre -- Andre Klapper (he/him) | Bugwrangler / Developer Advocate https://blogs.gnome.org/aklapper/

1 0

Deprecating sysopnames
by Huji Lee 07 Sep '19

07 Sep '19

Dear pywikibot users, Please be advised that "sysopnames" is now deprecated. For all actions that rely on a sysop account (such as deletion or protection), the code now checks the rights of the bot user (as defined by the "usernames" array) and if that user has the necessary right to perform the action, it will be used. If not, an attempt will still be made to use the sysop account as defined in the "sysopnames" array. However, we will soon clean up the code to remove this feature altogether. If a user wants to use different accounts for different actions, then the bot script must explicitly use multiple instance of the Site class. Once we completely remove support for "sysopnames", users who relied on the dualism it caused will be left with a few choices: either get their main account promoted to sysop (or any other group that has the desired rights), or update the bot script such that it explicitly enforces the use of a second account. While we are working on proving a cleaner example of how to deal with two accounts in the bot script, you can refer to this comment on Phabricator <https://phabricator.wikimedia.org/T71283#5446801> as a starter. This has been a long-desired change in pywikibot code base (see T71283 <https://phabricator.wikimedia.org/T71283>). You can contribute to this movement by removing "sysopnames" from your bot's user-config.py and reporting any issues that arise afterwards. You can also help us by updating the documentation on the MediaWiki wiki (for instance here <https://www.mediawiki.org/wiki/Manual:Pywikibot/delete.py>). Sincerely, Huji

1 0

Problems when trying to login as a bot into Commons
by Alex Brollo 28 Aug '19

28 Aug '19

I'm driving a flagged bot into Commons, but some time ago I migrated from Windows to Linux so I'd to reinstall pywikibot ad reconfiguring it. It seems that something is going wrong with the statement: usernames['commons']['commons'] = u'BrolloBot' When running pwb.py login -all, I successfully login into other bot accounts, but login in to commons fails, and I get a message: RuntimeError: dictionary changed size during iteration CRITICAL: Exiting due to uncaught exception <type 'exceptions.RuntimeError'> What is happening please? Alex brollo

2 2

Pywikibot at Wikimania Hackathon
by André Costa 29 Jul '19

29 Jul '19

Hi everyone, Wikimedia Sverige has been running the FindingGLAMs project <https://meta.wikimedia.org/wiki/FindingGLAMs> over the last year which has identified two areas (amongst others) where technical development is needed to unblock GLAM contributions. The first is support for Structured Data on Commons in the major upload tools (of which Pywikibot is a big one), the second is support for uploading Lexicographical data to Wikidata. I’ve seen some Pywikibot activity and discussion on both of these issues but neither seems to have taken off. To try and get either, or both, of these kickstarted I think it would be useful to get some of the active Pywikibot developers together. I believe that even just a joint schematic understanding of how these new features should be implemented would be a good outcome as this would aid actual implementation (as well as subsequent code review). Since Wikimedia Sverige is also hosting Wikimania this year the attached Hackathon <https://wikimania.wikimedia.org/wiki/2019:Hackathon> provides an excellent opportunity for this. If you are attending and interested in joining up for this I’d love to hear from you. Wikimedia Sverige also has a travel budget which can be used for bringing a few selected people working on these areas to the Wikimania Hackathon (this is outside of the scholarship process). Since our budget is limited I’d kindly ask that if you would be interested in receiving some financial support for the travel, please contact me off list and answer the following questions (each answer should be no longer than 100 words): 1. What would you be interested working on? 2. What relevant experience do you have? 3. What is the expected travel cost you would need support with? Related tasks - https://phabricator.wikimedia.org/T223820 - SDC on Commons - https://phabricator.wikimedia.org/T189321 - Lexicographical data - https://phabricator.wikimedia.org/T186200 - Rethink Wikibase data model implementation (may or may not be a blocker to either of the above) Cheers, André Costa (also Lokal_Profil when not wearing my Wikimedia Sverige hat) André Costa | Chief Operating Officer, Wikimedia Sverige | Andre.Costa(a)wikimedia.se | +46 (0)733-964574 Stöd fri kunskap, bli medlem i Wikimedia Sverige. Läs mer på blimedlem.wikimedia.se

1 1

getInterwiki
by masti 22 Jul '19

22 Jul '19

It used to be a getInterwiki method before Wikidata era. What is the easiest way to get them instead of parsing sitelinks list of WD object? masti

2 1

New stable release "3.0.20190722"
by info＠gno.de 22 Jul '19

22 Jul '19

Hi folks, a new stable release "3.0.20190722" was deployed which is available at pypi (as side package without scripts) or from our repository with the tags "3.0.20190430" and "stable". There was a long time between this stable branch and the previous one. The main reason was the implementation of sitelink badges and the implementation of closed wiki access which needed some additional work to get all tests passing. Please note that the last release is always tagged with the "stable" tag. For production systems you should always use this branch instead of master branch because master is under perpetual development (see T217908 at phabricator). PAWS is also based on the stable tagged release. There are currently 96,697 lines of code. The following changes where comming with this change: Important changes which needs operator's attention: * Deprecation warning: support for Python 2 will be dropped in 2020 (T213287). Please update your Python. * deprecate test_family, use wikipedia_family: Site('test', 'wikipedia') instead (T228375, T228300). To have access to test wiki you should instantiate the site object by Site('test', 'wikipedia') instead of Site('test', 'test'). The old method can be used further but can be dropped. * remove the unimplemented "proxy" variable in config.py. You should remove it from user-config.py too. Other changes: * Increase the throttling delay if maxlag >> retry-after (T210606) * deprecate test_family: Site('test', 'test'), use wikipedia_family: Site('test', 'wikipedia') instead (T228375, T228300) * Add "user_agent_description" option in config.py * APISite.fromDBName works for all known dbnames (T225590, 225723, 226960) * Make Family.langs property more robust (T226934) * Remove strategy family * Handle closed_wikis as read-only (T74674) * TokenWallet: login automatically * Add closed_wikis to Family.langs property (T225413) * Redirect 'mo' site code to 'ro' and remove interwiki_replacement_overrides (T225417, T89451) * Add support for badges on Wikibase item sitelinks (T128202) * Remove login.showCaptchaWindow() method * New parameter supplied in suggest_help function for missing dependencies * Remove NonMWAPISite class * Introduce Claim.copy and prevent adding already saved claims (T220131) * Fix create_short_link method after MediaWiki changes (T223865) * Validate proofreadpage.IndexPage contents before saving it * Refactor Link and introduce BaseLink (T66457) * Count skipped pages in BaseBot class * 'actionthrottledtext' is a retryable wikibase error (T192912) * Clear tokens on logout(T222508) * botirc.IRCBot has been dropped * Avoid using outdated browseragents (T222959) * textlib: avoid infinite execution of regex (T222671) * Add CSRF token in sitelogout() api call (T222508) * Refactor WikibasePage.get and overriding methods and improve documentation * Improve title patterns of WikibasePage extensions * Add support for property creation (T160402) For older changes refer HISTORY.rst. Also some changes for scripts where made like: * A lot of commoncat messages got twn support (219094) * script_wui.py was dropped and moved to archive folder (T222759) * default sign was added to welcome.py (T223044) * Quit option was added to disambredir.py (T223048) * Use additional twn messages with checkimages.py (T220178) * redirect.py does no longer fail for RuntimeError retrieving missing redirects (T130911) * Make transferbot.py continue when targetpage cannot be edited by bots or page does not exist (T223816) * Make coordinate_import.py work on a set of Wikidata items (T220806) * Implement -overwrite option in create_categories.py (T220305) * Enable choosing protect level with check_protection_level in protect.py (T225448) * Move categories without leaving a redirect which is the new default behaviour if suppressredirect right is available (T150093) * Namespace identifier can be ommitted for -start option for checkimages.py (T217824) Thanks all contributors for this code changes and all reviewers for their importand code checking and submitting it to the repository. Also thanks for all bug reporters and other persons who gave their valueable comments on phab tasks. Best xqt

1 0

Caching strategy
by Huji Lee 15 Jul '19

15 Jul '19

Hi all, I am working on a bot that fetches a list of anonymous editors on fawiki, uses WHOIS to retrieve more info about that IP, and uses a number of online APIs to check if the IP is a proxy or not.[1] I would like to improve the code by implementing a CIDR cache, so that if I run whois on 8.4.4.8 and determine that its ASN range is 8.4.4.0/24 and then I encounter 8.4.4.9 in the next iteration of my for loop, I would quickly determine this IP also belongs to the same range and skip the WHOIS part for it. The search space would consist of IP ranges like "8.4.4.0 - 8.4.4.25" (these are the beginning and end IP addresses of the 8.4.4.0/24 range). Obviously, we can convert these IPs to Hex to make comparisons easier. Given an IP like 8.4.4.9, we need the object to efficiently determine if it already has an IP range that encompasses this given IP and if so, return the previously cached details for that IP pair. If not, we will store that in cache. The part that I am not fully clear about is the following: how can I avoid having to loop through every range in the cache? Is there a way to create a hash function that checks two inequality comparisons efficiently? Thanks! Huji [1] https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/findproxy…

4 4

Re: [pywikibot] Caching strategy
by Huji Lee 13 Jul '19

13 Jul '19

@Lucas: cidr-trie was exactly what I wanted; thank you! @Strainu: In our use case (fawiki), the number of distinct IPs that make an edit in each day is not too large (usually a few hundred). Therefore, the memory intensity of using a CIDR-trie is minimal. The code at [1] has been updated to include caching in the way that I had desired. Thanks again, Huji [1] https://github.com/PersianWikipedia/fawikibot/blob/master/HujiBot/findproxy… ----------- I'm very curious if you can run at Wikipedia scale with such a trie in memory on a normal computer (e.g. with only tens of GiB of memory). Please let us know if you actually get this into production (or just submit the script for inclusion in the framework, it sounds really useful) Strainu Pe vineri, 12 iulie 2019, Lucas Werkmeister <mail at lucaswerkmeister.de <https://lists.wikimedia.org/mailman/listinfo/pywikibot>> a scris: >* You probably want to use a trie <https://en.wikipedia.org/wiki/Trie <https://en.wikipedia.org/wiki/Trie>> for *>* this – I found several available Python implementations, but I don’t know *>* what their advantages or disadvantages are, so I’ll just list them in *>* alphabetical order: *>>* - cidr-tree <https://github.com/Figglewatts/cidr-trie <https://github.com/Figglewatts/cidr-trie>> *>* - py-radix <https://github.com/Figglewatts/cidr-trie <https://github.com/Figglewatts/cidr-trie>> *>* - pysubnettree <https://github.com/zeek/pysubnettree <https://github.com/zeek/pysubnettree>> *>* - pytricia <https://github.com/jsommers/pytricia <https://github.com/jsommers/pytricia>> *>>* Cheers, *>* Lucas *>* On 12.07.19 04:43, Huji Lee wrote: *>

1 0

Structured data on Commons and Pywikibot
by Maarten Dammers 19 May '19

19 May '19

Hi everyone, It was nice to have the Pywikibot meet up at the hackathon (see https://phabricator.wikimedia.org/T221801 if you weren't there). One of the things we talked about was the lack of support for structured data on Commons. I played around a bit this weekend to just see what is possible and implemented importing of captions ( https://phabricator.wikimedia.org/T223746) and importing of statements ( https://phabricator.wikimedia.org/T223796 ). Now it's time to properly implement the support for structured data on Commons in Pywikibot. For that I created https://phabricator.wikimedia.org/T223820 . Any takers? Maarten

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot