Pywikipedia-svn March 2012

pywikipedia-svn@lists.wikimedia.org

6 participants
129 discussions

SVN: [10032] trunk/pywikipedia/featured.py
by valhallasw＠svn.wikimedia.org 21 Mar '12

21 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10032 Revision: 10032 Author: valhallasw Date: 2012-03-21 17:26:17 +0000 (Wed, 21 Mar 2012) Log Message: ----------- Improved Featured category for ruwiki. Patch by GanZ <ganz-ru(a)users.sourceforge.net> Reference: pywikipediabot-Patches-3465147 Modified Paths: -------------- trunk/pywikipedia/featured.py Modified: trunk/pywikipedia/featured.py =================================================================== --- trunk/pywikipedia/featured.py 2012-03-21 17:22:56 UTC (rev 10031) +++ trunk/pywikipedia/featured.py 2012-03-21 17:26:17 UTC (rev 10032) @@ -260,7 +260,7 @@ 'pl': (CAT, u"Dobre artykuły"), 'pt': (CAT, u"Artigos bons"), 'ro': (BACK, u"Articol bun"), - 'ru': (CAT, u"Википедия:Хорошие статьи"), + 'ru': (CAT, u"Википедия:Хорошие статьи по алфавиту"), 'simple': (CAT, u"Good articles"), 'sr': (BACK,u"Иконица добар"), 'sv': (CAT, u"Wikipedia:Bra artiklar"),

1 0

SVN: [10031] trunk/pywikipedia/wikipedia.py
by valhallasw＠svn.wikimedia.org 21 Mar '12

21 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10031 Revision: 10031 Author: valhallasw Date: 2012-03-21 17:22:56 +0000 (Wed, 21 Mar 2012) Log Message: ----------- Search compatibility for MW <= 1.16 http://www.mediawiki.org/wiki/Release_notes/1.16 : "Added fields to list=search output: size, wordcount, timestamp, snippet" Original patch by K.-M. Hansche (loxley(a)users.sourceforge.net) Reference: pywikipediabot-Patches-3408799 Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2012-03-21 17:10:58 UTC (rev 10030) +++ trunk/pywikipedia/wikipedia.py 2012-03-21 17:22:56 UTC (rev 10031) @@ -6095,7 +6095,10 @@ for s in data['search']: offset += 1 page = Page(self, s['title']) - yield page, s['snippet'], '', s['size'], s['wordcount'], s['timestamp'] + if self.versionnumber() >= 16: + yield page, s['snippet'], '', s['size'], s['wordcount'], s['timestamp'] + else: + yield page, '', '', '', '', '' else: #Yield search results (using Special:Search page) for query. throttle = True

1 0

SVN: [10030] trunk/pywikipedia/spellcheck.py
by valhallasw＠svn.wikimedia.org 21 Mar '12

21 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10030 Revision: 10030 Author: valhallasw Date: 2012-03-21 17:10:58 +0000 (Wed, 21 Mar 2012) Log Message: ----------- Refactoring of single page checking code Patch by K.-M. Hansche (loxley(a)users.sourceforge.net) Reference: pywikipediabot-Patches-3404332 Modified Paths: -------------- trunk/pywikipedia/spellcheck.py Modified: trunk/pywikipedia/spellcheck.py =================================================================== --- trunk/pywikipedia/spellcheck.py 2012-03-19 13:36:57 UTC (rev 10029) +++ trunk/pywikipedia/spellcheck.py 2012-03-21 17:10:58 UTC (rev 10030) @@ -420,6 +420,16 @@ newwords.append(self.word) return self.alternatives +def checkPage(page, checknames=True, knownonly=False): + try: + text = page.get() + except pywikibot.Error: + pass + else: + text = spellcheck(text, checknames=checknames, knownonly=knownonly, title=page.title()) + if text != page.get(): + page.put(text) + try: pageskip = [] edit = SpecialTerm("edit") @@ -499,38 +509,14 @@ try: if newpages: for (page, date, length, loggedIn, user, comment) in pywikibot.getSite().newpages(1000): - try: - text = page.get() - except pywikibot.Error: - pass - else: - text = spellcheck(text, checknames=checknames, - knownonly=knownonly, title=page.title()) - if text != page.get(): - page.put(text) + checkPage(page, checknames, knownonly) elif start: for page in pagegenerators.PreloadingGenerator(pagegenerators.AllpagesPageGenerator(start=start,includeredirects=False)): - try: - text = page.get() - except pywikibot.Error: - pass - else: - text = spellcheck(text, checknames=checknames, - knownonly=knownonly, title=page.title()) - if text != page.get(): - page.put(text) + checkPage(page, checknames, knownonly) if longpages: for (page, length) in pywikibot.getSite().longpages(500): - try: - text = page.get() - except pywikibot.Error: - pass - else: - text = spellcheck(text, checknames=checknames, - knownonly=knownonly, title=page.title()) - if text != page.get(): - page.put(text) + checkPage(page, checknames, knownonly) else: title = ' '.join(title) @@ -543,9 +529,7 @@ except pywikibot.IsRedirectPage: print "Page is a redirect page" else: - text = spellcheck(text,knownonly=knownonly, title=page.title()) - if text != page.get(): - page.put(text) + checkPage(page, knownonly=knownonly) title = pywikibot.input(u"Which page to check now? (enter to stop)") finally: pywikibot.stopme()

1 0

SVN: [10029] trunk/pywikipedia/pywikibot/textlib.py
by xqt＠svn.wikimedia.org 19 Mar '12

19 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10029 Revision: 10029 Author: xqt Date: 2012-03-19 13:36:57 +0000 (Mon, 19 Mar 2012) Log Message: ----------- missing coma, fix for r10028 (bug #3508665) Modified Paths: -------------- trunk/pywikipedia/pywikibot/textlib.py Modified: trunk/pywikipedia/pywikibot/textlib.py =================================================================== --- trunk/pywikipedia/pywikibot/textlib.py 2012-03-19 12:15:04 UTC (rev 10028) +++ trunk/pywikipedia/pywikibot/textlib.py 2012-03-19 13:36:57 UTC (rev 10029) @@ -178,7 +178,7 @@ replacement[groupMatch.end():] except IndexError: print '\nInvalid group reference:', groupID - print 'Groups found:\n' match.groups() + print 'Groups found:\n', match.groups() raise IndexError text = text[:match.start()] + replacement + text[match.end():]

1 0

SVN: [10028] trunk/pywikipedia/pywikibot/textlib.py
by xqt＠svn.wikimedia.org 19 Mar '12

19 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10028 Revision: 10028 Author: xqt Date: 2012-03-19 12:15:04 +0000 (Mon, 19 Mar 2012) Log Message: ----------- Debugging information for IndexError Modified Paths: -------------- trunk/pywikipedia/pywikibot/textlib.py Modified: trunk/pywikipedia/pywikibot/textlib.py =================================================================== --- trunk/pywikipedia/pywikibot/textlib.py 2012-03-18 18:05:09 UTC (rev 10027) +++ trunk/pywikipedia/pywikibot/textlib.py 2012-03-19 12:15:04 UTC (rev 10028) @@ -172,9 +172,14 @@ break groupID = groupMatch.group('name') or \ int(groupMatch.group('number')) - replacement = replacement[:groupMatch.start()] + \ - match.group(groupID) + \ - replacement[groupMatch.end():] + try: + replacement = replacement[:groupMatch.start()] + \ + match.group(groupID) + \ + replacement[groupMatch.end():] + except IndexError: + print '\nInvalid group reference:', groupID + print 'Groups found:\n' match.groups() + raise IndexError text = text[:match.start()] + replacement + text[match.end():] # continue the search on the remaining text

1 0

SVN: [10027] branches/rewrite/tests
by valhallasw＠svn.wikimedia.org 18 Mar '12

18 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10027 Revision: 10027 Author: valhallasw Date: 2012-03-18 18:05:09 +0000 (Sun, 18 Mar 2012) Log Message: ----------- Added textlib (section header) tests. See also: pywikipediabot-Bugs-2776438 Added Paths: ----------- branches/rewrite/tests/pages/ branches/rewrite/tests/pages/enwiki_help_editing.meta branches/rewrite/tests/pages/enwiki_help_editing.page branches/rewrite/tests/textlib_tests.py Added: branches/rewrite/tests/pages/enwiki_help_editing.meta =================================================================== --- branches/rewrite/tests/pages/enwiki_help_editing.meta (rev 0) +++ branches/rewrite/tests/pages/enwiki_help_editing.meta 2012-03-18 18:05:09 UTC (rev 10027) @@ -0,0 +1,2 @@ +Help:Editing. (2012, March 9). In Wikipedia, The Free Encyclopedia. Retrieved 17:43, March 18, 2012, from http://en.wikipedia.org/w/index.php?title=Help:Editing&oldid=480978372 +Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. Added: branches/rewrite/tests/pages/enwiki_help_editing.page =================================================================== --- branches/rewrite/tests/pages/enwiki_help_editing.page (rev 0) +++ branches/rewrite/tests/pages/enwiki_help_editing.page 2012-03-18 18:05:09 UTC (rev 10027) @@ -0,0 +1,174 @@ + +{{redirect|Wikipedia:Editing|the editing policy|Wikipedia:Editing policy}} +<noinclude>{{semiprotected|small=yes}}</noinclude> +{{Wikipedia how to|WP:H2EAP|WP:HEP|WP:HOW}} +{{See also|Wikipedia:Introduction|Wikipedia:Tutorial|Wikipedia:Starting an article|Wikipedia:Manual of Style}} +[[File:Wikipedia video tutorial-1-Editing-en.ogv|thumb|250px|Editing tutorial for Wikipedia]] +[[Wikipedia]] is a [[wiki]], meaning that anyone can edit any [[Wikipedia:Protection policy|unprotected page]] and improve articles immediately for all readers. You do not need to register to do this. Anyone who has edited is known as a "[[Wikipedia: Wikipedian|Wikipedian]]" and, no matter how trivial the edit may seem, can be proud that he or she has helped make Wikipedia what it is. All of these edits add up!  + +However, some pages are [[WP:Protection policy|protected from editing]]. These pages have a <tt>View source</tt> tab instead of an <tt>Edit</tt> tab. You can still edit these pages indirectly, by submitting an "[[Wikipedia:Edit requests|edit request]]" - an editor with the ability to edit the protected page will respond to your request. You can submit a request by clicking on the <tt>View source</tt> tab on that page and using the "Submit an edit request" link at the bottom right. + +<div class="messagebox" style="float: right; width: 150px; margin-left: 10px;">Note: You can use the [[Wikipedia:Sandbox|sandbox]] to experiment with page editing.</div> + +==Editing== +Editing most Wikipedia pages is simple. Simply click on the "'''edit'''" tab at the top of a Wikipedia page (or on a [[Wikipedia: Section|section-edit link]]). This will bring you to a new page with a [[text box]] containing the editable text of the current page. + +[[File:How to edit a page Edit box.png|thumb|600px|center|alt=Text in a large rectangle below two toolbars and next to a scrollbar.|Edit box showing the [[Help:wiki markup|wiki markup]] for the [[February 2011 Christchurch earthquake]] article.]] +<br clear=all> + +In this box, you can type in the text that you want to add, using [[Help:wiki markup|wiki markup]] to format the text and add other elements like images and tables. The [[Help:Edit toolbar|toolbar]] above the text box can help with formatting. A [[Wikipedia:Cheatsheet|quick reference]] to wiki markup can be opened in a new window by clicking 'Editing help', found near the 'save page' button. + +Never start a line with a leading space unless you want the special formatting it causes. Paragraphs can be separated with a blank line. The Wikipedia community has developed [[Wikipedia:Manual of Style|style guidelines]] to make articles and facts appear in a standardized form, and Wikipedia easier to use as a whole. + +If you add information to a page, [[Wikipedia:Citing sources|please provide references]], as [[Wikipedia:Verifiability|unreferenced statements are subject to removal]]. + +When you have finished editing, you should write a short [[Help:Edit summary|edit summary]] in the small field below the edit-box. You may use shorthand to describe your changes, as described in the [[Wikipedia:Edit summary legend|legend]]. To see how the page looks with your edits, press the "'''Show preview'''" button. To see the differences between the page with your edits and the previous version of the page, press the "'''Show changes'''" button. If you're satisfied with what you see, '''[[Wikipedia:Be bold|be bold]]''' and press the "'''Save page'''" button. Your changes will immediately be visible to all Wikipedia users. + +{{EditOptions|align=right|width=50em|font-size=80%}} + +Note: Do not sign the Edit Summary line with your <nowiki>~~~~</nowiki> signature as it does not work there. + +===Talk (discussion) pages=== +{{Further2|[[Help:Using talk pages]]|[[Wikipedia:Talk page guidelines]]}} +Talk pages are similar to articles in that they also have a "new section" tab to start a new section or edit the whole page. + +===Minor edits=== +{{Further2|[[Help:Minor edit]]}} +A check to the "minor edit" box signifies that only superficial differences exist between the version with your edit and the previous version: typo corrections, formatting and presentational changes, rearranging of text without modifying content, etc. A ''minor edit'' is a version that the editor believes requires no review and could never be the subject of a dispute. The "minor edit" option is one of several [[Wikipedia:Why create an account?#New editing options|options]] available only to [[Help:Logging in|registered users]]. + +===Major edits=== +{{see|Wikipedia:Editing policy#Talking and editing}} + +All editors are encouraged to [[Wikipedia:Be bold|be bold]], but there are several things that a user can do to ensure that major edits are performed smoothly. Before engaging in a major edit, a user should consider discussing proposed changes on the article discussion/talk page. During the edit, if doing so over an extended period, the {{tlx|In use}} tag can reduce the likelihood of an edit conflict. Once the edit has been completed, the inclusion of an [[Help:Edit summary|edit summary]] will assist in documenting the changes. These steps will help all to ensure that major edits are well received by the Wikipedia community. + +A major edit should be reviewed to confirm that it is consensual to all concerned editors. Therefore, any change that affects the ''meaning'' of an article is major (not minor), even if the edit is a single word. + +There are no necessary terms to which you have to agree when doing major edits, but the recommendations above have become best practice. If you do it your own way, the likelihood of your edits being reedited may be higher. + + +When performing a large edit, it is suggested you periodically, and before pressing 'save page', copy your edits into an external text editor (preferably one without formatting, such as [[Notepad (Windows)|Notepad]]). This ensures that in the case of a browser crash you will not lose your work. If you are adding substantial amounts of work, it is also a good idea to save changes in stages. + +==="View source" – protected pages=== +Some pages are [[WP:PROT|protected from editing]]. These pages have a <tt>View source</tt> tab instead of an <tt>Edit</tt> tab. You can still edit these pages indirectly, by submitting an "[[Wikipedia:Edit requests|edit request]]" - an editor with the ability to edit the protected page will respond to your request. You can submit a request by clicking on the <tt>View source</tt> tab on that page and using the "Submit an edit request" link at the bottom right. + +===Adding references=== +There is a [[Wikipedia:refToolbar 2.0|tool]] built into the top of the edit box that makes it easier to add and properly format references. The tool can use a single piece of data, such as the [[PMID]] or [[ISBN]], to fill in the rest of the details. + +==Wiki markup== +{{main|Help:Wiki markup}} +Wiki markup is the extra information (apart from the text which will be displayed) you enter in the edit window which tells the [[MediaWiki]] software how to display, categorize and process the article. + +==More information on editing wiki pages== +===Getting started=== +* [[Wikipedia:Starting an article|Starting an article]] + +===Policies and conventions=== +Make sure that you submit information which is relevant to Wikipedia's specific purpose, or your content might be deleted. You can always use the [[Help:Talk page|talk pages]] to ask questions or check to see if your idea will be accepted. Please make note of the license your contributions will be covered by. +* [[Wikipedia:Policies and guidelines|Policies and guidelines]] + +===Helpful tips=== +* [[Wikipedia:Contributing to Wikipedia|Contributing to Wikipedia]] +* [[Wikipedia:FAQ/Editing|FAQ/Editing]] +* [[Wikipedia:Cheatsheet|Cheatsheet]] +* [[Wikipedia:Page size#If you have problems editing a long article|If you have problems editing a long article]] +* [[Help:Protection|If the article is protected from editing]] +* [[Help:Page history|About page history]] + +===Naming and moving=== +* [[Wikipedia:Moving a page|Moving a page to a new name]] +* [[Wikipedia:Article titles|Article titles]] +* [[Wikipedia:Namespace|Namespace]] + +===Style and layout=== +* [[Wikipedia:Layout|Layout]] +* [[Wikipedia:Manual of Style|Manual of Style]] +* [[Wikipedia:Annotated article|Annotated article]] + +===Tools=== +* [[Wikipedia:Text editor support|Text editor support]] + +===See also=== +{{Meta}} +* [[WP:WikiProject|WikiProjects]], if you are writing an article about something that belongs to a group of topics, check here first! +* [[outreach:Bookshelf|Wikimedia Bookshelf]], learning materials, videos, and handouts +{{clear}} +{{Leave feedback|format=table}} + +<noinclude> +[[Category:Wikipedia how-to|How to edit a page]] +[[Category:Wikipedia basic information|How to edit a page]] +[[Category:Wikipedia editing| ]] + + +</noinclude> + +[[af:Wikipedia:Redigeringsinstruksies]] +[[als:Hilfe:Seite bearbeiten]] +[[am:እርዳታ:የማዘጋጀት እርዳታ]] +[[ang:Wikipedia:How to edit a page]] +[[ar:ويكيبيديا:كيف تعدل صفحة]] +[[as:সহায়:সম্পাদনা]] +[[az:Vikipediya:Məqalələrin redaktə qaydaları]] +[[bn:উইকিপিডিয়া:কীভাবে একটি পৃষ্ঠা সম্পাদনা করবেন]] +[[zh-min-nan:Help:Pian-chi̍p]] +[[be-x-old:Вікіпэдыя:Як рэдагаваць існуючы артыкул]] +[[bg:Уикипедия:Редактиране на страници]] +[[bs:Wikipedia:Kako izmijeniti stranicu]] +[[br:Skoazell:Penaos kemmañ ur bajenn]] +[[ca:Ajuda:Com es modifica una pàgina]] +[[cv:Broken/Статьясене тӳрлетесси çинчен]] +[[cs:Nápověda:Jak editovat stránku]] +[[ny:Wikipedia:Chithandizo cha Kulemba pa Wikipedia]] +[[cy:Wicipedia:Sut i olygu tudalen]] +[[de:Hilfe:Seite bearbeiten]] +[[dv:ވިކިޕީޑިޔާ: ޞަފްހާ އަކަށް ނުވަތަ ލިޔެވިފައިވާ މަޒްމޫނަކަށް ބަދަލު ގެނައުމަށް އެހީ އެއް]] +[[el:Βοήθεια:Πώς να επεξεργαστείτε μια σελίδα]] +[[es:Ayuda:Cómo se edita una página]] +[[eo:Helpo:Kiel redakti paĝon]] +[[eu:Laguntza:Artikuluak nola aldatzen diren]] +[[fa:ویکی‌پدیا:راهنمای ویرایش صفحه‌ها]] +[[fr:Aide:Comment modifier une page]] +[[ga:Vicipéid:Conas alt a chur in eagar]] +[[gv:Wikipedia:Kanys duillag dy reaghey]] +[[gl:Wikipedia:Como editar unha páxina]] +[[hy:Վիքիփեդիա:Ինչպես խմբագրել էջը]] +[[hr:Wikipedija:Kako uređivati stranicu]] +[[id:Wikipedia:Menyunting sebuah halaman]] +[[ia:Wikipedia:Como formatar texto]] +[[it:Aiuto:Modifica]] +[[he:עזרה:עריכת דף]] +[[ku:Wîkîpediya:How does one edit a page]] +[[la:Vicipaedia:De recensendo]] +[[lv:Vikipēdija:Kā rediģēt lapu]] +[[lb:Hëllef/Textgestaltung]] +[[lt:Pagalba:Kaip redaguoti puslapį]] +[[hu:Wikipédia:Szerkesztés]] +[[mk:Помош:Како да уредам страница]] +[[ms:Wikipedia:Menyunting]] +[[mn:Тусламж:Арай өндөр төвшинд текст засварлах талаар]] +[[nl:Help:Uitleg]] +[[ja:Help:ページの編集]] +[[no:Hjelp:Hvordan man redigerer en side]] +[[nn:Hjelp:Endra ei side]] +[[or:ଉଇକିପିଡ଼ିଆ:କିପରି ଗୋଟିଏ ଫରଦରେ ଅଦଳ ବଦଳ କରିବେ]] +[[pl:Pomoc:Formatowanie tekstu]] +[[pt:Ajuda:Guia de edição/Formatação]] +[[ro:Ajutor:Cum modific o pagină]] +[[ru:Википедия:Как править статьи]] +[[si:උදවු:සංස්කරණය]] +[[simple:Help:How to change pages]] +[[sk:Pomoc:Prehľad editácie stránok]] +[[sl:Wikipedija:Urejanje strani]] +[[sr:Помоћ:Уређивање]] +[[su:Wikipedia:Cara ngédit kaca]] +[[fi:Ohje:Kuinka sivuja muokataan]] +[[sv:Wikipedia:Redigering]] +[[th:วิธีใช้:การแก้ไขหน้า]] +[[tg:Википедиа:Вироиши мақолаҳо]] +[[tr:Yardım:Sayfa nasıl yazılır?]] +[[ur:صفحہ کس طرح ترمیم کریں]] +[[vi:Wikipedia:Sửa đổi]] +[[yi:װיקיפּעדיע:וויאזוי צו רעדאקטירן א בלאט]] +[[yo:Ìrànwọ́:Báwo lẹṣe le ṣe àtúnṣe ojúewé]] +[[zh-yue:Wikipedia:點改嘢]] +[[zh:Help:编辑页面]] Added: branches/rewrite/tests/textlib_tests.py =================================================================== --- branches/rewrite/tests/textlib_tests.py (rev 0) +++ branches/rewrite/tests/textlib_tests.py 2012-03-18 18:05:09 UTC (rev 10027) @@ -0,0 +1,52 @@ +# -*- coding: utf-8 -*- +# +# (C) Pywikipedia bot team, 2007 +# +# Distributed under the terms of the MIT license. +# +__version__ = '$Id: api_tests.py 8238 2010-06-02 13:50:48Z xqt $' + +import unittest +import codecs +import os + +import pywikibot +import pywikibot.textlib as textlib + +files = {} +dirname = os.path.join(os.path.dirname(__file__), "pages") + +for f in ["enwiki_help_editing"]: + files[f] = codecs.open(os.path.join(dirname, f + ".page"), 'r', 'utf-8').read() + +class TestSectionFunctions(unittest.TestCase): + def contains(self, fn, sn): + return textlib.does_text_contain_section( + files[fn], sn) + + def assertContains(self, fn, sn, *args, **kwargs): + self.assertEqual(self.contains(fn, sn), True, *args, **kwargs) + + def assertNotContains(self, fn, sn, *args, **kwargs): + self.assertEqual(self.contains(fn, sn), False, *args, **kwargs) + + def testCurrentBehaviour(self): + self.assertContains("enwiki_help_editing", u"Editing") + + @unittest.expectedFailure + def testSpacesInSection(self): + self.assertContains("enwiki_help_editing", u"Minor_edits") + self.assertNotContains("enwiki_help_editing", u"Minor edits", "Incorrect, '#Minor edits' does not work") + self.assertNotContains("enwiki_help_editing", u"Minor Edits", "section hashes are case-sensitive") + self.assertNotContains("enwiki_help_editing", u"Minor_Edits", "section hashes are case-sensitive") + + @unittest.expectedFailure + def testNonAlphabeticalCharactersInSection(self): + self.assertContains("enwiki_help_editing", u"Talk_.28discussion.29_pages", "As used in the TOC") + self.assertContains("enwiki_help_editing", u"Talk_(discussion)_pages", "Understood by mediawiki") + +if __name__ == '__main__': + try: + unittest.main() + except SystemExit: + pass

1 0

SVN: [10026] branches/rewrite/pywikibot/textlib.py
by valhallasw＠svn.wikimedia.org 18 Mar '12

18 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10026 Revision: 10026 Author: valhallasw Date: 2012-03-18 17:26:49 +0000 (Sun, 18 Mar 2012) Log Message: ----------- Applies r10024 to the rewrite Modified Paths: -------------- branches/rewrite/pywikibot/textlib.py Modified: branches/rewrite/pywikibot/textlib.py =================================================================== --- branches/rewrite/pywikibot/textlib.py 2012-03-18 17:23:48 UTC (rev 10025) +++ branches/rewrite/pywikibot/textlib.py 2012-03-18 17:26:49 UTC (rev 10026) @@ -934,3 +934,14 @@ text += u'|%s=%s\n' % (item, params[item]) return u'{{%s\n%s}}' % (template, text) + +#---------------------------------- +# Page parsing functionality +#---------------------------------- + +def does_text_contain_section(pagetext, section): + """ Determines whether the page text contains the given + section title. + """ + m = re.search("=+[ ']*%s[ ']*=+" % re.escape(section), pagetext) + return bool(m)

1 0

SVN: [10025] branches/rewrite/pywikibot/textlib.py
by valhallasw＠svn.wikimedia.org 18 Mar '12

18 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10025 Revision: 10025 Author: valhallasw Date: 2012-03-18 17:23:48 +0000 (Sun, 18 Mar 2012) Log Message: ----------- (somewhat blunt) merge of trunk/pywikibot/textlib.py Modified Paths: -------------- branches/rewrite/pywikibot/textlib.py Modified: branches/rewrite/pywikibot/textlib.py =================================================================== --- branches/rewrite/pywikibot/textlib.py 2012-03-18 17:08:41 UTC (rev 10024) +++ branches/rewrite/pywikibot/textlib.py 2012-03-18 17:23:48 UTC (rev 10025) @@ -18,6 +18,7 @@ import re from pywikibot.i18n import translate +from HTMLParser import HTMLParser def unescape(s): """Replace escaped HTML-special characters by their originals""" @@ -221,6 +222,40 @@ return toRemoveR.sub('', text) +def removeHTMLParts(text, keeptags = ['tt', 'nowiki', 'small', 'sup']): + """ + Return text without portions where HTML markup is disabled + + Parts that can/will be removed are -- + * HTML and all wiki tags + + The exact set of parts which should NOT be removed can be passed as the + 'keeptags' parameter, which defaults to ['tt', 'nowiki', 'small', 'sup']. + """ + # try to merge with 'removeDisabledParts()' above into one generic function + + # thanks to http://www.hellboundhackers.org/articles/841-using-python-39;s-htmlparser-c… + parser = _GetDataHTML() + parser.keeptags = keeptags + parser.feed(text) + parser.close() + return parser.textdata + +# thanks to http://docs.python.org/library/htmlparser.html +class _GetDataHTML(HTMLParser): + textdata = u'' + keeptags = [] + + def handle_data(self, data): + self.textdata += data + + def handle_starttag(self, tag, attrs): + if tag in self.keeptags: self.textdata += u"<%s>" % tag + + def handle_endtag(self, tag): + if tag in self.keeptags: self.textdata += u"</%s>" % tag + + def isDisabled(text, index, tags = ['*']): """ Return True if text[index] is disabled, e.g. by a comment or by nowiki tags. @@ -269,12 +304,24 @@ #------------------------------------------------- # Functions dealing with interwiki language links #------------------------------------------------- -# Note - MediaWiki supports two kinds of interwiki links; interlanguage and -# interproject. These functions only deal with links to a -# corresponding page in another language on the same project (e.g., -# Wikipedia, Wiktionary, etc.) in another language. They do not find -# or change links to a different project, or any that are formatted -# as in-line interwiki links (e.g., "[[:es:Articulo]]". (CONFIRM) +# Note - MediaWiki supports several kinds of interwiki links; two kinds are +# interlanguage links. We deal here with those kinds only. +# A family has by definition only one kind of interlanguage links: +# 1 - interlanguage links inside the own family. +# They go to a corresponding page in another language in the same +# family, such as from 'en.wikipedia' to 'pt.wikipedia', or from +# 'es.wiktionary' to 'arz.wiktionary'. +# Families with this kind have several language-specific sites. +# They have their interwiki_forward attribute set to None +# 2 - language links forwarding to another family. +# They go to a corresponding page in another family, such as from +# 'commons' to 'zh.wikipedia, or from 'incubator' to 'en.wikipedia'. +# Families having those have one member only, and do not have +# language-specific sites. The name of the target family of their +# interlanguage links is kept in their interwiki_forward attribute. +# These functions only deal with links of these two kinds only. They +# do not find or change links of other kinds, nor any that are formatted +# as in-line interwiki links (e.g., "[[:es:Articulo]]". def getLanguageLinks(text, insite=None, pageLink="[[]]", template_subpage=False): """ @@ -287,6 +334,10 @@ """ if insite is None: insite = pywikibot.getSite() + fam = insite.family + # when interwiki links forward to another family, retrieve pages & other infos there + if fam.interwiki_forward: + fam = pywikibot.Family(fam.interwiki_forward) result = {} # Ignore interwiki links within nowiki tags, includeonly tags, pre tags, # and HTML comments @@ -299,20 +350,28 @@ # interwiki link. # NOTE: language codes are case-insensitive and only consist of basic latin # letters and hyphens. + #TODO: currently, we do not have any, but BCP 47 allows digits, and underscores. + #TODO: There is no semantic difference between hyphens and underscores -> fold them. interwikiR = re.compile(r'\[\[([a-zA-Z\-]+)\s?:([^\[\]\n]*)\]\]') for lang, pagetitle in interwikiR.findall(text): lang = lang.lower() # Check if it really is in fact an interwiki link to a known # language, or if it's e.g. a category tag or an internal link - if lang in insite.family.obsolete: - lang = insite.family.obsolete[lang] - if lang in insite.validLanguageLinks(): + if lang in fam.obsolete: + lang = fam.obsolete[lang] + if lang in fam.langs.keys(): if '|' in pagetitle: # ignore text after the pipe pagetitle = pagetitle[:pagetitle.index('|')] # we want the actual page objects rather than the titles - site = insite.getSite(code = lang) - result[site] = pywikibot.Page(pywikibot.Link(pagetitle, site)) + site = pywikibot.getSite(code=lang, fam=fam) + try: + result[site] = pywikibot.Page(site, pagetitle, insite=insite) + except pywikibot.InvalidTitle: + pywikibot.output( + u"[getLanguageLinks] Text contains invalid interwiki link [[%s:%s]]." + % (lang, pagetitle)) + continue return result @@ -386,7 +445,11 @@ if s: if site.language() in site.family.interwiki_attop or \ u'' in oldtext: - newtext = s + separator + s2.replace(marker,'').strip() + #do not add separator if interiki links are on one line + newtext = s + \ + [separator, u''][site.language() in + site.family.interwiki_on_one_line] + \ + s2.replace(marker, '').strip() else: # calculate what was after the language links on the page firstafter = s2.find(marker) @@ -407,6 +470,12 @@ site) + separator + s newtext = replaceCategoryLinks(s2, cats, site=site, addOnly=True) + # for Wikitravel's language links position. + # (not supported by rewrite - no API) + elif site.family.name == 'wikitravel': + s = separator + s + separator + newtext = s2[:firstafter].replace(marker,'') + s + \ + s2[firstafter:] else: if template or template_subpage: if template_subpage: @@ -451,13 +520,11 @@ ar = interwikiSort(links.keys(), insite) s = [] for site in ar: - obj = links[site] - if isinstance(obj, pywikibot.Link): - link = obj.astext(insite) - else: - # Page - link = obj.title(asLink=True, forceInterwiki=True) - s.append(link) + try: + link = unicode(links[site]).replace('[[:', '[[') + s.append(link) + except AttributeError: + s.append(getSite(site).linkto(links[site], othersite=insite)) if insite.lang in insite.family.interwiki_on_one_line: sep = u' ' else: @@ -497,7 +564,8 @@ # Functions dealing with category links #--------------------------------------- -def getCategoryLinks(text, site): +def getCategoryLinks(text, site=None): + import catlib """Return a list of category links found in text. List contains Category objects. @@ -505,6 +573,8 @@ """ result = [] + if site is None: + site = pywikibot.getSite() # Ignore category links within nowiki tags, pre tags, includeonly tags, # and HTML comments text = removeDisabledParts(text) @@ -522,7 +592,7 @@ return result -def removeCategoryLinks(text, site, marker=''): +def removeCategoryLinks(text, site=None, marker=''): """Return text with all category links removed. Put the string marker after the last replacement (at the end of the text @@ -533,6 +603,8 @@ # interwiki link, plus trailing whitespace. The language code is grouped. # NOTE: This assumes that language codes only consist of non-capital # ASCII letters and hyphens. + if site is None: + site = pywikibot.getSite() catNamespace = '|'.join(site.category_namespaces()) categoryR = re.compile(r'\[\[\s*(%s)\s*:.*?\]\]\s*' % catNamespace, re.I) text = replaceExcept(text, categoryR, '', @@ -554,6 +626,8 @@ if there is no replacement). """ + if site is None: + site = pywikibot.getSite() if separator: mymarker = findmarker(text, u'@C@') newtext = removeCategoryLinks(text, site, mymarker) @@ -609,7 +683,8 @@ Replace the category links given in the wikitext given in oldtext by the new links given in new. - 'new' should be a list of Category objects. + 'new' should be a list of Category objects or strings + which can be either the raw name or [[Category:..]]. If addOnly is True, the old category won't be deleted and the category(s) given will be added (and so they won't replace anything). @@ -671,7 +746,8 @@ def categoryFormat(categories, insite = None): """Return a string containing links to all categories in a list. - 'categories' should be a list of Category objects. + 'categories' should be a list of Category objects or strings + which can be either the raw name or [[Category:..]]. The string is formatted for inclusion in insite. @@ -680,7 +756,15 @@ return '' if insite is None: insite = pywikibot.getSite() - catLinks = [category.aslink() for category in categories] + + if isinstance(categories[0],basestring): + if categories[0][0] == '[': + catLinks = categories + else: + catLinks = ['[[Category:'+category+']]' for category in categories] + else: + catLinks = [category.aslink(noInterwiki=True) for category in categories] + if insite.category_on_one_line(): sep = ' ' else: @@ -834,3 +918,19 @@ # Add it to the result result.append((name, params)) return result + + +def glue_template_and_params(template_and_params): + """Return wiki text of template glued from params. + + You can use items from extract_templates_and_params here to get + an equivalent template wiki text (it may happen that the order + of the params changes). + """ + (template, params) = template_and_params + + text = u'' + for item in params: + text += u'|%s=%s\n' % (item, params[item]) + + return u'{{%s\n%s}}' % (template, text)

1 0

SVN: [10024] trunk/pywikipedia
by valhallasw＠svn.wikimedia.org 18 Mar '12

18 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10024 Revision: 10024 Author: valhallasw Date: 2012-03-18 17:08:41 +0000 (Sun, 18 Mar 2012) Log Message: ----------- Split off section recognition functionality to textlib.py. See also: pywikipediabot-Bugs-2776438 Modified Paths: -------------- trunk/pywikipedia/pywikibot/textlib.py trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/pywikibot/textlib.py =================================================================== --- trunk/pywikipedia/pywikibot/textlib.py 2012-03-17 18:03:32 UTC (rev 10023) +++ trunk/pywikipedia/pywikibot/textlib.py 2012-03-18 17:08:41 UTC (rev 10024) @@ -931,3 +931,14 @@ text += u'|%s=%s\n' % (item, params[item]) return u'{{%s\n%s}}' % (template, text) + +#---------------------------------- +# Page parsing functionality +#---------------------------------- + +def does_text_contain_section(pagetext, section): + """ Determines whether the page text contains the given + section title. + """ + m = re.search("=+[ ']*%s[ ']*=+" % re.escape(section), pagetext) + return bool(m) Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2012-03-17 18:03:32 UTC (rev 10023) +++ trunk/pywikipedia/wikipedia.py 2012-03-18 17:08:41 UTC (rev 10024) @@ -823,14 +823,13 @@ self._redirarg = redirtarget else: raise IsRedirectPage(redirtarget) - if self.section(): - m = re.search("=+[ ']*%s[ ']*=+" % re.escape(self.section()), - pageInfo['revisions'][0]['*']) - if not m: - try: - self._getexception - except AttributeError: - raise SectionError # Page has no section by this name + + if self.section() and \ + not textlib.does_text_contain_section(pagetext, self.section()): + try: + self._getexception + except AttributeError: + raise SectionError # Page has no section by this name return pagetext def _getEditPageOld(self, get_redirect=False, throttle=True, sysop=False, @@ -965,15 +964,14 @@ self._redirarg = redirtarget else: raise IsRedirectPage(redirtarget) - if self.section(): - m = re.search("=+[ ']*%s[ ']*=+" % re.escape(self.section()), - text) - if not m: - try: - self._getexception - except AttributeError: - raise SectionError # Page has no section by this name + if self.section() and \ + not textlib.does_text_contain_section(text, self.section()): + try: + self._getexception + except AttributeError: + raise SectionError # Page has no section by this name + return pagetext def getOldVersion(self, oldid, force=False, get_redirect=False,

1 0

SVN: [10023] trunk/pywikipedia/splitwarning.py
by xqt＠svn.wikimedia.org 17 Mar '12

17 Mar '12

http://www.mediawiki.org/wiki/Special:Code/pywikipedia/10023 Revision: 10023 Author: xqt Date: 2012-03-17 18:03:32 +0000 (Sat, 17 Mar 2012) Log Message: ----------- done by previous commit Modified Paths: -------------- trunk/pywikipedia/splitwarning.py Modified: trunk/pywikipedia/splitwarning.py =================================================================== --- trunk/pywikipedia/splitwarning.py 2012-03-17 17:54:57 UTC (rev 10022) +++ trunk/pywikipedia/splitwarning.py 2012-03-17 18:03:32 UTC (rev 10023) @@ -22,8 +22,6 @@ def splitwarning(folder): files={} count={} - - # TODO: Variable log filename fn = pywikibot.config.datafilepath("logs", "interwiki.log") logFile = codecs.open(fn, 'r', 'utf-8') rWarning = re.compile('WARNING: (?P<family>.+?): \[\[(?P<code>.+?):.*')

1 0

← Newer
1
2
3
4
5
6
7
8
...
13
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Pywikipedia-svn March 2012