[Pywikipedia-l] match and list, but not replace

List overview All Threads
Download

newer

older

[Pywikipedia-l] Yet another idea...

[Pywikipedia-l] XMLreader.py

Chris Watkins

1 Apr 2010 1 Apr '10

6:05 a.m.

I want to generate a list of matches for a search, but not do anything to the page. E.g. I want to list all pages that contain "redirect[[:Category", but I don't want to modify the pages. I guess that it's possible to modify redirect.py (I don't speak python, but it shouldn't be hard) and run it with -log. But maybe there's a simpler way? Thanks in advance. -- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Attachments:

attachment.htm (text/html — 896 bytes)

Show replies by date

Daniel Mietchen

1 Apr 1 Apr

11:25 a.m.

Perhaps http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py will do the trick, or simply http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py in -debug mode? Daniel On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

...

-- http://www.google.com/profiles/daniel.mietchen

Chris Watkins

2:26 p.m.

Thanks Daniel... I'm confused though. On Thu, Apr 1, 2010 at 20:25, Daniel Mietchen < daniel.mietchen(a)googlemail.com> wrote:

...

Perhaps http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py will do the trick,

I can't see how to use it for matching a specific string.

...

or simply http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py in -debug mode?

Where can I find information on -debug mode? I see there is -verbose mode which "may be helpful when debugging", but I don't see how that helps. I may be missing something obvious &-) Chris

...

Daniel On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

but

it shouldn't be hard) and run it with -log. But maybe there's a simpler

way?

Thanks in advance. -- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- http://www.google.com/profiles/daniel.mietchen _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Daniel Mietchen

3:12 p.m.

Hi Chris, On Thu, Apr 1, 2010 at 2:26 PM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

...

Thanks Daniel... I'm confused though. On Thu, Apr 1, 2010 at 20:25, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

Perhaps http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py will do the trick,

I can't see how to use it for matching a specific string.

Nor do I - sorry. What I had in mind was to apply it to a page that contains your search string, and to restrict the search for "copyright violations" to your site. But this may indeed be a dead end.

...

or simply http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py in -debug mode?

Where can I find information on -debug mode? I see there is -verbose mode which "may be helpful when debugging", but I don't see how that helps.

I thought that most PWB scripts had it, but apparently replace.py does not. but if the def __init__(self, reader, force, append, summary, minor, autosummary, debug): line contains "debug" (as in the example above, taken from http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikipedia/pagefromfile.… ), then -debug is an option with which the script can be run such that it performs all its actions except editing the pages. I am not very experienced with Python or PWB either, but since nobody had replied so far, I wrote out my ideas as they came to mind. Sorry for the confusion, Daniel

...

I may be missing something obvious &-)

Me too.

...

Chris

Daniel On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- http://www.google.com/profiles/daniel.mietchen

Chris Watkins

12 Apr 12 Apr

9:54 a.m.

So I haven't found a way to make a list of matches without replacing. I suspect there's a very simple way, or it would take very simple changes to replace.py. I tried editing replace.py myself, to make it do everything except replace the files. Then I could hack the log files to get the list I want. But I had no success - I'm not coder, so it was guesswork. I copied replace.py to a new file intended to do everything except put files, and called it *replacenoput.py* (i.e. "replace," but no "put") My first attempt was to remove this section (commented it out first, but then removed to be sure): if self.acceptall and new_text != original_text: try: page.put(new_text, self.editSummary) except wikipedia.EditConflict: wikipedia.output(u'Skipping %s because of edit conflict' % (page.title(),)) except wikipedia.SpamfilterError, e: wikipedia.output( u'Cannot change %s because of blacklist entry %s' % (page.title(), e.url)) except wikipedia.PageNotSaved, error: wikipedia.output(u'Error putting page: %s' % (error.args,)) except wikipedia.LockedPage: wikipedia.output(u'Skipping %s (locked page)' % (page.title(),)) Fail - it made the changes all the same. Then I figured out that wikipedia.py was being used to put the files. So I copied that to a new file *wikipedianoput.py* and changed every wikipedia reference in *replacenoput.py* to wikipedianoput. Then I scanned through wikipedianoput.py looking for what I need to block... but I couldn't tell. Can anyone help? Or even better, is there a more elegant way? Thanks Chris On Fri, Apr 2, 2010 at 00:12, Daniel Mietchen < daniel.mietchen(a)googlemail.com> wrote:

...

Hi Chris, On Thu, Apr 1, 2010 at 2:26 PM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

Thanks Daniel... I'm confused though. On Thu, Apr 1, 2010 at 20:25, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

Perhaps http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py will do the trick,

I can't see how to use it for matching a specific string.

Nor do I - sorry. What I had in mind was to apply it to a page that contains your search string, and to restrict the search for "copyright violations" to your site. But this may indeed be a dead end.

or simply http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py in -debug mode?

Where can I find information on -debug mode? I see there is -verbose mode which "may be helpful when debugging", but I don't see how that helps.

I may be missing something obvious &-)

Me too.

Chris > > Daniel > > On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins > <chriswaterguy(a)appropedia.org> wrote: > > I want to generate a list of matches for a search, but not do anything > > to > > the page. > > > > E.g. I want to list all pages that contain "redirect[[:Category", but

> > don't want to modify the pages. > > > > I guess that it's possible to modify redirect.py (I don't speak

python,

> > but > > it shouldn't be hard) and run it with -log. But maybe there's a

simpler

way? Thanks in advance. -- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Merlijn van Deen

10:05 a.m.

Searching by using a text dump sounds more reasonable to me. If you insist on changing replace.py, make sure you are removing all occurences of both put and put_async. Best regards, Merlijn 'valhallasw' van Deen On 12 April 2010 09:54, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

...

Hi Chris, On Thu, Apr 1, 2010 at 2:26 PM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

Thanks Daniel... I'm confused though. On Thu, Apr 1, 2010 at 20:25, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

Perhaps http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py will do the trick,

I can't see how to use it for matching a specific string.

Nor do I - sorry. What I had in mind was to apply it to a page that contains your search string, and to restrict the search for "copyright violations" to your site. But this may indeed be a dead end.

or simply http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py in -debug mode?

Where can I find information on -debug mode? I see there is -verbose

mode

which "may be helpful when debugging", but I don't see how that helps.

I may be missing something obvious &-)

Me too.

Chris > > Daniel > > On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins > <chriswaterguy(a)appropedia.org> wrote: > > I want to generate a list of matches for a search, but not do

anything

> > to > > the page. > > > > E.g. I want to list all pages that contain "redirect[[:Category", but

> > don't want to modify the pages. > > > > I guess that it's possible to modify redirect.py (I don't speak

python,

> > but > > it shouldn't be hard) and run it with -log. But maybe there's a

simpler

> way? > > Thanks in advance. > > -- > Chris Watkins > > Appropedia.org - Sharing knowledge to build rich, sustainable lives. > > blogs.appropedia.org > community.livejournal.com/appropedia > identi.ca/appropedia > twitter.com/appropedia > > _______________________________________________ > Pywikipedia-l mailing list > Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l > > -- http://www.google.com/profiles/daniel.mietchen _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

Chris Watkins

3:32 p.m.

On Mon, Apr 12, 2010 at 18:05, Merlijn van Deen <valhallasw(a)arctus.nl>wrote;wrote:

...

Searching by using a text dump sounds more reasonable to me.

How would you do this? E.g. I want to create a list all pages with tables - i.e. with the string "{|". The MediaWiki search won't do this, but I assume it's possible with a site dump. But I don't know the command to use. Thanks If you insist on changing replace.py, make sure you are removing all

...

occurences of both put and put_async. Best regards, Merlijn 'valhallasw' van Deen On 12 April 2010 09:54, Chris Watkins <chriswaterguy(a)appropedia.org>wrote;wrote:

Hi Chris, On Thu, Apr 1, 2010 at 2:26 PM, Chris Watkins <chriswaterguy(a)appropedia.org> wrote:

Thanks Daniel... I'm confused though. On Thu, Apr 1, 2010 at 20:25, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote: > > Perhaps > http://meta.wikimedia.org/wiki/Pywikipediabot/copyright.py > will do the trick, I can't see how to use it for matching a specific string.

Nor do I - sorry. What I had in mind was to apply it to a page that contains your search string, and to restrict the search for "copyright violations" to your site. But this may indeed be a dead end.

> or simply > http://meta.wikimedia.org/wiki/Pywikipediabot/replace.py > in -debug mode? Where can I find information on -debug mode? I see there is -verbose

mode

which "may be helpful when debugging", but I don't see how that helps.

I may be missing something obvious &-)

Me too.

Chris > > Daniel > > On Thu, Apr 1, 2010 at 6:05 AM, Chris Watkins > <chriswaterguy(a)appropedia.org> wrote: > > I want to generate a list of matches for a search, but not do

anything

> > to > > the page. > > > > E.g. I want to list all pages that contain "redirect[[:Category",

but I

> > don't want to modify the pages. > > > > I guess that it's possible to modify redirect.py (I don't speak

python,

> > but > > it shouldn't be hard) and run it with -log. But maybe there's a

simpler

> > way? > > > > Thanks in advance. > > > > -- > > Chris Watkins > > > > Appropedia.org - Sharing knowledge to build rich, sustainable lives. > > > > blogs.appropedia.org > > community.livejournal.com/appropedia > > identi.ca/appropedia > > twitter.com/appropedia > > > > _______________________________________________ > > Pywikipedia-l mailing list > > Pywikipedia-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l > > > > > > > > -- > http://www.google.com/profiles/daniel.mietchen > > _______________________________________________ > Pywikipedia-l mailing list > Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l -- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

_______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Nicolas Dumazet

10:31 a.m.

Hello, 2010/4/1 Chris Watkins <chriswaterguy(a)appropedia.org>rg>:

...

I want to generate a list of matches for a search, but not do anything to the page. E.g. I want to list all pages that contain "redirect[[:Category", but I don't want to modify the pages.

List pages? That's a job for pagegenerators.py python pagegenerators.py -search[.....] But honestly Chris, as I wrote before, you should try to learn Python. It's not this hard :) The time required to write a 3 lines long code snippet yielding all pages matching a Mediawiki search (2 minutes) should be about 500 times shorter than the average latency of this mailing list: from pagegenerators import SearchPageGenerator for page in SearchPageGenerator("my query"): print page I haven't been coding with pywikipedia for months, this code is provided without any guarantee. But you get the simplicity of the operation I guess :) Best regards, Nicolas.

...

I guess that it's possible to modify redirect.py (I don't speak python, but it shouldn't be hard) and run it with -log. But maybe there's a simpler way? Thanks in advance. -- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Nicolas Dumazet — NicDumZ

Chris Watkins

3:15 p.m.

Thanks Nicolas, On Mon, Apr 12, 2010 at 18:31, Nicolas Dumazet <nicdumz(a)gmail.com> wrote:

...

Hello, 2010/4/1 Chris Watkins <chriswaterguy(a)appropedia.org>rg>:

I want to generate a list of matches for a search, but not do anything to the page. E.g. I want to list all pages that contain "redirect[[:Category", but I don't want to modify the pages.

List pages? That's a job for pagegenerators.py python pagegenerators.py -search[.....]

Thanks - this is strange, I initially looked at this, and then somehow got the impression that it did something quite different. And I couldn't find any documentation at all for pagegenerators.py - but I've now created a page on the wiki. Now, this will be very useful, but one of my tasks won't work with this command. I want to list all pages with tables - i.e. look for "{|". The MediaWiki search won't do this, so I'll ask Merlijn about his text dump idea... But honestly Chris, as I wrote before, you should try to learn Python.

...

It's not this hard :) The time required to write a 3 lines long code snippet yielding all pages matching a Mediawiki search (2 minutes) should be about 500 times shorter than the average latency of this mailing list:

I did look at python, and I think you greatly underestimate how much it would take me, with almost zero coding experience, to learn a useful amount of python. I also have a lot of patience in waiting for answers. And I hope that we'll have more tech volunteers working on Appropedia, by later this year. Then I can do other things like community management and looking for funding, while they do things with Python & MediaWiki. For now, my main contribution to the Pywikipediabot project is to help out with documentation where I can... Anyway, I appreciate your help! Chris from pagegenerators import SearchPageGenerator

...

for page in SearchPageGenerator("my query"): print page I haven't been coding with pywikipedia for months, this code is provided without any guarantee. But you get the simplicity of the operation I guess :) Best regards, Nicolas.

I guess that it's possible to modify redirect.py (I don't speak python,

but

it shouldn't be hard) and run it with -log. But maybe there's a simpler

way?

-- Nicolas Dumazet — NicDumZ _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Bináris

1 May 1 May

5:36 p.m.

2010/4/1 Chris Watkins <chriswaterguy(a)appropedia.org>

...

I want to generate a list of matches for a search, but not do anything to the page. I tell you the trick. This is not my own wisdom, I have learnt it from

hu:user:Pasztilla. For example, you have a user:MyBot/try subpage. You copy replace.py to replace2.py, and then modify as itt follows: 1. Look for " def run(self):" line, and write under it: list ='' This will start a new list. Alternatively, list = wikipedia.Page(wikipedia.getSite(), 'User:MyBot/try').get() will continue your existing list. 2. Under " if choice == 'y':" look for page.put_async(new_text, self.editSummary) and comment it out: #page.put_async(new_text, self.editSummary) and write just below this line: list+= u'# [[%s]]\n' %page.title() listsite = wikipedia.Page(wikipedia.getSite(), 'User:MyBot/try') listsite.put(list, self.editSummary) This will write the title of the page to your subpage. 3. Under the " if self.acceptall and new_text != original_text:" line make the same changes as in step 2. Enjoy! -- Bináris

Chris Watkins

8 Jun 8 Jun

10 p.m.

...

2010/4/1 Chris Watkins <chriswaterguy(a)appropedia.org> I want to generate a list of matches for a search, but not do anything to

the page. I tell you the trick. This is not my own wisdom, I have learnt it from

-- Chris Watkins Appropedia.org - Sharing knowledge to build rich, sustainable lives. blogs.appropedia.org community.livejournal.com/appropedia identi.ca/appropedia twitter.com/appropedia

Bináris

7 Nov 7 Nov

7:59 a.m.

Sorry for being late with the answer. At that time I already had the solution in my mind that was accepted yesterday. So replace.py r8700 with the new option -save sould be the perfect answer for your needs. And no more individual fiddling. :-)) Bináris 2010/6/8 Chris Watkins <chriswaterguy(a)appropedia.org>

...

Hi Bináris, Thanks - this looks very promising. I wonder if you could post a copy of your replace2.py, so I can follow the instructions more clearly - I'm a bit confused by a couple of points. Thanks again! On Sat, May 1, 2010 at 22:36, Bináris <wikiposta(a)gmail.com> wrote: > 2010/4/1 Chris Watkins <chriswaterguy(a)appropedia.org> > > I want to generate a list of matches for a search, but not do anything to >> the page. >> >>

4912

days inactive

5132

days old

pywikibot@lists.wikimedia.org

Manage subscription

11 comments

5 participants

tags (0)

participants (5)

Bináris
Chris Watkins
Daniel Mietchen
Merlijn van Deen
Nicolas Dumazet