Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

List overview All Threads
Download

newer

older

Thoughts about Wikidata in Rewrite...

Re: [Pywikipedia-l] Script halted,...

legoktm

24 Apr 2013 24 Apr '13

1:11 p.m.

Hi all, I had mentioned this in the rewrite roadmap, and noticed it came up on IRC as well, so I'd like to run this by the mailing list: User:The Earwig has written a pure-python (with optional C-speedups) MediaWiki text parser named mwparserfromhell[1]. Currently we have the textlib library and some various regexes that implement this in a non-perfect way. From my experience using mwparser (over 400k successful edits with no issues) I believe it is ready to be bundled with the framework. I think it would still be a good idea to keep textlib in as a fallback or for users who are currently using it and don't need to migrate. As for actually adding it, in the rewrite branch we can just add it as a dependency in setup.py, and then convert various methods over. In trunk, I'm guessing we would need to add it as an external. (I'm not sure how that's actually done.) [1] https://github.com/earwig/mwparserfromhell -- Legoktm

Attachments:

attachment.htm (text/html — 1.1 KB)

Show replies by date

Bináris

25 Apr 25 Apr

8:17 a.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

Sounds interesting. What are your news for trunk users? -- Bináris

Hazard-SJ

22 May 22 May

3:50 a.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

Yes, that'd be nice. Hazard-SJ ________________________________ From: legoktm <legoktm.wikipedia(a)gmail.com> To: WikiMedia Mailing Lists <pywikipedia-l(a)lists.wikimedia.org> Sent: Wednesday, April 24, 2013 6:11 AM Subject: [Pywikipedia-l] Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods Hi all, I had mentioned this in the rewrite roadmap, and noticed it came up on IRC as well, so I'd like to run this by the mailing list: User:The Earwig has written a pure-python (with optional C-speedups) MediaWiki text parser named mwparserfromhell[1]. Currently we have the textlib library and some various regexes that implement this in a non-perfect way. From my experience using mwparser (over 400k successful edits with no issues) I believe it is ready to be bundled with the framework. I think it would still be a good idea to keep textlib in as a fallback or for users who are currently using it and don't need to migrate. As for actually adding it, in the rewrite branch we can just add it as a dependency in setup.py, and then convert various methods over. In trunk, I'm guessing we would need to add it as an external. (I'm not sure how that's actually done.) [1] https://github.com/earwig/mwparserfromhell -- Legoktm _______________________________________________ Pywikipedia-l mailing list Pywikipedia-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l

legoktm

10 Jul 10 Jul

9:18 a.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

I've gone ahead and implemented this in rewrite with r11737<https://www.mediawiki.org/wiki/Special:Code/pywikipedia/11737>… r11738. It's opt-in, so if you want to use it you need to install the mwparserfromhell <https://github.com/earwig/mwparserfromhell> library yourself, and set "use_mwparserfromhell = True" in your user-config.py. Eventually I'd like to add mwpfh as a requirement in the setup.py and make it an opt-out option. I haven't looked much into how it would be implemented in trunk, however I assume it will require something similar, which I'll do once its fully working in rewrite. If you notice any weird bugs, please let me know so I can fix them in pywikibot or report them upstream :D Thanks, -- Legoktm On Wed, Apr 24, 2013 at 7:11 AM, legoktm <legoktm.wikipedia(a)gmail.com>wrote;wrote:

...

Dr. Trigon

10:16 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24.04.2013 13:11, legoktm wrote:

...

As for actually adding it, in the rewrite branch we can just add it as a dependency in setup.py, and then convert various methods over. In trunk, I'm guessing we would need to add it as an external. (I'm not sure how that's actually done.)

I would be happy to support you with this in trunk (and rewrite). The system I implemented in trunk (and I am currently trying to promote... ;) could easily take the link [1] and make sure that this dependency gets installed automatically at the first bot run. [1] https://github.com/earwig/mwparserfromhell/archive/develop.zip Greetings DrTrigon

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHdwQYACgkQAXWvBxzBrDAZtwCdFs51BpaC/zkYynh4rtmh5lBT YPoAoOBQw92rlPxOL28os/hWYw5Yf1ZH =1V+L -----END PGP SIGNATURE-----

Ben Kurtovic

10:42 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

...

[1] https://github.com/earwig/mwparserfromhell/archive/develop.zip

At this stage, it might be best to use stable releases [1] instead of the development version, so you'd want [2] (or [3], which will always point to the latest version). Also, as a note, the parser only supports Python 2.7 and 3, not Python 2.6. Is it worth extending compatibility to 2.6? [1] https://github.com/earwig/mwparserfromhell/releases [2] https://github.com/earwig/mwparserfromhell/archive/v0.2.zip [2] https://github.com/earwig/mwparserfromhell/archive/master.zip Earwig

Dr. Trigon

11 Jul 11 Jul

10:20 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Yes of course! Makes a lot more sense to use a fixed stable revision. I implemented this in trunk in r11744 [1]. The version used is v0.2.zip. [1] http://www.mediawiki.org/wiki/Special:Code/pywikipedia/11744 Now the following works:

...

>> import wikipedia import externals

(here you will get asked whether you want to download 'mwparserfromhell' once)

...

>> import mwparserfromhell

it can be easily tested with: python -c 'import wikipedia;import externals;import mwparserfromhell' Greetings and all the best! DrTrigon On 10.07.2013 22:42, Ben Kurtovic wrote:

...

[1] https://github.com/earwig/mwparserfromhell/archive/develop.zip

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHfE5gACgkQAXWvBxzBrDAm+wCg4J2lueNLUXtXwDbs+q2iwmi8 RHEAn3yKJzCo2mthQseObyY1pC6yUXm2 =ieds -----END PGP SIGNATURE-----

Dr. Trigon

11:37 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 As I am known to be a fool (sometimes ;) I have to correct myself here:

...

>> import wikipedia import externals >> externals.check_setup('mwparserfromhell') # here you will get

asked whether you want to download 'mwparserfromhell' once

...

>> import mwparserfromhell

it can be easily tested with: $ python -c 'import wikipedia;import externals;externals.check_setup("mwparserfromhell");import mwparserfromhell' Sorry for the inconvenience and enjoy! Greetings DrTrigon On 11.07.2013 22:20, Dr. Trigon wrote: > Yes of course! Makes a lot more sense to use a fixed stable > revision. > > I implemented this in trunk in r11744 [1]. The version used is > v0.2.zip. > > [1] http://www.mediawiki.org/wiki/Special:Code/pywikipedia/11744 > > Now the following works: > >>>> import wikipedia import externals > (here you will get asked whether you want to download > 'mwparserfromhell' once) >

...

>> import mwparserfromhell

> > it can be easily tested with: > > python -c 'import wikipedia;import externals;import > mwparserfromhell' > > Greetings and all the best! DrTrigon > > > On 10.07.2013 22:42, Ben Kurtovic wrote: >>> [1] >>> https://github.com/earwig/mwparserfromhell/archive/develop.zip > > >> At this stage, it might be best to use stable releases [1] >> instead of the development version, so you'd want [2] (or [3], >> which will always point to the latest version). > >> Also, as a note, the parser only supports Python 2.7 and 3, not >> Python 2.6. Is it worth extending compatibility to 2.6? > >> [1] https://github.com/earwig/mwparserfromhell/releases [2] >> https://github.com/earwig/mwparserfromhell/archive/v0.2.zip [2] >> https://github.com/earwig/mwparserfromhell/archive/master.zip > >> Earwig > > > >> _______________________________________________ Pywikipedia-l >> mailing list Pywikipedia-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l > > > > _______________________________________________ Pywikipedia-l > mailing list Pywikipedia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l >

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHfJZsACgkQAXWvBxzBrDCCowCdGofn/XW6I5mZ1uYCqpVPkxcm pVgAn3bNqnhJjbvacJkllNjcrGosXvhY =0cQK -----END PGP SIGNATURE-----

Dr. Trigon

14 Jul 14 Jul

12:48 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 One important question for me here is "How is the handling/behaviour for malform(at)ed wiki syntax, like e.g. a text body: ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### === bad header 1 == A text containing mathematical equations like a = b + c or even something that could look like a header, like a == b == c but is anything BUT a header. == bad header 2 === Some lorem ipsum bla bla ... ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### I am despirately seeking a parser that has the same error behaviour and gives the same results like the original mw parser also in case of malform(at)ed wiki text. Is this the case for mwparserfromhell?? Thanks and Greetings DrTrigon On 24.04.2013 13:11, legoktm wrote:

...

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHigfUACgkQAXWvBxzBrDDGHgCfa+FV2kt8NnJMCg2gv8NGxbqU txEAoL/TyTHKZIgLCOL52qiB0NyeVDbm =GAQ4 -----END PGP SIGNATURE-----

Ben Kurtovic

8:58 p.m.

New subject: Using a true MediaWiki parser (mwparserfromhell) instead of textlib methods

On Jul 14, 2013, at 6:48 AM, Dr. Trigon wrote:

...

One important question for me here is "How is the handling/behaviour for malform(at)ed wiki syntax [...] I am despirately seeking a parser that has the same error behaviour and gives the same results like the original mw parser also in case of malform(at)ed wiki text. Is this the case for mwparserfromhell??

Why don't you test it? One of my main intentions designing the parser was to have it handle malformed wikitext faithfully. I think I've done a pretty good job, and the example you gave is parsed correctly. I mean, that's partly the reason why we're using a non-regex-based parser in the first place: to avoid regex's limitations. Do point out situations where it makes mistakes, though, so they can be fixed. Earwig

3937

days inactive

4018

days old

pywikipedia-l@lists.wikimedia.org

Manage subscription

9 comments

5 participants

tags (0)

participants (5)

Ben Kurtovic
Bináris
Dr. Trigon
Hazard-SJ
legoktm