hi, thank you. Where can I find documentation for an example to extract links https://github.com/earwig/mwparserfromhell or https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/xmlreader… ? I'd be very grateful if you can point me to an example for links extraction and redirect. Shall I use them against the xml dump or as bot to api.wikimedia? I would like to use offline, but mwparserfromhell seems to use online against api.wikipedia.. where are documentation of scripts in mediawiki.org? https://www.mediawiki.org/w/index.php?search=xmlparser&title=Special%3A… thank you! On Mon, Jan 18, 2016 at 8:05 PM, Morten Wang <nettrom(a)gmail.com> wrote:

On Tue, Jan 19, 2016 at 4:13 PM, Aaron Halfaker <aaron.halfaker(a)gmail.com> wrote: Well, other languages have different namespace names..., so that script is English only. -- John Vandenberg

On Wed, Jan 20, 2016 at 1:06 AM, Aaron Halfaker <aaron.halfaker(a)gmail.com> wrote: Your walk through uses Dutch Wikipedia as its example, and currently it does not support Dutch Wikipedia. Would you please fix that. -- John Vandenberg

Hi All, and thank your suggestions. I will have a look to all them, I started from 'mwparserfromhell' - helluwa of a name ! I found my way with: import mwparserfromhellAPI_URL = "https://en.wikipedia.org/w/api.php" def parse(title): data = {"action": "query", "prop": "revisions", "rvlimit": 1, "rvprop": "content", "format": "json", "titles": title} raw = urlopen(API_URL, urlencode(data).encode()).read() res = json.loads(raw) text = res["query"]["pages"].values()[0]["revisions"][0]["*"] return mwparserfromhell.parse(text) test = parse('DNA') # and test.filter_wikilinks() Some links are like [[Gunther Stent|Stent, Gunther Siegmund]] So with '|' in the middle. The first is the canonical form, but what does the second token represent? e.g. I try: *parse('Stent, Gunther Siegmund')* Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 6, in parse KeyError: 'revisions' What does this error mean? Is it a redirect? A few more questions about these tools: 1. Where can I find a documentation about the use of methods in mwparserfromhell? e.g. wikilinks() method takes argumetns, which can I use? Could not find much here: http://mwparserfromhell.readthedocs.org/en/latest/api/mwparserfromhell.html… 2. I would like to use with a generator from dump: I understand this module would to do the job to fetch pages and pipe to mwh [aka mwparserfromhell :)] https://tools.wmflabs.org/paws/public/EpochFail/examples/mwxml.py.ipynb Correct ? Any doc around for this as well? 3. how to handle redirect AND/OR curid ? as example, dbpedia analysed recursive redirect, they call it "transitiive redirect". Any module to handle transitive redirect / to handle redirect (I would do it recursively) E.g. if I use example above: *parse('dna')* u'#REDIRECT [[DNA]] {{R from other capitalisation}}' I would like to obtain already 'DNA', or even better a module returning the _ID of page (so far I build it myself with a dictionary, I'd like to ask to MW team if they already could suggest some tool to handle recursive redirect more efficiently). On Wed, Jan 20, 2016 at 8:27 AM, Morten Wang &lt;nettrom(a)gmail.com&gt; wrote: