Robert Ullmann wrote:
I've been spending hours on the parsing now and don't find it simple at all due to the fact that templates can be nested. Just extracting the Infobox as one big lump is hard due to the need to match nested {{ and }}
Andrew Dunbar (hippietrail)
Hi,
Come now, you are over-thinking it. Find "{{Infobox [Ll]anguage" in the text, then count braces. Start at depth=2, count up and down 'till you reach 0, and you are at the end of the template. (you can be picky about only counting them if paired if you like ;-)
Then just regex match the lines/parameters you want.
However, if you are pulling the wikitext with the API, the XML parse tree option sounds good; then you can just use elementTree (or the like) and pull out the parameters directly
Robert
Or you could use the pyparsing Python library, with which you can implement the grammar of your choice, making matching nested template extraction trivial. Using the psyco package to accelerate it, you can parse a whole en: dump in a few hours.
See the code below for a sample grammar...
-- Neil
------------------------------------------------
# Use pyparsing, enablePackrat() _and_ psyco for a considerable speed-up from pyparsing import * import psyco # These two must be in the correct order, or bad things will happen ParserElement.enablePackrat() psyco.full()
wikitemplate = Forward()
wikilink = Combine("[[" + SkipTo("]]") + "]]")
wikiargname = CharsNotIn("|{}=") wikiargval = ZeroOrMore( wikilink | Group(wikitemplate) | CharsNotIn("[|{}") | "[" | "{" | Regex("}[^}]"))
wikiarg = Group(Optional(wikiargname + Suppress("="), default="??") + wikiargval)
wikitemplate << (Suppress("{{") + wikiargname + Optional(Suppress("|") + delimitedList(wikiarg, "|")) + Suppress("}}"))
wikitext = ZeroOrMore(CharsNotIn("{") | Group(wikitemplate) | "{" )
def parse_page(text): return wikitext.parseString(text)