On 09/20/2012 07:40 PM, MZMcBride wrote:
Scanning dumps (or really dealing with them in any form) is pretty awful. There's been some brainstorming in the past for how to set up a system where users (or operators) could run arbitrary regular expressions on all of the current wikitext regularly, but such a setup requires _a lot_ of anything involved (disk space, RAM, bandwidth, processing power, etc.). Maybe one day Labs will have something like this.
We have a dump grepper tool in the Parsoid codebase (see js/tests/dumpGrepper.js) that takes about 25 minutes to grep an XML dump of the English Wikipedia. The memory involved is minimal and constant, the thing is mostly CPU-bound.
It should not be hard to hook this up to a web service. Our parser web service in js/api could serve as a template for that.
Gabriel