Thanks Sidd for responding actively in this thread.
The biggest problem here: the algorithm used in this research were bad. They produced nonsense that wasn't remotely grammatical. You should have caught most of these problems. (The early version of the bot (for just plays) had a poor success rate as well, but it seemed plausible that a template for tiny play articles could be effectively filled out with automation.)
Two interesting results IMO:
+ A nonsensical article with a decent first sentence & sections, and refs (however random), can serve as encouragement to write a real article. Possibly more of an encouragement than just the first sentence alone. I believe there's some related research into how people respond to cold emails that include mistakes & nonsense. (Surely there's a more effective \ non-offensive way to produce similar results)
+ We could use even a naive measure of the coverage & consistency of new article review. (If it drops below a certain threshhold, we could do something like change the background color & search-engine metadata for pages that haven't been properly reviewed yet)
For future researchers:
If we encourage people to spend more time making tools work – rather than doing something simple (even counterproductive) and writing a paper about it – everyone will benefit. The main namespace is full of bots, both fully automatic and requiring a human to run them. Anyone considering or implementing wiki automation should look at them and talk to the community of bot maintainers.