[Toolserver-l] Query Service Inquiry

[Ilmari Karonen]

On Thu, 2011-04-28 at 21:49 -0600, Jim Hutchinson wrote:
> Thank you for the information and feedback. What I need is somewhat
> more complicated than a list of contributors to a single page. In
> fact, there is already a tool in Wikipedia (just called contributors,
> I think) that lists all the contributors to an article and their
> number of edits. What I need to do is, using that list of
> contributors, select the top 20 or so (excluding bots) for each of the
> hundred selected articles and get a list of all of the other articles
> to which each of them contributed with a frequency count of edits.
> Ideally, this data would be in a table of sorts for each article
> selected (so 100 tables).
> 
> This could, of course, be done manually by searching for contributions
> by username. however, this will be time consuming and possibly error
> prone. My hope was that a query could grab this information fairly
> quickly as well as automatically count frequencies of edits per
> article, etc.
> 
> I don't have the expertise to do this myself, but I do know someone
> who can and has requested an account. However, he is afraid he will
> not be granted an account for what will likely be a one time project.
> 
> Is there likely an API that can do what I described or would a query
> be an easier or more efficient way to go?

Technically, most of this shouldn't be too hard to do using SQL queries
on the toolserver.  One disadvantage, though, is that the toolserver
does not have (direct) access to page text.  This could be a problem if
you, say, wanted to exclude reverts from the edit count, weigh edits by
the amount of text added or do some other kind of fine-grained
processing.

Basically, you have three steps you want to do:

 1. Select 200 random articles.
 2. Get the top contributors for each of them.
 3. Get the edit counts for those contributors.

The first step is easy, as long as the (not quite uniform) random page
selection algorithm built into MediaWiki is good enough for you.  You
could do it using a Toolserver SQL query, or just by clicking the
"random page" link 200 times (by hand or by bot), but the simplest way
would probably be to use the API:
http://www.mediawiki.org/wiki/API:Random

If you wanted a more uniform sample, you could download the page table
SQL dump (page.sql.gz), extract the page titles from it (with
appropriate filtering, e.g. to exclude redirects) and randomly select
200 of them.

The second step could be easily done on the Toolserver, as long as you
only wanted to count edits.  For more fine-grained filtering based on
page text, you could use Special:Export to obtain a "mini-dump" of the
pages in your sample, including their full history, in XML format.
Alternatively, the same information is also available using the API.

(The detail about excluding bots comes down to determining what is a
bot.  MediaWiki does feature a "bot flag", which can be used to filter
out users having it.  Unfortunately, for various reasons, not all bot
accounts necessarily have the flag set.  You might be able to filter out
more bots by looking at, say, the categories on their user page, but
ultimately you may still end up having to do some manual filtering.)

The last step could, again, be fairly easily done on the Toolserver as
long as you only wanted the raw edit counts.  In fact, it would probably
be best to start with that data anyway, and then refine it by looking at
the relevant page histories if necessary.

-- 
Ilmari Karonen