How many links did TWL account recipients add to Wikipedia with their access? - Wiki-research-l

Aaron Halfaker

13 Jan 13 Jan

11:23 p.m.

New subject: How many links did TWL account recipients add to Wikipedia with their access?

I don't think that you can do this with quarry since you'll need to parse wiki content in order to extract external links. I don't think they are stored in a table anywhere. However, I think that we can do it fairly easily with a process on the XML dumps. If you can give me the list of editors and partner websites, I could put a script together and talk to you about the bits. -Aaron On Tue, Jan 13, 2015 at 5:01 PM, Jake Orlowitz <jorlowitz(a)gmail.com> wrote:

...

Hi all, There are 2000 editors who have received access to 20 different online databases. We know the usernames of these editors and the url prefixes of the websites they were given access to. We need to know: - from July 18th 2014 to January 11th 2014 - on English Wikipedia - for the cohort of 2000 TWL editors - ...how many times did they add links to any of the 20 partner websites I have my fingers crossed that Quarry can solve this but I need some help to write a query. Bonus queries: 1) In that date range, how many links did these editors add using partner websites on *all Wikipedias* (any language) 2) What is the baseline change in all external links on English Wikipedia in that date range 3) What is the baseline change in all external links on *all Wikipedias* since July 18th 2014 Thanks so much for any guidance on this! Jake (Ocaasi) The Wikipedia Library _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Reply

Oliver Keyes

8 a.m.

New subject: How many links did TWL account recipients add to Wikipedia with their access?

Actually, the API structure has a "grab all of the external links from [page]" query: I'm not sure if it can be applied to historical revisions, but we can see! On Wed, Jan 14, 2015 at 2:23 AM, Gerard Meijssen <gerard.meijssen(a)gmail.com> wrote:

...

Hoi, These same people may have added content to Wikidata ... Obviously it has not been considered. However, you can query for these people there. You can also query how many external references were added by bot. It may provide the groundwork going to Wikipedias and find who did it .. references to external sources are added all the time. Thanks, GerardM On 14 January 2015 at 00:23, Aaron Halfaker <ahalfaker(a)wikimedia.org> wrote:

I don't think that you can do this with quarry since you'll need to parse wiki content in order to extract external links. I don't think they are stored in a table anywhere. However, I think that we can do it fairly easily with a process on the XML dumps. If you can give me the list of editors and partner websites, I could put a script together and talk to you about the bits. -Aaron On Tue, Jan 13, 2015 at 5:01 PM, Jake Orlowitz <jorlowitz(a)gmail.com> wrote:

Hi all, There are 2000 editors who have received access to 20 different online databases. We know the usernames of these editors and the url prefixes of the websites they were given access to. We need to know: - from July 18th 2014 to January 11th 2014 - on English Wikipedia - for the cohort of 2000 TWL editors - ...how many times did they add links to any of the 20 partner websites I have my fingers crossed that Quarry can solve this but I need some help to write a query. Bonus queries: 1) In that date range, how many links did these editors add using partner websites on *all Wikipedias* (any language) 2) What is the baseline change in all external links on English Wikipedia in that date range 3) What is the baseline change in all external links on *all Wikipedias* since July 18th 2014 Thanks so much for any guidance on this! Jake (Ocaasi) The Wikipedia Library _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Reply

mjn

1:32 p.m.

New subject: How many links did TWL account recipients add to Wikipedia with their access?

Aaron Halfaker <ahalfaker(a)wikimedia.org> writes:

...

...you'll need to parse wiki content in order to extract external links. I don't think they are stored in a table anywhere.

The links themselves are, but it isn't tied to editor information, so I don't think will answer this particular query. In the database dumps at dumps.wikimedia.org, the table that's dumped as xxwiki-yyyymmdd-externallinks.sql.gz lists external links per-page. So if you just wanted counts of link additions (or removals), you could grab two dumps from different dates and compare. But you'll need to parse the full revision information to get a count of who added which links. -Mark -- mjn | http://www.anadrome.org

Reply

Andrew Gray

15 Jan 15 Jan

8:52 a.m.

New subject: How many links did TWL account recipients add to Wikipedia with their access?

One approach might be to get a list of all pages/links added using Mark's method, then for each of the sets do a manual sample of a few percent of the new links and see who added them - this would let you know if you're looking at a situation where "almost all links to XYZ were added by TWL users" or "only some links were added by non-TWL users", and estimate accordingly. (I suspect some will definitely be in the first batch) Andrew. On 14 January 2015 at 13:32, mjn <mjn(a)anadrome.org> wrote:

...

Aaron Halfaker <ahalfaker(a)wikimedia.org> writes:

...you'll need to parse wiki content in order to extract external links. I don't think they are stored in a table anywhere.

The links themselves are, but it isn't tied to editor information, so I don't think will answer this particular query. In the database dumps at dumps.wikimedia.org, the table that's dumped as xxwiki-yyyymmdd-externallinks.sql.gz lists external links per-page. So if you just wanted counts of link additions (or removals), you could grab two dumps from different dates and compare. But you'll need to parse the full revision information to get a count of who added which links. -Mark -- mjn | http://www.anadrome.org _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

-- - Andrew Gray andrew.gray(a)dunelm.org.uk

Reply