Re: [Wiki-research-l] [Analytics] [Announcement] Daily Social Media Traffic Report for English Wikipedia articles

Hi all, I wanted to provide some advanced notification that as of May 31st, we no longer plan to support the pilot social media traffic report that we have been running. As a reminder, in response to a request in a Disinformation Meetup during Wikimania 2019 [1], we have been publishing daily reports [2] that list articles on English Wikipedia that received over 500 pageviews the day before from readers coming from one of four external platforms (Youtube, Twitter, Reddit, Facebook) [3]. As of May 31st, we (the WMF Research team [4]) will no longer maintain the job that produces these daily reports. Factors in our decision: We ran some basic analyses [5] based on the first two months of data (3324 unique articles) but do not see strong evidence that this data has had a clearly valuable impact on patrolling: * We saw no significant change in the number of edits: between the two weeks prior to an article being published to the report (10.7; 99% confidence interval: [8.3-13.7]) and the two weeks following (10.8; 99% confidence interval: [8.5-13.3]). * We saw no significant change in the number of reverts: between the two weeks prior to an article being published to the report (0.83 [0.70-0.99]) and the two weeks following (0.83 [0.70-0.98]). * We saw a slight uptick in the amount of page restrictions: 64 (2%) of the articles saw an increase in protections while the rest remained the same (we did not verify whether this is "normal" for similar articles over the time range we are studying). * We saw steady but low traffic to the report: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=… * We are aware that a large number of users -- even outside of this listserv -- received notice of the report: https://en.wikipedia.org/w/index.php?title=Special:WhatLinksHere/User:HostB… * We received some good feedback early on in the pilot that we have captured: https://meta.wikimedia.org/wiki/Research_talk:Social_media_traffic_report_p… We have seen a number of positive things come out of this work: * The above analyses suggest that the organic traffic coming from external platforms like Youtube and Facebook that link to Wikipedia articles as context for fact-checking is not having a significant deleterious impact on Wikipedia or placing an additional burden on patrollers. * We now have a working process that allows us to deploy reports like this one, which will make it much easier to prototype additional data releases of this sort in the future. * Though the early evidence suggests that the Social Media Traffic Report as an intervention has not led to a substantial change in patrolling behavior around these articles, we now have a public dataset of externally-referred traffic for two months that can support further research into the impact of platforms (and users on those platforms) that link to Wikipedia articles. If you have a compelling reason to extend the pilot, any questions, thoughts, or other feedback, don't hesitate to reach out or post on the project talk page [6]. We remain committed to support you in your important patrolling efforts and will continue to prototype new interventions where we can. [1] https://wikimania.wikimedia.org/wiki/2019:Meetups/Disinformation [2] https://en.wikipedia.org/wiki/User:HostBot/Social_media_traffic_report [3] https://meta.wikimedia.org/wiki/Research:Social_media_traffic_report_pilot/… [4] https://research.wikimedia.org/team.html [5] https://phabricator.wikimedia.org/T241768#6152394 [6] https://meta.wikimedia.org/wiki/Research_talk:Social_media_traffic_report_p… Thanks, Isaac On Mon, Mar 23, 2020 at 2:24 PM Jonathan Morgan <jmorgan(a)wikimedia.org> wrote: -- Isaac Johnson (he/him/his) -- Research Scientist -- Wikimedia Foundation