I think I brought this up in another post about developing a methodology to determine this based on a number of offline conversations I was having. As this might be of interest to people doing research on women's involvement for Wikipedia, I'm crossposting this here from my blog. The entry can be found at http://ozziesport.com/2012/06/measuring-the-influence-of-wikipedia-on-public.... It isn't necessarily the one I will ultimately use when I set out to do this, but working to set up some of the tools to allow measuring for this in the near future. If anyone has any thoughts on how to better measure the influence of Wikipedia for public thought formation, especially as it pertains to women's issues, that would be much appreciated.

* * *Measuring the influence of Wikipedia on public thinking (in Australian women’s sport): A proposed methodology* This entry was posted by Laura on Monday, 18 June, 2012

I spend a lot of time thinking about Wikipedia and talking to people in Australia’s sport sector about Wikipedia, Wikinews and Commons. Some of those I have talked to tend to agree that Wikipedia has value, potentially more value than a news story from a traditional outlet where the story quickly disappear. There is still a question of: “Why does Wikipedia matter?” The most obvious answer is “Page views demonstrate meaning. People are going to Wikipedia for information about sport, Australian sport, and women’s sport. The page views clearly demonstrate that Wikipedia matters and is worth contributing to as an organisation.” This argument doesn’t always work and I’ve been challenged to demonstrate Wikipedia’s influence on the topic of Australian women’s sport.

This is in some ways a frustrating endeavor. How do you measure Wikipedia’s influence beyond page views? As a person who loves research design, I have a number of ideas but how to implement and analyze information is still something I am struggling with because I keep coming back to the need to possibly use a qualitative approach reliant on survey research… and that always makes me nervous. Still, with this in mind, the following methodology is one I have been leaning towards trying to write up more formally.

* Benchmark English Wikipedia, English Wikinews and Commons coverage of Australian women’s sport:*

Develop a list of all existing articles pertaining to Australian women’s sport on English Wikipedia. The list would include biographies, sport teams, leagues, organisations and people connected to administering women’s sport, competitions, articles about women’s sport in the country, articles about sport in the country, general sport articles that broadly intersect with Australian women’s sport such as women’s basketball and the Olympic games. Once the list is created, benchmark the following for each article: Determine the existing size of the article. Identify projects the article is part of. Identify current article assessment. Date article was created and who created it. Determine the number of contributors to the article. Determine the gender ratio for article contributors. Determine the geographic location of article contributors. Determine the historic page views for the article. Determine if the article has appeared at Did you Know, and average traffic before and after it appeared. Determine if there is a spoken word version of the article. Count the number of pictures on each article. Develop a list of all existing news stories about Australian women’s sport on English Wikinews. Determine the historical traffic to these articles. Identify the categories these articles are included in. Identify if the article is synthesis or original research. Identify all Wikipedia articles the story is linked on. Develop a list of all photographs pertaining to Australian women’s sport on Commons. Develop a list what articles these images are used on across Wikimedia Foundation projects. Develop a list of contributors for images in this space.

This data will provide a framework for understanding the story of Australian women’s sport coverage on Wikipedia, and will enable case studies to be developed around any potential editing efforts affiliated with a study. This data is largely background, which can help to contextualise data around Wikipedia as it pertains to influence thought formation.

*Benchmark interest in English Wikipedia, English Wikinews and Commons coverage of Australian women’s sport:*

The next step is possibly the more difficult one: How do we contextualise Wikipedia articles to understand where they sit as resources people turn to for information? Below are quantitative, web based measures to try to determine this:

The News Find the number of articles in the media that link to/mention an article on Wikipedia about a topic. Check articles about a topic to find examples of media plagiarism of Wikipedia articles. Academia Identify the number of times an academic text uses Wikipedia as a source. Identify which academics reference Wikipedia. Sport Institutions Count the number of links to Wikimedia content on their website. Count the number of links to Wikimedia content on their social media related portals. The Community Facebook Measure the number of likes for a Wikipedia article on Facebook. Twitter Measure the number of links to an article. Identify who is tweeting about the article, get their individual Twitter metric data. Get the metric data for all the followers of a person tweeting about a topic. Google+ Measure the number of links to an article. Identify who is tweeting about the article, get their individual Google+ information. LiveJournal and clones Measure the number of mentions for an article on LiveJournal and its clones. Get the community or user metrics for including the links. Google search Determine the ranking of the Wikipedia and Wikinews articles on Google. Using Google Trends, determine the relative number of searches for these topics and where they are origination from, both with and without Wikipedia in the search phrase. Yahoo!Answers Measure the number of questions about the topic. (Closed, and open.) Measure the number of questions on Wikipedia about articles. Measure the number of answers that reference the article.

While these can help measure a certain level of influence, they do not necessarily explain why people are citing or how this forms thinking. Results would just suggest current levels of awareness but not necessarily influence. The two are really separate points.

*Survey attitudes towards English Wikipedia, English Wikinews and Commons coverage of Australian women’s sport:*

A third methodology is needed to complement the previous two. As much as it pains me, a qualitative methodology needs to be used: Survey work needs to be completed. The following groups need to be surveyed with questions seeking specific information in certain areas. The following are some broad themes for this group.

The media As a journalist, what is their opinion of Wikipedia in this area? Have they used Wikipedia, Wikinews or other projects as a resource when working on a story? Academics and students As an academic or student, what is their opinion of Wikipedia in this area? Have they used Wikipedia, Wikinews or other projects as a resource when working on papers? Sport industry participants including athletes As members of the sport industry, what is their opinion of Wikipedia in this area? Have they actively sought to improve content related to themselves or their sport organisation on WMF projects? Sport fans and participants Do they use Wikipedia for information and how reliable do they consider it to be? Do they contribute to Wikipedia?

This information would then need to be wrapped around existing research that discusses the influence of the media, in this case in an Australian and sport context, academia and industry influence on forming public perception. An approach also needs to be developed which can include a before and after treatment for a content improvement drive in this area. In doing the first part, it should hopefully become apparent where there is a need for articles to be improved or created to develop a class of articles for improvement.

Sincerely, Laura Hale