Hello all,
This is sort of related to the previous thread on consent, and is something that's been on my mind for a while.
As someone who is reasonably visible in the wiki research community and Wikipedia, I get asked fairly often to either a) help recruit participants for studies of Wikipedia, or b) participate myself in such studies.
The common theme of such requests is often: * The researcher wants to find people who are invested in Wikipedia, for qualitative studies of contributors * The researcher does not know how to go about doing this * What "invested" means is often poorly defined, since the researcher is often trying to figure out what participation looks like more generally * The researcher has done the standard things (posted on the mailing list, on the village pump) and hasn't gotten any results; or has semi-randomly posted on people's talk pages, potentially getting a warning about spamming in the process
As a result: * many of the same people (i.e. very visible contributors) keep getting asked to participate in different studies; or * the researcher is left with a self-selected group of people from the mailing lists or other places, which may in no way represent 'the community' (my hypothesis is that we have many small communities, working under the greater umbrella of Wikipedia); and who may be people who are particularly outspoken or disgruntled; or * the researcher does not get enough participants to do a good study
So: * Is there a good solution for these problems? * Can we come up with "best practices" or advice for people who are trying to recruit Wikipedians for studies? * What about some sort of infrastructure or wikiproject to support these requests? Every time I get one of these emails I would really like to pass it on to a group of people to deal with, but I am not sure who, and this mailing list seems too small and focused to support such requests.
best, Phoebe
p.s. I think "any wiki with a large base of contributors" could be substituted for "Wikipedia" here -- this is probably a problem with studying any large community-run site. But most of my requests have come from people specifically interested in Wikipedia.
- phoebe s. ayers | phoebe.ayers@gmail.com
phoebe ayers wrote:
- The researcher has done the standard things (posted on the mailing
list, on the village pump) and hasn't gotten any results; or has semi-randomly posted on people's talk pages, potentially getting a warning about spamming in the process
As a result:
- many of the same people (i.e. very visible contributors) keep
getting asked to participate in different studies; or
- the researcher is left with a self-selected group of people from the
mailing lists or other places, which may in no way represent 'the community' (my hypothesis is that we have many small communities, working under the greater umbrella of Wikipedia); and who may be people who are particularly outspoken or disgruntled; or
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User If anybody wants a truly random sample of Wikipedia users, that's a good way to do it.
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
On Wed, Nov 19, 2008 at 2:54 PM, Piotr Konieczny piokon@post.pl wrote:
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User If anybody wants a truly random sample of Wikipedia users, that's a good way to do it.
Thats more than a slightly tedious process: The overwhelming majority of user accounts on the Enwp (and to a lesser extent on other large projects) are inactive/have no-non-deleted etc.
If a researcher wants a constrained sample of users such as "Users who have made more than 1 edit and who have edited in the past month" or whatever, they can ask anyone with a toolserver account to make them such a list.
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
It might be useful to create a way for users to opt-out in bulk, I doubt many would do it, but it might help keep the few who would from becoming upset and going on a crusade after the "dreadful spammer".
On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny piokon@post.pl wrote:
phoebe ayers wrote:
- The researcher has done the standard things (posted on the mailing
list, on the village pump) and hasn't gotten any results; or has semi-randomly posted on people's talk pages, potentially getting a warning about spamming in the process
As a result:
- many of the same people (i.e. very visible contributors) keep
getting asked to participate in different studies; or
- the researcher is left with a self-selected group of people from the
mailing lists or other places, which may in no way represent 'the community' (my hypothesis is that we have many small communities, working under the greater umbrella of Wikipedia); and who may be people who are particularly outspoken or disgruntled; or
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User If anybody wants a truly random sample of Wikipedia users, that's a good way to do it.
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
This seems like an interesting idea for random studies.
Many of the studies I have encountered are looking for "some experienced wikipedians" to participate, and that seems like a much harder thing to pull off. The chances of getting an "experienced" user who is willing to talk to you out of the 8 million registered accounts on en:wp seems low if you go about it by trying to randomly poll users. I think a lot of researchers want people who are 'experienced' by some metric so they can give informed feedback on processes, policies, social dynamics, interfaces... but it's also pretty common that most contributors focus in on only a few areas. So how to get experienced people who can speak to the questions you are interested in, particularly if you're not a wikipedian yourself?
-- phoebe
p.s. This message is not a comment on Avanidhar's study or Stuart's study or any other particular studies... rather, it's something I've been thinking about for a long time as an issue for everyone studying Wikipedia.
Dear All,
Thanks for bringing this topic up on the list. True, we do seek random users from the Wikipedia community for our studies. But, like phoebe mentioned, we usually expect a particular trait/quality in the users. Generally, most studies are aimed at "experienced editors": those who have been along for a while, who understand wikipedia policies and who make significant contributions to any article/set of articles over a period of time.
It would be great if there were some mechanism to invite editors to participate in our studies without giving an image of us being "spammers" or "vandals". After all, all we need is a sizeable number of participants out of millions of editors, which would not only make our research credible, but also worth mentioning while contemplating future directions for any Wiki based system.
~ Avanidhar
On Nov 19 2008, phoebe ayers wrote:
On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny piokon@post.pl wrote:
phoebe ayers wrote:
- The researcher has done the standard things (posted on the mailing
list, on the village pump) and hasn't gotten any results; or has semi-randomly posted on people's talk pages, potentially getting a warning about spamming in the process
As a result:
- many of the same people (i.e. very visible contributors) keep
getting asked to participate in different studies; or
- the researcher is left with a self-selected group of people from the
mailing lists or other places, which may in no way represent 'the community' (my hypothesis is that we have many small communities, working under the greater umbrella of Wikipedia); and who may be people who are particularly outspoken or disgruntled; or
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User If anybody wants a truly random sample of Wikipedia users, that's a good way to do it.
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
This seems like an interesting idea for random studies.
Many of the studies I have encountered are looking for "some experienced wikipedians" to participate, and that seems like a much harder thing to pull off. The chances of getting an "experienced" user who is willing to talk to you out of the 8 million registered accounts on en:wp seems low if you go about it by trying to randomly poll users. I think a lot of researchers want people who are 'experienced' by some metric so they can give informed feedback on processes, policies, social dynamics, interfaces... but it's also pretty common that most contributors focus in on only a few areas. So how to get experienced people who can speak to the questions you are interested in, particularly if you're not a wikipedian yourself?
Hi all -
I am relatively new to the list but I have been looking at Wikipedia (quant) data for a few years now. I recently started contacting people for interviews regarding socialization/social networks of collaboration, so I second your concerns. It's hard to contact users who have left and even harder to get a response from inexperienced users, which I would love to know more about in order to understand the process through which people gain experience/ make sense of their Wikipedia participation.
Is there a way to do this through the Wikimedia Foundation, any thoughts?
Thank you everyone for being so willing to help (each other), this list is such a wonderful resource!
Regards, Andreea
-----Original Message----- From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of avani@cs.umn.edu Sent: Wednesday, November 19, 2008 4:40 PM To: Research into Wikimedia content and communities Subject: Re: [Wiki-research-l] soliciting participants for Wikipedia studies
Dear All,
Thanks for bringing this topic up on the list. True, we do seek random users from the Wikipedia community for our studies. But, like phoebe mentioned, we usually expect a particular trait/quality in the users. Generally, most studies are aimed at "experienced editors": those who have been along for a while, who understand wikipedia policies and who make significant contributions to any article/set of articles over a period of time.
It would be great if there were some mechanism to invite editors to participate in our studies without giving an image of us being "spammers" or "vandals". After all, all we need is a sizeable number of participants out of millions of editors, which would not only make our research credible, but also worth mentioning while contemplating future directions for any Wiki based system.
~ Avanidhar
On Nov 19 2008, phoebe ayers wrote:
On Wed, Nov 19, 2008 at 11:54 AM, Piotr Konieczny piokon@post.pl wrote:
phoebe ayers wrote:
- The researcher has done the standard things (posted on the mailing
list, on the village pump) and hasn't gotten any results; or has semi-randomly posted on people's talk pages, potentially getting a warning about spamming in the process
As a result:
- many of the same people (i.e. very visible contributors) keep
getting asked to participate in different studies; or
- the researcher is left with a self-selected group of people from the
mailing lists or other places, which may in no way represent 'the community' (my hypothesis is that we have many small communities, working under the greater umbrella of Wikipedia); and who may be people who are particularly outspoken or disgruntled; or
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User If anybody wants a truly random sample of Wikipedia users, that's a good way to do it.
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
This seems like an interesting idea for random studies.
Many of the studies I have encountered are looking for "some experienced wikipedians" to participate, and that seems like a much harder thing to pull off. The chances of getting an "experienced" user who is willing to talk to you out of the 8 million registered accounts on en:wp seems low if you go about it by trying to randomly poll users. I think a lot of researchers want people who are 'experienced' by some metric so they can give informed feedback on processes, policies, social dynamics, interfaces... but it's also pretty common that most contributors focus in on only a few areas. So how to get experienced people who can speak to the questions you are interested in, particularly if you're not a wikipedian yourself?
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
On Wednesday 19 November 2008, Gorbatai, Andreea wrote:
I am relatively new to the list but I have been looking at Wikipedia (quant) data for a few years now. I recently started contacting people for interviews regarding socialization/social networks of collaboration, so I second your concerns. It's hard to contact users who have left and even harder to get a response from inexperienced users, which I would love to know more about in order to understand the process through which people gain experience/ make sense of their Wikipedia participation.
I'll just note that my one brief experience in soliciting interviews online was rather troubled (see below). Fortunately, for practical and theoretical reasons I preferred making use of public practice and discussion. In any case, should I need to do so again, the best "interviews" I did make were by going to F2F meetings and connecting with Wikipedians. This isn't a random or representative sample of course.
[[ http://reagle.org/joseph/2005/ethno/leadership.html
On a suggestion, I developed a brief questionnaire to engage with editors of the Harry Potter Project pages but, as expected, received few responses. Open content communities are, presently, often studied (with similar questionnaires) and participants might have little interest in taking time away from their actual (volunteer) work to respond to yet another. (As a participant, I have never responded to such a questionnaire.) Contacting actual participants can be difficult as well, as Lorenzon (2005) noted: "Many editors have their own user page which give information about them but few give out their real names and contact information." I made my solicitation on the Talk page for the Project as well as the Talk pages of a handful of prominent editors, without much success. Additionally, because most all the discourse is public and the community is otherwise so reflective, there is an abundance of existing data situated in actual practice. This is not to say such research discussions are not useful; once I developed my questions I was interested in receiving answers and the single response was informative. Fortunately, while responses to questionnaires can be hard to obtain, I also do not think them necessary to understand this community. Instead, one must follow (or even engage) in the practice: "A culture is expressed (or constituted) only by the actions and words of its members and must be interpreted by, not given to, a field worker" (Van Maanen 1988).
]]
Piotr Konieczny wrote:
Is asking for a survey spamming? That's a good question. If we could raise it on a community page and get a consensus that it is not, than we could potentially create a bot that could be fed a survey and would deliver it to x random users via the above page.
One of my colleagues was metaphorically eaten alive for posting requests for research participation on user talk pages (a few dozen IIRC). In her case the users in question were chosen fairly specifically (perhaps by the kinds of pages they edited?), and I think she had 100-200 candidate participants.
I've bcc'ed her in case she wants to chime in.
Let me voice my strong support for Phoebe's effort. I'm not sure what I can do personally (though please enlighten me if you know), but some process with community buy-in for soliciting random Wikipedia users for research purposes would be extremely useful to the research community.
Reid
"survey spamming"
In medical trials subjects are very often paid. I would guess that the response rate for a Wikipedia survey would increase if the researcher mentioned that the subjects/survey taker would be paid.
In a survey for our "Lyngby" programming toolbox we made a lottery among the respondents and gave away a gift card for the Amazon online bookstore to the winner. Among the 270 registered users we got 28 responses and one regarding our survey as spam.
/Finn
___________________________________________________________________
Finn Aarup Nielsen, DTU Informatics, Denmark Lundbeck Foundation Center for Integrated Molecular Brain Imaging http://www.imm.dtu.dk/~fn/ http://nru.dk/staff/fnielsen/ ___________________________________________________________________
Piotr Konieczny wrote:
There is an interesting tool: http://en.wikipedia.org/wiki/Special:Random/User
http://en.wikipedia.org/wiki/Special:Random/User_talk is even better
Also, a thought on how to reduce spammer criticism: creata a list for opt-outs that the survey bot would consider.
If we could indeed get advanced help from somebody on toolserv, who could add options to the bot that would allow us to screen for things like recent activity or min. number of edits, it would be great.
wiki-research-l@lists.wikimedia.org