Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen
Looking at the list of WikiProjects you pointed at, they seem to be a mixture of what I would call "process" projects (e.g. Articles for Creation, Deletion Sorting) vs "content" projects (e.g. Military History, Television) vs a third group like "Women in Red" (which is part process, part content).
Generally the "content" projects will tag Talk pages with their WikiProject Banner. But "process" projects don't seem to always do this. For example, I don't think Women in Red has a project banner generally, although I think they do tag articles that arise from specific Edit-a-thons. Some of the process projects seem to use hidden categories for their work.
I would suggest only working with content projects initially. Content projects are more similar to one another in how they operate compared to process projects, and I think it is easier to judge if a user is showing an interest in a content project than in the process project because of standard use of content project banners on articles. So I think you can probably get a better understanding if the referral mechanism is working or not with content projects, whereas I think process projects have a lot of variability in them that may make it difficult to work out if you are seeing success or not.
And at the end of the day, as an encyclopedia, we live or die on our content. Processes are (or at least should be) supportive of good content development but are a second-order effect.
I can certainly see some issues arising from pointing newcomers at process projects as they are unlikely to be aware of the processes at that stage. And indeed some process project do not accept new editors (think of Articles for Creation and new page patrolling). I'd see this as a second project if the content project referral mechanism seems to be working.
Anyhow, that my 10cc!
Kerry
----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Bowen Yu Sent: Tuesday, 20 June 2017 4:35 PM To: Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: [Wiki-research-l] Research about WikiProject Recommendation
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi Bowen,
If you are going to promote wikiprojects by recommendation then you need to test different styles of recommendation. Taking what may still be the two biggest wikiprojects, MILHIST and professional wrestling, what worked as an invitation for either might be quite different than what would work for Opera or chemistry. Tone of voice is important when you are seeking to entice volunteers.
You also need to allow for the effect of different existing recruitment programs. These tend to be subtle, but they will vary, and that variation could mask your project. The most obvious recruitment is via wikiproject tagging of articles, and that isn't necessarily done by people who are active in the project concerned.
Regards
Jonathan
On 20 Jun 2017, at 07:35, Bowen Yu yuxxx856@umn.edu wrote:
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
There are pros and cons.
Having a standard invitation makes for better research as the project outcomes are more comparable but perhaps worse for recruitment.
Perhaps the WikiProjects involved could write the invitation for the person to participate, hopefully they can get the tone right. They might want to particularly encourage (or discourage) people with specific skills or interests, e.g. "We are particularly interested in expanding our articles on Pacific Island 17th century wrestling. We are in desperate need of people who can develop templates. Our project prides itself on fully cited articles." But then differences in the invitation may lead to differences in the uptake. Better recruitment, but worse research.
And of course what are the variables being measured for the outcome: * number of people invited to each WikiProject (presumably easy enough) * number of people who take up the invitation - how do we determine this? listing themselves on the Project page under Participants (yikes, I am active in many projects where I haven't done that), increasing their level of editing on articles associated with that project, increased activity on the project Talk page? Opinion of project leaders (do we have project leaders)? Self-identifying as such when asked by researchers? * level of activity wrt to the project at various periods after the invitation is accepted (when is it accepted? See above)
Kerry
-----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Jonathan Cardy Sent: Tuesday, 20 June 2017 8:02 PM To: Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation
Hi Bowen,
If you are going to promote wikiprojects by recommendation then you need to test different styles of recommendation. Taking what may still be the two biggest wikiprojects, MILHIST and professional wrestling, what worked as an invitation for either might be quite different than what would work for Opera or chemistry. Tone of voice is important when you are seeking to entice volunteers.
You also need to allow for the effect of different existing recruitment programs. These tend to be subtle, but they will vary, and that variation could mask your project. The most obvious recruitment is via wikiproject tagging of articles, and that isn't necessarily done by people who are active in the project concerned.
Regards
Jonathan
On 20 Jun 2017, at 07:35, Bowen Yu yuxxx856@umn.edu wrote:
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Thanks for your thoughts, Kerry and Jonathan!
Here is some response for your comments.
1. Regarding the targeted projects. Yes, we will definitely focus on content projects only for the reasons you mentioned. "Women in Red" is kind of special, and I don't think we will include it for now. Sorry for not being clear about this.
2. Regarding the invitation tactics. We do think there is a spectrum here, by either providing templates or giving total freedom to the recruiters. Also, not sure if some projects have their own templates of recruiting new members, but for now, we will provide general guidance for constructing recruiting the message. For instance, as mentioned in the meta-page, we will encourage project organizers to write personalized welcome messages, make specific task requests, or provide resource to start. Hope this should do the job - getting the right tone while still having it under control.
3. Yeh, for evaluation like if the invitation is taken, thanks for providing all those reasonable possibilities. Listing themselves on the project page might be one approach, but as Kerry mentioned, it might not work sometime. Rather, we can just see if they make any edit on the project (talk) pages as a sign of getting involved, or more loosely, if they keep editing project related articles. We will also provide short survey questions for each recommended editor to let organizers/project leaders evaluate the recommendation quality. We are expecting the project leaders to self-identify themselves when we post a recruiting message on the targeted projects to look for volunteer participants for our study (we will explicitly mention looking for "project leaders" or using similar descriptions).
On Tue, Jun 20, 2017 at 4:11 PM, Kerry Raymond kerry.raymond@gmail.com wrote:
There are pros and cons.
Having a standard invitation makes for better research as the project outcomes are more comparable but perhaps worse for recruitment.
Perhaps the WikiProjects involved could write the invitation for the person to participate, hopefully they can get the tone right. They might want to particularly encourage (or discourage) people with specific skills or interests, e.g. "We are particularly interested in expanding our articles on Pacific Island 17th century wrestling. We are in desperate need of people who can develop templates. Our project prides itself on fully cited articles." But then differences in the invitation may lead to differences in the uptake. Better recruitment, but worse research.
And of course what are the variables being measured for the outcome:
- number of people invited to each WikiProject (presumably easy enough)
- number of people who take up the invitation - how do we determine this?
listing themselves on the Project page under Participants (yikes, I am active in many projects where I haven't done that), increasing their level of editing on articles associated with that project, increased activity on the project Talk page? Opinion of project leaders (do we have project leaders)? Self-identifying as such when asked by researchers?
- level of activity wrt to the project at various periods after the
invitation is accepted (when is it accepted? See above)
Kerry
-----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Jonathan Cardy Sent: Tuesday, 20 June 2017 8:02 PM To: Research into Wikimedia content and communities < wiki-research-l@lists.wikimedia.org> Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation
Hi Bowen,
If you are going to promote wikiprojects by recommendation then you need to test different styles of recommendation. Taking what may still be the two biggest wikiprojects, MILHIST and professional wrestling, what worked as an invitation for either might be quite different than what would work for Opera or chemistry. Tone of voice is important when you are seeking to entice volunteers.
You also need to allow for the effect of different existing recruitment programs. These tend to be subtle, but they will vary, and that variation could mask your project. The most obvious recruitment is via wikiproject tagging of articles, and that isn't necessarily done by people who are active in the project concerned.
Regards
Jonathan
On 20 Jun 2017, at 07:35, Bowen Yu yuxxx856@umn.edu wrote:
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page <https://meta.wikimedia.org/wiki/Research:WikiProject_
Recommendation>.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us
know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Hi, Bowen!
Are you an active Wikipedian yourself? Or just interested in studying them? If the latter, please be aware that Wikipedians are not entirely normal people (I think that’s the kindest way to say it) and their methods for organising themselves and their organisational cultures may not be reflective of broader society. If I had to try to put a label on it, I would say it is libertarian. So you may not be able to engage with them in the ways you might expect. Specifically …
I am not sure if people will identify as “project leaders”. I am very active with WikiProject Australia myself and somewhat active in others, but I would not be able to tell you who the project leaders were in any of them. Indeed, I think if you were to go to those projects and ask its leaders to put their hand up, I don’t think anyone would, precisely because there is no process which confers leadership in any way. I think you might do better to ask for people “willing to assist with the newly recruited users to your project” to nominate, which isn’t asking anyone to say “I’m the leader here”. At the end of the day, you need someone to take the first steps to engage with the newcomer, so you might as well ask for it directly.
Because of the lack of identified project leaders, I suspect even questions like “does this project want to be part of this recruitment activity?” may be bounced around on the project Talk page without ever coming to a conclusion (there generally isn’t a formal process closing on those kind of conversations as there is on the more procedural conversations like Articles for Deletion, Request for Comment, etc.). My suggestion here is to kick off the question about being involved in this invited recruitment and then wait a few days to see if there seem to be some folk saying positive things about the idea (inevitably somebody will not like the idea, that’s life on Wikipedia). If the conversation terminates without a clear consensus YES or NO (and I think this is likely to be the case), then say something like “well, some of you think it’s worth trying, so are any of you willing to be first contact points for the invitation/recruitment process?”. In a way, you don’t really care if the project as a whole buys into the idea or not, but you do need individuals to be involved as the interface into the project.
What I suspect some projects will do will be to say “no need to invite people or appoint individuals to assist the newcomer, any newcomer can become a member of the project the same way as everyone else did”. (Classic Wikipedian behaviour – if it was good enough for me, it’s good enough for everyone else) I think you need a comeback line like “not everyone may feel that confident, some people may feel they need to be invited and may need a point of contact to ask for project-specific help as they get started”. But be aware that there are many Wikipedians who seem to dislike “newbies” in any form, despise any activity that recruits/assists newbies (e.g. TeaHouse) or oppose hand-holding in any form. In their opinion, real Wikipedians are born not made and those who can’t make it to high level of contributions on their own (“like I did”) are best left to wither and die on the vine.
So it might be harder than you to think to sign up WikiProjects, or more importantly, the mentor. The other thing to bear in mind is that you should not introduce new people into a project faster than the mentors available can cope with the influx. The ones who are relatively “new to Wikipedia” are quite a lot of work (I do Wikipedia edit training and so get to mentor a lot of very new people at the same time and it can get quite demanding at times). I don’t know what the right speed of recruiting is; but maybe it’s something you need to negotiate with the individual mentors. It might be a case of “tell me when you are ready to be introduced to another recruit”. With others, they might be happy to receive many newcomers each day as all they intend to do is point them at a To Do list and say “off you go, ask me if you need help”.
There may be a need to do some matching between the recruitee and the mentor. One factor that plays into the relationship is “time of day on Wiki”. If the newcomer needs help now, getting from a mentor active at a different time of day makes for a very slow turn-around for a conversation. Some people are only active on weekends. Unfortunately requests for help can come too late. They’ve created a new article, they’ve got other editors complaining, threatening to delete, edit wars, nasty conversations on Talk pages, etc. All of this can happen fairly quickly, and by the time you’ve seen the request for your help , it is often really hard to help them at that stage of things (too many people are mad at them to recover the situation).
Another factor to bear in mind is the edit tool people are using. Most of our existing contributors are source-edit people. Our newer people will be a mix of source-edit and Visual Editor people. A Visual Editor user cannot write on a Project Page or any Talk page with the Visual Editor because the Visual Editor has not been enabled for those name spaces (it’s one of those “it was good enough for me” issues). You can get around this for an individual Project page or Talk page by adding the template
{{VEFriendly}} at the top of the page, see it in action on my User Talk page
https://en.wikipedia.org/wiki/User_talk:Kerry_Raymond https://en.wikipedia.org/wiki/User_talk:Kerry_Raymond
Aside. I notice this template does not appear on the top of
https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Research https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Research
https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Research
Is there any reason why not? BTW, who are the “project leaders” in that project?
But back to my point, if you have an invitee who appears to be a VE user, you probably want to connect them to a VE-Friendly mentor (they don’t have to be an active VE user themselves, but ought to have some understanding of how the VE User “sees” an article and how to give advice in “VE Speak” not source-editor notation). If the first piece of advice the new VE recruit gets is to “change to the source editor”, it’s not exactly welcoming. So you probably want to ask your mentors about the source-vs-visual editor issue. You probably need each project to have a mentor who is OK with VE or else you may be recruiting someone you can’t match with a mentor. I think all mentors will be fluent in source editor but fewer will be comfortable with VE.
Now it may be that others on this list see WikiProjects in a different way to me because the WikiProjects they are involved with operate in different ways and have a different culture. So hopefully people will chime in with other perspectives.
Kerry
From: Bowen Yu [mailto:yuxxx856@umn.edu] Sent: Wednesday, 21 June 2017 12:00 PM To: kerry.raymond@gmail.com; Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation
Thanks for your thoughts, Kerry and Jonathan!
Here is some response for your comments.
1. Regarding the targeted projects. Yes, we will definitely focus on content projects only for the reasons you mentioned. "Women in Red" is kind of special, and I don't think we will include it for now. Sorry for not being clear about this.
2. Regarding the invitation tactics. We do think there is a spectrum here, by either providing templates or giving total freedom to the recruiters. Also, not sure if some projects have their own templates of recruiting new members, but for now, we will provide general guidance for constructing recruiting the message. For instance, as mentioned in the meta-page, we will encourage project organizers to write personalized welcome messages, make specific task requests, or provide resource to start. Hope this should do the job - getting the right tone while still having it under control.
3. Yeh, for evaluation like if the invitation is taken, thanks for providing all those reasonable possibilities. Listing themselves on the project page might be one approach, but as Kerry mentioned, it might not work sometime. Rather, we can just see if they make any edit on the project (talk) pages as a sign of getting involved, or more loosely, if they keep editing project related articles. We will also provide short survey questions for each recommended editor to let organizers/project leaders evaluate the recommendation quality. We are expecting the project leaders to self-identify themselves when we post a recruiting message on the targeted projects to look for volunteer participants for our study (we will explicitly mention looking for "project leaders" or using similar descriptions).
On Tue, Jun 20, 2017 at 4:11 PM, Kerry Raymond <kerry.raymond@gmail.com mailto:kerry.raymond@gmail.com > wrote:
There are pros and cons.
Having a standard invitation makes for better research as the project outcomes are more comparable but perhaps worse for recruitment.
Perhaps the WikiProjects involved could write the invitation for the person to participate, hopefully they can get the tone right. They might want to particularly encourage (or discourage) people with specific skills or interests, e.g. "We are particularly interested in expanding our articles on Pacific Island 17th century wrestling. We are in desperate need of people who can develop templates. Our project prides itself on fully cited articles." But then differences in the invitation may lead to differences in the uptake. Better recruitment, but worse research.
And of course what are the variables being measured for the outcome: * number of people invited to each WikiProject (presumably easy enough) * number of people who take up the invitation - how do we determine this? listing themselves on the Project page under Participants (yikes, I am active in many projects where I haven't done that), increasing their level of editing on articles associated with that project, increased activity on the project Talk page? Opinion of project leaders (do we have project leaders)? Self-identifying as such when asked by researchers? * level of activity wrt to the project at various periods after the invitation is accepted (when is it accepted? See above)
Kerry
-----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org mailto:wiki-research-l-bounces@lists.wikimedia.org ] On Behalf Of Jonathan Cardy Sent: Tuesday, 20 June 2017 8:02 PM To: Research into Wikimedia content and communities <wiki-research-l@lists.wikimedia.org mailto:wiki-research-l@lists.wikimedia.org >
Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation
Hi Bowen,
If you are going to promote wikiprojects by recommendation then you need to test different styles of recommendation. Taking what may still be the two biggest wikiprojects, MILHIST and professional wrestling, what worked as an invitation for either might be quite different than what would work for Opera or chemistry. Tone of voice is important when you are seeking to entice volunteers.
You also need to allow for the effect of different existing recruitment programs. These tend to be subtle, but they will vary, and that variation could mask your project. The most obvious recruitment is via wikiproject tagging of articles, and that isn't necessarily done by people who are active in the project concerned.
Regards
Jonathan
On 20 Jun 2017, at 07:35, Bowen Yu <yuxxx856@umn.edu mailto:yuxxx856@umn.edu > wrote:
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org mailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org mailto:Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org