Thanks Leila. I just want to note that although I can't be present during the meeting tomorrow, I'm keenly interested in the VE research and will look into using lessons from that research to benefit the video project that's currently incubating as an IEG draft [1]. Regards, Pine [1] https://meta.wikimedia.org/wiki/Grants:IEG/Motivational_and_educational_vid… On Mon, Jul 27, 2015 at 2:47 PM, Leila Zia <leila(a)wikimedia.org> wrote:

I read the summary of the VE study, and I have a question. Anecdotally, I am hearing from multiple sources that new editors *who attend workshops or editathons in person* prefer VE over wikitext for ease of use. Do we have any data specifically about the productivity and longevity of this population of users when they are introduced to to Wikipedia editing on VE instead of wikitext? Thanks! Pine On Jul 29, 2015 11:09 AM, "Leila Zia" <leila(a)wikimedia.org> wrote:

I watched the video, in which Aaron did discuss social and motivational barriers as being more complex and difficult to solve than technical issues with VisualEditor. I liked the questions that Aaron asked ("Did you make friends? Did you find the work rewarding? Did you identify with the community?") because my understanding is that in the wide world of volunteer associations, questions like those are strongly related to volunteer retention and activity levels. I have a hunch that in-person workshops and editathons can do a lot to improve the onboarding and retention experience for new editors. My understanding from WMF Learning and Evaluation is that editathon series and writing contest series are particularly effective at retaining editors. I would guess that this effect happens because people in general may find it easier to make friends and identify with a community when they have face-to-face, positive interactions with other members of that community. However, also note that most students who write Wikimedia content for their classroom assignments don't remain active contributors after the completion with their assignments, so I speculate that the third issue ("did you find the work rewarding?") may be significantly affected by the intrinsic motives and interests of potential contributors, as well as competition for the time of those potential contributors from other activities (like good grades, fulfilling jobs, or happy activities with family and friends) that also provide rewards. There is ongoing work to improve the effectiveness of mentorships and wikiprojects on English Wikipedia, which may also help to address the "did you make friends" and "did you identity with the community" questions. I'm thinking about how I can implicitly take these issues into account when designing the content of the video project that I linked earlier in this thread, and how the video content could help with lowering social-motivational barriers. Suggestions from other participants on Research-l would be most welcome. Thanks, Pine Pine On Sun, Aug 2, 2015 at 1:20 AM, Kerry Raymond <kerry.raymond(a)gmail.com> wrote:

Thanks for posting your feedback. I also watched the video but my takeaways were so different from yours that I am tempted to rewatch the whole thing before responding. I do recall thinking that Aaron's presentation was significantly less boring than the student one, but that the students had a few points that need to be addressed. As far as Aaron's presentation goes I was most struck by his comments afterwards (so fast forward to about halfway through the video to catch those) and I remember thinking that I wished he could work on that stuff instead of VE stuff. I promise to revisit these comments later, so stay tuned. On Mon, Aug 3, 2015 at 9:43 AM, Pine W <wiki.pine(a)gmail.com> wrote:

Hey folks, I'm glad the presentation came across so well. I really appreciate the discussion. Pine, I really appreciate those plots that you linked. It seems that you can identify the progression through barrier types by following the hexagonal graphs clockwise. Concerns start with complex rules and (to a lesser extend) the difficulty of editing and progress to concerns negative social behavior and access to reference materials. Regarding editathons, I'm not quite sure the right way to measure their effects. I suspect that one of the biggest effects of editathons are the result of discussions that people have with their friends and family after the event. "I edited Wikipedia and it was fun. It turns out that there's a lot of different types of ways to contribute. You don't have to be an expert." -- is a conversation I imagine is relatively common after an editathon. The awareness (I can edit Wikipedia?!), new registrations and contributions that result from such once-removed discussions would be nearly impossible to track. Jane, seem more of my work exploring the rising social/motivational barriers here: https://www.youtube.com/watch?v=bozyc1z25aQ#t=24m49s In the conclusion of that talk, I bring up Snuggle[1] as an example of a technological strategy for supporting desirable social behaviors. My recent work on the Revision Scoring[2] was originally inspired by my work to extend Snuggle beyond English Wikipedia -- I needed vandalism prediction scores beyond English Wikipedia! Generally, I think we (as Wikipedian community members) have a lot deeper insight into the types of behaviors (e.g. reactions to newcomer contributions) that are desirable than we had in 2006 and that, if we were to redesign counter-vandalism tools from scratch with these insights in mind, we'd be able to dramatically reduce this type of social/motivational barrier. I think Snuggle is a good example of such a new type of tool and the idea with Revision Scoring is that I'd like to make it *really easy* for others to experiment with their own strategies. The next thing I want to do is to try empowering WikiProjects with automated quality control/socialization tools. I suspect that, WikiProject members will be highly motivated to socialize potential good newcomers and help them work productively within the topical context of their WikiProject -- if they had the means to do so efficiently. 1. https://en.wikipedia.org/wiki/Wikipedia:Snuggle 2. https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service -Aaron On Mon, Aug 3, 2015 at 7:36 AM, Jane Darnell <jane023(a)gmail.com> wrote:

+1 I've been working with James Hare (WikiProject X) to use the revision scoring system to support their work. Their bot is actually pulling scores from our service right now. :) Also +1 for Wikiprojects as entry points for newcomers. I've been pitching a WikiProject recommender service for a while. I'd really like to start experimenting with that. I think that might have been one of the next logical steps for the Growth team if it was still around. -Aaron On Mon, Aug 3, 2015 at 1:46 PM, Pine W <wiki.pine(a)gmail.com> wrote:

And the VE won’t solve the “community” problem. Given the hostility of the en.WP community to the VE (probably it’s not widespread but a rather a very vocal minority), it is not clear to me if tagging edits as being VE is actually a “red flag to a bull” to the VE-haters. That is, might a VE-hater behave (even more) aggressively towards new users using the VE? Some months ago, we altered the banner for WikiProject Australia to include an email help(a)wikimedia.org.au <mailto:help@wikimedia.org.au> as a way to reach out to new editors of Australian content who need help and don’t know the Wikipedia ways of getting help. It was successful in that we did receive emails, so there’s a tip – newbies find it easier to email to get help. Aside: What was unexpected about it was that many of them were a conflict-of-interest situation, where the editor was either the subject of the article or an employee or otherwise affiliated. Now in most cases they hadn’t created the article but they felt something was wrong and needed to be fixed or wanted to add something. Generally the articles weren’t puffery and the edits desired weren’t unreasonable (not white-washing) and generally we’ve helped them. None of them attempted to hide their connection to the article; most went to some trouble to establish their bona fides to make it clear that they were “authorised” to request this change. I don’t think any of these people wanted to learn to edit themselves, they just wanted the article fixed. Perhaps Wikipedia might be better off if we encouraged this kind of email request rather than try to force everyone to become editors. The other thing that pops up (sometimes CoI, sometimes not) is “I saw these messages about needing more references. I’ve added loads of references but no matter how many I add, those messages just don’t go away. What more can I do?”. Not understanding how Wikipedia works is another “cognitive impediment” for the newbie that the VE won’t solve. Of course, if we added in Aaron’s automated assessment tool, maybe we could provide a better way to give some article quality feedback than persisting with the belief that WikiProjects are alive and well and actively reassessing quality. So, while I do disagree with Jane about the technical impediments, I do agree 100% about the reverts as a big issue. It happens during edit training and it really upsets the people, even though I am there to hold their hand. It’s really hard when I cannot understand myself why the edit was reverted. And, as vast majority of attendees at edit training are female, I see a very strong reaction from the women that reverting is “not nice” and “very rude” (this is strong stuff, likely to lead to mentions of “little hitlers”). I know the “Club House” paper didn’t detect any difference between male and female reactions, but in terms of verbal reactions to being reverted, the women are quite vocal and all agree on the “not nice”. I suspect the men are just as offended but don’t mention it (the strong silent stereotype would prevent them telling me, a woman). When we look at the gender gap, I have to wonder if we have something very basically wrong with the Bold – Revert – Discuss approach. I don’t think it works for women, who tend to Propose – Discuss – Discuss – Discuss – Eventually Implement. I think the whole “Bold – Revert” is very libertarian ideology so I am unsurprised it doesn’t appeal to women. Women tend to do things more slowly but I think get happier outcomes, probably because women are socialised to keep people happy. Kerry From: wiki-research-l-bounces(a)lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Jane Darnell Sent: Monday, 3 August 2015 10:36 PM To: Research into Wikimedia content and communities &lt;wiki-research-l(a)lists.wikimedia.org&gt; Subject: Re: [Wiki-research-l] July 2015 Research showcase OK I am replying to this mail, as this one has the link to Youtube in it with the two presentations. I am only responding to the first presentation by Aaron here. In general I like the idea of focussing attention on the "New Editor Activation Funnel". This area is of course the reason why we have a decline in new editors, and it all has to do with an increase in "barriers to entry" (which btw I am not convinced is the same thing as "technical impediments"). It is useful to split these barriers up into Permission, Literacy (here wikimarkup is lumped together with policies), and Social/Motivational (human interaction) issues, but I think the whole presentation misses the point on the need for more dissection of the reverts problem (shown a bit towards the end). I personally think that demotivational behavior by experienced Wikipedians is the biggest factor in the decline of new editor contributions, but unlike most people I don't think this has to do with what the experienced Wikipedians do, but rather what they don't do. They don't welcome people in person (because they don't see their edits) and they don't give timely feedback on first edits to pages on their watchlist (no way to see if those edits are first time edits). They don't show them the ropes in that if one wants to make a BLP, or an article about a company or building or place, or an article about an artwork, you should look at existing examples and start from there. Having said this, I do think we spend an inordinate amount of time on things like extending the page about WHAT WIKIPEDIA IS NOT (which btw I have yet to read). It seems that our best way of dealing with newcomers is to throw CAPS at them, though we all hate CAPS. The point of this study was to prove these two: H1: VE will increase the amount of desirable edits by newbies and H2: VE will increase the amount of undesirable edits by newbies (aka VANDALISM). Guess what? Both H1 & H2 show no significance and if anything, less vandalism came from VE editors. I could have told you that beforehand - yawn. It angers me when people assume that others are not technical enough for Wikipedia. Sorry, but it is not rocket science. This type of thinking is not just on Wikipedia, I see this also in health occupations, where doctors tell their patients not to go look things up on the Internet. Just trust the doctors because they studied it! Yeah right, like I am going to trust all aspects of my future health and well-being to someone who sees my future health and well-being as a 10-minute interlude in their 9-5 workday. No, I will nod politely (one must always remain friendly) while googling my way to better health, thanks. And if I want to make an article about something that I think needs an article on Wikipedia, I am going to try to do it on my own as far as I can get, and I am probably not interested in talking about it until I am done. The whole AfC queue thing is absolutely horrible because it puts these edits on ice until the person totally forgets what the password was that they dreamed up for their user account. As far as spelling corrections go, if I correct an error and see it deleted (like from Kiev to Kyiv, which will be reverted by a bot probably), then I will probably not come back. I am very eager to hear more about the revision scoring though! I wish there was a better way to do that than manually however. Jane On Wed, Jul 29, 2015 at 8:07 PM, Leila Zia &lt;leila(a)wikimedia.org <mailto:leila@wikimedia.org> > wrote: A friendly reminder that this is happening in 23 min. :-) YouTube stream: https://www.youtube.com/watch?v=vGyrVg_qKSM IRC: #wikimedia-research Best, Leila On Mon, Jul 27, 2015 at 2:47 PM, Leila Zia &lt;leila(a)wikimedia.org <mailto:leila@wikimedia.org> > wrote: Hi everyone, The next Research showcase will be live-streamed this Wednesday, July 29 at 11.30 PT. The streaming link will be posted on the lists a few minutes before the showcase starts (sorry, we haven't been able to solve this, yet. :-() and as usual, you can join the conversation on IRC at #wikimedia-research. We look forward to seeing you! Leila This month: VisualEditor's effect on newly registered users By <https://www.mediawiki.org/wiki/User:Halfak_%28WMF%29> Aaron Halfaker It's been nearly two years since we ran an initial study <https://meta.wikimedia.org/wiki/Research:VisualEditor%27s_effect_on_newly_registered_editors/June_2013_study> of VisualEditor's effect on newly registered editors. While most of the results of this study were positive (e.g. workload on Wikipedians did not increase), we still saw a significant decrease in the newcomer productivity. In the meantime, the Editing <https://www.mediawiki.org/wiki/Editing> team has made substantial improvements to performance and functionality. In this presentation, I'll report on the results of a new experiment designed to test the effects of enabling this improved VisualEditor software for newly registered users by default. I'll show what we learned from the experiment and discuss some results have opened larger questions about what, exactly, is difficult about being a newcomer to English Wikipedia. Wikipedia knowledge graph with DeepDive By Juhana Kangaspunta and Thomas Palomares (10-week student project) Despite the tremendous amount of information present on Wikipedia, only a very little amount is structured. Most of the information is embedded in text and extracting it is a non-trivial challenge. In this project, we try to populate Wikidata, a structured component of Wikipedia, using DeepDive tool to extract relations embedded in the text. We finally extracted more than 140,000 relations with more than 90% average precision. We will present DeepDive and the data that we use for this project, we explain the relations we focused on so far and explain the implementation and pipeline, including our model, features and extractors. Finally, we detail our results with a thorough precision and recall analysis. _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org <mailto:Wiki-research-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wiki-research-l