Re: [Foundation-l] New editor engagement experiments team

---------------------------------------------------------------------- Message: 1 Date: Wed, 21 Mar 2012 14:52:19 -0700 From: Maryana Pinchuk &lt;mpinchuk(a)wikimedia.org&gt; To: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Subject: Re: [Foundation-l] [Wikimedia Announcements] Fwd: Announcement: New editor engagement experiments team! Message-ID: &lt;CAOjELoaxymDeK=xd650g+ERAGxwRmKCxK11ENUS4pDzi6hL2xA(a)mail.gmail.com&gt; Content-Type: text/plain; charset=ISO-8859-1 On Wed, Mar 21, 2012 at 3:30 PM, MZMcBride &lt;z(a)mzmcbride.com&gt; wrote: Sorry, just want to jump in here and provide a citation for Zack's speculation on new user quality. We actually did this study<http://meta.wikimedia.org/wiki/Research:Newcomer_quality#Conclusio…on>:) (Props and shout-outs to Aaron Halfaker, who set this up.) With all the usual caveats about small-scale one-time qualitative research studies in place... the conclusion appears to be that the quality of new editors hasn't really changed much over the years, and most new editors are still (and always have been) trying to help the encyclopedia. Perhaps when viewed from the perspective of new page patrollers, there appears to be a significant rise in spammers and SPAs, but it's important to remember that there are many non-article-creating newbies out there. The other important thing to note from this study is that the rate of rejection (deletion or reverts) of new users' edits is disproportionate to the number of poor quality contributions, which means there are just as many good new editors now as there always have been, but they're entering an environment that's increasingly suspicious and critical of their work and, predictably, they aren't sticking around. So, personally, no, I'm not too worried that by opening the door a little wider for new contributors (and by holding it open long enough for them to learn all the social and technical nuances of editing), we're going to attract a flood of spammers and self-promoters. Those people will always be there, of course, but the community has developed pretty good methods of dealing with them, and ultimately they're a small part of a big community of people who just want to write a damn good encyclopedia :) Maryana -- Maryana Pinchuk Community Organizer, Wikimedia Foundation wikimediafoundation.org ------------------------------ Message: 2 Date: Wed, 21 Mar 2012 15:01:18 -0700 From: En Pine &lt;deyntestiss(a)hotmail.com&gt; To: &lt;foundation-l(a)lists.wikimedia.org&gt; Subject: [Foundation-l] Editor retention (was "Announcement: New editor engagement experiments team!") Message-ID: &lt;BLU154-ds5A876A39662C890698D92A6400(a)phx.gbl&gt; Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Responding to MZMcBride's question, "And a bit larger than this, what's an acceptable cost for keeping new editors around? For example, deleting a new user's article is probably the easiest way to discourage him or her, but is the alternative (allowing their spammy page to sit around for a while) an acceptable cost for the potential benefit?" First, I think that the new visual editor will help. Second, I think that the NOTFACEBOOK policy is a bit counterproductive in its current form. Wikipedia is a collaborative work and I've seen the NOTFACEBOOK policy pushed in the faces of people who engage in personal conversation on their talk pages. We want people to develop collaborative relationships here, right? I don't mean to suggest that people should turn userpages entirely into personal blogs, but I also think that the statement "Wikipedians have their own user pages, but they may be used only to present information relevant to working on the encyclopedia" is overkill and discourages people from forming friendly collaborative relationships. I think that we should move in the opposite direction, permitting and possibly even encouraging people to be social (within reasonable limits) while working collaboratively on our collective project of Wikipedia. Pine ------------------------------ Message: 3 Date: Wed, 21 Mar 2012 16:26:18 -0600 From: Will Takatoshi &lt;willtakatoshi(a)gmail.com&gt; To: foundation-l(a)lists.wikimedia.org Subject: Re: [Foundation-l] [Wikimedia Announcements] Fwd: Announcement: New editor engagement experiments team! Message-ID: &lt;CAF-2vByE2RDp7niHGx45gOtyaJv+K7_LFO-foSj-hpXrm6pvOQ(a)mail.gmail.com&gt; Content-Type: text/plain; charset=ISO-8859-1 There is no need for the Foundation to try to improve content quality. I keep careful tabs on quality studies and perform independent tests of Wikipedia quality regularly. By every measure, quality continues to improve, both organically from transient editors and structurally. Transient editors, whether registered or IP address users, have always been the largest source of the bulk of Wikipedia content, contrary to frequent claims that a core group writes most content. Certainly long term Wikipedians have large edit counts, but they represent a very small minority by total number of bytes added to articles. The evidence is detailed at http://www.aaronsw.com/weblog/whowriteswikipedia which is more true now than ever as transient editors are displacing long term frequent contributors on the largest wikipedias in article space. Structural quality improvements which have impressed me recently include the establishment of http://en.wikipedia.org/wiki/Wikipedia:Short_popular_vital_articles which in the 10 days that it has existed, more than 270 of its listed articles have been improved, each of which have gained an average of more than 150 bytes. At that rate, most of the level 4 vital articles will have more than 9,000 bytes of content in less than a year, as opposed to the prior rate of improvement which was closer to six years to meet the same goal. Another very impressive structural improvement involves User:Dispenser's enhancements to http://en.wikipedia.org/wiki/Category:Wikipedia_backlog where most of the article backlog count numbers are now clickable, such that they will show a list of the backlog category's articles sorted by importance, measured by the number of incoming links. For example, http://toolserver.org/~dispenser/cgi-bin/categorder.py?page=Category:All_un… As the number of incoming backlinks strongly correlates with the number of page views, this represents a quantum improvement for dealing with quality issue backlogs. There is no reason to believe that such organics and structural quality improvements will not continue. -Will ------------------------------ Message: 4 Date: Wed, 21 Mar 2012 15:32:28 -0700 From: Zack Exley &lt;zexley(a)wikimedia.org&gt; To: MZMcBride &lt;z(a)mzmcbride.com&gt; Cc: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Subject: Re: [Foundation-l] [Wikimedia Announcements] Fwd: Announcement: New editor engagement experiments team! Message-ID: &lt;CA+iYU95J-jJg9gsUqAG15bzMhS1LceWvZOGdC7MZHF8fGWALwg(a)mail.gmail.com&gt; Content-Type: text/plain; charset=ISO-8859-1 On Wed, Mar 21, 2012 at 3:30 PM, MZMcBride &lt;z(a)mzmcbride.com&gt; wrote: OK, don't know what you're talking about there... did moveon ever work on mandatory voting laws? but anyways... I agree that most new users are not high quality and many are spammers, PR people, band managers, etc... with little regard for the values of the projects. There are hundreds of thousands of such users each year. But the vast majority of new users have always been destined not to become great wikimedians. That's not new. But each year there has also been a large number (in the low thousands -- just guestimating) of new users who really want to be part of creating a great project and are fully aligned with the values of the project they're trying to join. When we look back at user-to-user interactions in 2001-2004, we see that established users had very high standards and were often unwelcoming or even rude, but they were putting effort into finding the needles in haystacks who would be great Wikimedians. They were saying over and over, "It's really hard to do what we do, but we're doing something amazing, if you stick around and learn the ropes, we could really use you." Today those kinds of communications happen much more rarely. My hunch is that templates caused that. Now, we just leave template messages instead of writing a personal note about a specific edit. I know the solution is not to just stop using templates. But I'm just trying to make clear (since you didn't hear it the first time I said it) that I wasn't arguing for coddling spammers or even investing time into encouraging all good faith users. There are a ton of amazing new users who make their 10th -- or 100th, or 1000th -- high quality edit every week. We just need to encourage them (instead of merely blanketing their talk pages with impersonal warnings). -- Zack Exley Chief Community Officer Wikimedia Foundation ------------------------------ Message: 5 Date: Wed, 21 Mar 2012 22:57:57 +0100 From: Kim Bruning &lt;kim(a)bruning.xs4all.nl&gt; To: rm(a)slmr.com, Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Subject: Re: [Foundation-l] Subject: Re: [WikiEN-l] Stopping the presses:, , Britannica to stop printing books Message-ID: &lt;20120321225757.A6572(a)bruning.lan&gt; Content-Type: text/plain; charset=us-ascii Ha! I'd choose Wikipedia2 anyday! ;-) (my favorite articles keep getting deleted from wikipedia. How's that useful to anyone?) Notability was originally a stopgap for verifiability IIRC. It's gone off the rails imo. On Wed, Mar 21, 2012 at 09:40:20AM -0700, Robin McCain wrote: I'm thinking wikipedia needs a reboot anyway. We'll probably end up with a replay of wikipedia/nupedia if we reboot a wp2 with tidied up and streamlined "policy" (redesign as a pattern language), integrated Prod/AFD, deprecated arbcom in favor of DRN, and most importantly: ensuring new users all get mentors. Acculturation failure has severely harmed WP1, we need some way to bring experienced and inexperienced users together reliably. This is the simplest and best way to retain editors. :-) sincerely, Kim Bruning ------------------------------ Message: 6 Date: Wed, 21 Mar 2012 22:49:48 +0000 From: David Gerard &lt;dgerard(a)gmail.com&gt; To: Wikimedia Foundation Mailing List &lt;foundation-l(a)lists.wikimedia.org&gt; Subject: Re: [Foundation-l] [Wikimedia Announcements] Fwd: Announcement: New editor engagement experiments team! Message-ID: &lt;CAJ0tu1HD8s91qgQG9sxGs4=DCXjAx2s98JvDkwEYp4wDWNFpaQ(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 On 21 March 2012 22:32, Zack Exley &lt;zexley(a)wikimedia.org&gt; wrote: And it turns out the new editors often assume the templates are completely bot-generated. That is: the editors using templates are, literally, failing the Turing test. I think it should be given serious consideration. I realise why Twinkle and Huggle exist, but they turn Wikipedia into a first-person shooter with the newbies as the targets. I suggest that this is not the sort of gamification that is useful. That said, anyone who's ever done Special:Newpages will deeply empathise with ax-crazy newpages patrollers, because Special:Newpages is a firehose of *shit*. How's the article wizard's output looking? - d. ------------------------------ _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/foundation-l

Message: 1 Date: Tue, 20 Mar 2012 22:24:24 -0700 From: Sue Gardner &lt;sgardner(a)wikimedia.org&gt; To: Wikimedia Announce Mailing List &lt;WikimediaAnnounce-l(a)lists.wikimedia.org&gt; Subject: [Foundation-l] [Wikimedia Announcements] Fwd: Announcement: New editor engagement experiments team! Message-ID: &lt;CAGZ0=LN=B+wmG1K2AfPWFxx4XnVfnxxG6mFUFQdu-H3LcnSrew(a)mail.gmail.com&gt; Content-Type: text/plain; charset="utf-8" Hey folks, I sent the note below to the staff and board a few hours ago: sharing now with everyone :-) Thanks, Sue ---------- Forwarded message ---------- From: Sue Gardner &lt;sgardner(a)wikimedia.org&gt; Date: 20 March 2012 19:17 Subject: Announcement: New editor engagement experiments team! To: Staff All &lt;wmfall(a)lists.wikimedia.org&gt; Hey folks, A couple of changes at the Wikimedia Foundation that I want you to know about. Everybody knows that reversing stagnating/declining participation in Wikimedia?s projects is our top priority. To make better progress, as of April 16 we're going to bring together resources from the Community and Engineering/Product departments into a new cross-functional team tasked specifically with conducting small, rapid experiments designed to improve editor retention. We already know some of the fixes that will solve the editor retention problem, and we're working to put them in place. The purpose of *this* team will be to identify the fixes we don't yet know about. Separately, Zack has to move back to Missouri for family reasons. When Zack told me about that, we agreed that it?s an extra impetus for this new team to be launched now. This means that going forward, ?Zack?s department will focus solely on fundraising, and some members of his department will move permanently into other groups. There have been lots of conversations about this over the past few weeks, which have included everyone affected. So here?s what we?re going to do: FUNDRAISING: Zack will manage fundraising remotely. He?ll continue to be part of the C-level team, but he?ll do it from Missouri. He?ll travel back to San Francisco frequently, and he?ll probably be here throughout the fundraising campaign every year and spend other longer chunks of time here when needed. We don?t yet know what the title of Zack?s department will be, or what Zack?s title will be. Neither Zack nor I care very much about titles, and we are in the happy position of not particularly needing to impress anyone -- so, we do not need fancy euphemistic titles. It would be nice to have titles that are clear and direct and understandable, and also to have ones that reflect the creative/storytelling/community aspect of the fundraising team?s work. So, we are leaving this piece open for the time being, and we?ll just call the department ?fundraising? until and unless we think of something better. Folks with suggestions should talk with Zack. :-) EDITOR ENGAGEMENT EXPERIMENTATION: Reflecting the importance of editor engagement in the Wikimedia Foundation?s strategy, we will have the following teams directly focused on it: ?**the Visual Editor group (led by Trevor as lead developer, and by the soon-to-be hired Technical Product Analyst) which is making the visual editor; ?**the Editor Engagement group (led by Fabrice Florin as Product Manager and Ian Baker as ScrumMaster) which is working on medium-term projects improving Wikimedia?s handling of reputation/identity and of notifications; ?**the new team focused on rapid experimentation, led by Karyn as Product Manager and a to-be-hired engineering lead/ScrumMaster, tentatively titled something like Research & Experimentation, Editor Engagement Innovation Lab or the Rapid Experimentation Team. Our thinking is basically this: we know the Visual Editor will help with editor retention. We know that improving notifications, messaging, identity and other core features of MediaWiki will help with editor retention. But there are a handful of other smaller projects --maybe just simple tweaks, maybe ideas that should become fully-fledged new features-- that will also help. The purpose of the new experimentation team will be to conduct many quick experiments, which will identify a handful of small changes that can either be accomplished by the team itself, or be queued up as part of our overall product backlog. Staff moving from the Community Dept to Engineering and Product Development (AKA Tech) are: Karyn Gladstone, Maryana Pinchuk, Steven Walling, and Ryan Faulkner. They will form a team tasked with rapid experimentation to find policy, product or other changes that will increase editor retention. Karyn will head product thinking and maintain the experimentation backlog, reporting to Howie. Alolita will hire and manage the engineers for this team, and will help interface them with the rest of the engineering organization. The important thing to know about this team is that they are being tasked with one of our absolutely most important objectives: to figure out new ways to increase editor engagement and retention. Karyn will report to Howie. Maryana, Ryan Faulker, and Steven will report to Karyn. The group has never had engineering resources assigned to it, and it?s clear they need engineering resources. Therefore, Alolita will work in close partnership with Karyn to recruit an engineering team --mostly developers but also UI/design people-- to support the new group. If you have ideas for people we should be recruiting for this, please tell Alolita or Karyn! Dario Taraborelli will join the editor engagement experimentation team as senior researcher and help design the roadmap and the individual experiments the team will run. We don?t yet have a firm title for the experimentation team, nor do we know yet what Karyn?s title will be, or whether other people?s titles (like, Steven or Maryana?s) will change. The team will figure this out, and to that end they?re kicking it around with other staff and with some folks in the community. FELLOWSHIPS: As most of you know, Siko runs our fellowship program. The fellowship program has lots of similarities to Asaf?s work on the grants program -- they are both, at heart, about giving funding and other support to members of the Wikimedia community to enable people to do useful work. The community-building projects that fellows often take on line up with some of Global Dev?s work, particularly as the fellowship program expands its global reach. So as we?ve been talking through Zack?s move and the implications for the Community department, it makes sense to shift Siko to Global Development. Siko?s title remains Head of Community Fellowships for now and she will report to Barry. Fellows and fellowship projects are continuing as planned, and you are still highly encouraged to keep an eye peeled for community members with good fellowship ideas. :) EDITOR RETENTION OVERALL: Finally, I want to talk for a minute about editor retention overall. As you know, we started the year with two major goals: the increase the number of mobile pageviews to two billion, and to push up the number of active editors to 95K. We?re doing fine on mobile reach (yay!) but we are completely failing to move the needle on the number of active Wikimedia editors. That doesn?t reflect poorly on the people who work on editor retention. It?s a complex problem that took a decade to develop, and the team doesn?t control all the variables affecting it. It makes sense that it will take time to fix. But it does mean that we need to increase the resources focused on it, so we can get more done faster. That?s what we?re doing here. We?re reorganizing to focus our existing resources more tightly, and we?ll also be adding new resources -- starting now, and continuing through the 2012-13 financial planning process. And, we?re going to move many/most of the editor-retention related people up to the 6th floor by the collab space. I really love the model Zack developed for the annual campaign -- the war room in Yongle, work visible on the walls for everyone, the buzz of people working hard towards a common purpose. I want us to have that same energy and momentum and focus for the editor retention work. Sorry for this long note, but I figure you will all be curious about this and have questions, so the goal here was to anticipate everything and get it answered up-front. This note was crafted collectively by many people :-) If questions remain, please feel free to ask them, or to talk with any of the individuals involved. And thanks to everyone who contributed to creating this plan: I very much appreciate everyone?s single-minded focus on attacking the editor retention problem, and I look forward to future success moving the needle on it. Everything I talk about in this mail will take effect April 16. Once it?s in your in-box, it?s no longer confidential, and you can feel free to talk about it publicly. I will forward it to announce-l, after I give you a couple of hours to read it yourselves. And please join me in congratulating the folks who are going to work on this important new team :-) Thanks, Sue Q What?s the impetus behind these changes? A Two things. Mainly, we want to redouble our focus on attacking the editor retention problem, and it makes sense for us therefore to consolidate our efforts into a single focused mega-team. Secondarily, Zack has decided he needs to relocate to Missouri. We had already been talking about whether consolidation made sense -- with Zack moving, that accelerated those conversations. Q What?s happening to Zack? A Zack will lead our fundraising remotely, as a C-level employee. His title and the title of his team will change to reflect that, but no final decisions have been made yet about what those titles will be. Ideally we?d like to have a title for that department, and for Zack, that reflects the storytelling aspect of their work, telling the community?s stories to the world. But in the end we may settle for just calling it Fundraising, if we don?t think of anything better. Q What?s happening to the Community Department? A We initially created a community department because it made sense to have focused resources dedicated to understanding the community and being a centre of expertise about it for the Wikimedia Foundation. At the time that was the right thing to do, because although some individual staff members had lots of community understanding, the organization as a whole did not -- which meant it made sense for us to focus our energy, for a time, on researching and documenting and analysing the community. But having a Community Department was never a perfect fit for the Wikimedia Foundation the way it is for other internet companies, because community is not a small subset or a single aspect of what we do at the Wikimedia Foundation --- all our departments have interactions with community members for multiple different purposes, and over time we have been growing specific community expertise and responsibilities in multiple departments throughout the organization. As expertise grew elsewhere, having a community department became a less-good fit for us. Basically: it made sense to have a Community Department at the time when we did it, and it makes less sense now as the organization has matured and evolved. Q Why are you integrating the Karyn/Ryan/Steven/Maryana group into Engineering and Product Development? A The goal is to create a better model for rapid experimentation/innovation, with minimal hindrance to the work of our active editors and maximal gain in new community members. That group is not necessarily a perfect fit with Engineering and Product Development, but we think that?s okay: it?s a good fit, and being in that department will enable the team to increase its impact overall, by giving it better access to UI/design and engineering resources. Q How is the new product team different from Fabrice?s team? A The new team won?t focus on critical major changes to the platform (like the Visual Editor) or critical but equally complex projects like improving the mechanisms by which editors communicate and collaborate on the projects. This team will be much more fast-paced and experimental, identifying small-scale interventions that might make an impact on editor retention and quickly iterating through them on a weekly (or even daily) basis. Unlike Fabrice?s team, which has a list of projects that are known to be important and impactful, this team will quickly cycle through a large number of ideas that have not yet been tested in order to identify what does and doesn?t work, and what can be integrated into existing product roadmaps. Q Will this new team be building new features? A No. Ideas for new features that come out of successful experiments will be handed off to Fabrice?s team or elsewhere in engineering/tech as is appropriate for the task. Q What kinds of projects will this new team be working on? A Some of our projects will be similar to the template A/B testing conducted by Steven, Maryana, and Ryan Faulkner: small tweaks to existing community-built systems like template messaging. Others will focus on more innovative ways to engage new and current editors, using notifications, task assignment, and different kinds of incentives to keep editing. All of the projects will be temporary tests, not permanent large-scale changes, and focus on measuring effects to inform further decisions. Q How can volunteers give input on this work? A Ping Steven or Maryana. Q Who will be in charge of this work? A The C-level in charge of this team is Erik Moeller. The team reports to Karyn Gladstone. Q Where will fellowships live and how do fellows fit with other teams? A The fellowships program will move to Global Development, but the structure and scope of current and planned fellowships will not change. Fellows will continue to be recruited from the community to work on their own projects, supported by Siko Bouterse, Head of the Fellowships Program. Q Where will other community projects live (e.g., summer researchers/analytics, community convenings)? A Current community department projects will be integrated into either the new team or other teams in the organization, depending on their purpose. Convenings of various kinds will be staged on an ad hoc basis by multiple groups, including this one. Q What?s happening to fundraising and storytelling? A The fundraising team, which includes storytelling, will be managed by Zack and will continue to operate as planned. Q Why is this all so confusing? The Wikimedia Foundation changes people?s titles and reporting lines all the time! A Yes, we do :-) The Wikimedia Foundation is a pretty young organization: it?s growing, and doing lots of experimentation. We learn new things all the time, and we want to be able to apply what we?re learning, which includes restructuring/reinventing ourselves. If we were a hundred-year-old organization, or if we had tens of thousands of employees, it would be hard for us to adapt and change, because there would be too many layers of people who would need to be involved, and the downstream implications of even small changes would be serious. That?s a problem for big/old organizations, because it limits their adaptability. Luckily, at this point the Wikimedia Foundation is still small enough and young enough that we can afford to be reasonably flexible. That said, we know that this kind of change can be confusing for people who aren?t involved (at a minimum, it?s one more long e-mail to read), so we appreciate everybody?s patience :-) Q How can I join this team? A We?re hiring! We?re looking for more experienced editors to help us design experiments, track results, and communicate what we?re doing. We are also looking for strong front-and back-end developers to deploy experiments. If you?re interested in working with us, please check out available positions on the Wikimedia Foundation jobs page: http://wikimediafoundation.org/wiki/Job_openings -- ?Sue Gardner Executive Director Wikimedia Foundation 415 839 6885?office 415 816 9967?cell Imagine a world in which every single human being can freely share in the sum of all knowledge.? Help us make it a reality! http://wikimediafoundation.org/wiki/Donate _______________________________________________ Please note: all replies sent to this mailing list will be immediately directed to Foundation-L, the public mailing list about the Wikimedia Foundation and its projects. For more information about Foundation-L: https://lists.wikimedia.org/mailman/listinfo/foundation-l _______________________________________________ WikimediaAnnounce-l mailing list WikimediaAnnounce-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimediaannounce-l ------------------------------ _______________________________________________ foundation-l mailing list foundation-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/foundation-l End of foundation-l Digest, Vol 96, Issue 68 ********************************************