TLDR: How fast should new content (maps) features be rolled out, and how ready should they be? Constant but smaller improvements seems better.
As we gradually roll out maps to wider and wider audience, I would like to get some feedback on how we approach new feature roll-outs.
Frankly, WMF has botched a number of releases in the past. In a way, this is great, because it means both WMF and volunteers are still eager to improve Wikipedia, we are still trying to make things better. On the other hand, it is never a good thing to irritate the most important group - our community. So there is always a compromise: when and what to roll out, how to make it least disruptive vs how to improve the usability and the content quality.
For wikis, there are two types of improvements: user interface and content. User interface features change how one views and edits site's content, so any change immediately affects everyone. We try to mitigate it with "Beta features" -- logged in users may enable new functionality before it is enabled by default for everyone, but the vast majority of the readers are not logged in, so when enabled, it is still a serious and instant change. Maps are not really a part of interface because they only appear as part of the page content, which is fully controlled by the community.
Content features allow editors to add new type of content - maps, graphs, sheet music, text formatting or new types of templates. There is no way to hide a new content feature behind a "beta flag" because everyone sees the same content, but content features are not disruptive because they depend on the editors to add them to the pages. Community has full control, and if it does not like a feature, or if it feels the feature is not ready yet, it will not be used. The only time content changes are disruptive is when the support for a widely used feature changes or gets disabled. This is clearly not the case with the maps roll out on Wikipedia.
The WOW effect, and marketing in general, have both positive and negative effect on a feature roll out. If a feature is quietly enabled, only the more engaged community members will experiment with it and discuss the feature's best usage, give feedback on how to improve it, and eventually enable it at its own pace. A massive marketing of a feature would attract a lot of attention and expedite adaption, but may also create some amount of negativity if the community feels the feature is not yet ready.
I also feel it is very dangerous to delay releasing new features until they are perfect. We (developers/PMs/...) may think we know what feature is needed, but most likely we are wrong. If we delay, we may spend a lot of resources on polishing something that is not needed. Instead, by releasing early, community's feedback would put us back on the right track. Yes, it may not be as good, but at least we will quickly change direction, producing a truly needed feature that can be polished later. Of course this is much easier to do with the content features rather than user interface changes (hence the UI's "feature flag").
In light of this, I feel it is better to continuously roll-out small content-related features without much publicity (e.g. Village pump is OK, blog might be less so), and continuously improve based on community feedback. Once the feature has been out for some time and there is a general consensus that the feature is good, we can start the marketing push. This approach creates less stress on the community lesions, developers, and servers. Feedback is received in smaller portions and can be properly acted on.
Disclaimer: I'm an Agile Coach at the WMF, and have worked with Discovery a lot, but only a little bit with the interactive team. These thoughts are my own; I am not speaking for the foundation, nor for any department or team.
On Sat, Nov 5, 2016 at 6:26 PM, Yuri Astrakhan yastrakhan@wikimedia.org wrote:
TLDR: How fast should new content (maps) features be rolled out, and how ready should they be? Constant but smaller improvements seems better.
My tl;dr: "It's complicated."
For those who may have been confused about the context of this email, it is a continuation of some internal team/vertical conversations we have been having. Those debates have not been as simple as "fast vs. slow" or "release incrementally or wait until it is perfect". Everyone I have spoken to is in favor of fast and incremental development.
So there is always a compromise: when and what to roll out, how to make it least disruptive vs how to improve the usability and the content quality.
I agree, although other groups besides communities can be disrupted. Notably other WMF teams like legal, Visual Editor, ops, and others. Finding the right balance between all the stakeholders can be challenging.
Maps are not really a part of interface because they only appear as part of
the page content, which is fully controlled by the community.
This is a reasonable basic framing, but there is a lot of grey. The moment an editor adds a map to a page, that map becomes part of the interface. Any changes to the maps code will alter the user experience of that page. If Visual Editor needs a new feature to support maps, then all editors (even first-time/casual editors) will experience the pros and cons of that new editing feature.
content features are not disruptive because they depend on the editors to add them to the pages.
I disagree with that blanket statement, for reasons given above.
Community has full control, and if it does not like a feature, or if it feels the feature is not ready yet, it will not be used.
"Community" is vague and amorphous. Some editors might love a content features, while some readers might hate it. Or some people might love this month's release, but hate the update a month later. Once a feature is in use, removing even small aspects of it can be very contentious.
I also feel it is very dangerous to delay releasing new features until they are perfect.
I believe everyone in Discovery, and most of Engineering, agree with that statement. "Release early and often" is pretty well-accepted here, I think. The question becomes: How early, and how often?
In light of this, I feel it is better to continuously roll-out small content-related features without much publicity (e.g. Village pump is OK, blog might be less so), and continuously improve based on community feedback. Once the feature has been out for some time and there is a general consensus that the feature is good, we can start the marketing push. This approach creates less stress on the community liaisons, developers, and servers. Feedback is received in smaller portions and can be properly acted on.
A given feature could be somewhere on the complete <-> incomplete spectrum, but it is also somewhere on the buggy <-> bugfree spectrum. Releasing incomplete features quickly is a pretty easy sell. Releasing buggy features can be problematic. In addition to "features" and "bugs", there are legal, scalability, reliability, usability, and accessibility issues. They all combine into the magic question: Is some code *ready* to be released.
In order to answer that question, the team needs some processes and norms. Design research, focus group testing, RFCs, conversations with various communities, Automated testing, manual testing, load testing, staging environments, beta features, and small-wiki rollouts can all be part of a strategy to release quality software quickly. The interactive team is still relatively new, and has been moving very fast, so it is still figuring out its strategy with all of this.
As for publicity: I can understand not blogging about a new feature until it has proven its value. But my own naive inclination is to be very open in communicating about upcoming plans, imminent features, and just-released features. Village pump is one channel, and mailing lists would be another. Of course, phabricator is also a channel, but it is extremely difficult for most people outside the development team to follow. Personally, I think communities should have a chance to know what is coming rather than only being able to respond after it has been deployed.
To conclude, these are useful conversations about important topics. There are no easy answers, and generally there are not single "correct" answers. Let's acknowledge that it's messy.
Kevin Smith Agile Coach, Wikimedia Foundation