Thanks for the detailed comments, Erik.
As someone who spent several volunteer hours reviewing the current Annual
Plan, I would appreciate getting an understanding of how the change of
emphasis to quarterly reviews affects budgets, hiring plans, and
fundraising goals. Is that something that you can address or should I ask
Garfield?
Thanks,
Pine
On Oct 9, 2014 4:44 AM, "Erik Moeller" <erik(a)wikimedia.org> wrote:
On Thu, Oct 9, 2014 at 10:26 AM, Pine W
<wiki.pine(a)gmail.com> wrote:
I'm sure a Board member, Lila, or Erik will
correct me if I am mistaken,
but my understanding is that there is internal agreement at Board level
that the Product side of the org needs some systemic changes, that Lila
was
chosen with the goal of making those changes, and
that some changes are
already happening.
There's agreement at all levels that we want to continue down the path
set by Sue back in 2012 [1] for WMF to truly understand itself as a
technology and grantmaking organization. That path led to where we are
today:
1) As part of the ED transition, Sue recommended (and the Board
accepted the recommendation) to seek an ED with a strong
technology/product background, and we hired Lila Tretikov as Sue's
successor who matches those requirements.
2) In November 2012, I recommended that we prepare for building out
new functions for UX and Analytics, and prepare for dedicated
leadership for Engineering and Product. Sue accepted this
recommendation. I hired Directors for UX and Analytics in 2013,
followed by Community Engagement in 2014, and finally we hired a VP
Engineering last week to complete the process.
3) To better account for the need to learn quickly and adjust course
as appropriate, we introduced quarterly reviews in December 2012 [3]
and increasingly reduced the specificity of Annual Plan level
commitments while increasing the focus on metrics and accountability
in the reviews.
4) On the technology and product front, many improvements to process
and support infrastructure have been implemented in the last couple of
years, including but not limited to:
- Development of MediaWiki Vagrant as a standardized dev environment,
to reduce failure cases due to developer environment inconsistencies
- Improvements to continuous integration infrastructure for PHP unit
tests and QUnit JavaScript unit tests, and increased focus (but not
nearly enough yet) on automated tests, especially for newly developed
features
- Introduction and continued improvement of BetaLabs as a staging
environment for all commits, increased use of automated end-to-end
browser tests and QA testing by humans to catch bugs and regressions
prior to production rollouts
- Introduction and use of various tools for measuring the impact of
features, including EventLogging as a standard instrumentation
framework for measuring feature usage, dashboards for visualizing
usage, WikiMetrics for analyzing editor cohort behavior, Editor
Engagement Vital Signs for understanding system-wide user behavior,
analysis of pageview data using Hadoop (just rolled out), etc.
- Highly specialized automated testing frameworks for specific
projects, e.g. Parsoid round-trip testing and visual diffing (!) to
detect dirty diffs or output problems
- Introduction of design research as a discipline in the UX team
(through hiring of Abbey Ripstra as User Research Lead) and
incorporation of user studies in a much more systematic way across
products
- Community liaisons dedicated to key products, responding to user
feedback and helping Product Managers understand more complex
community needs
- Continued shortening of release/deployment cycles; significant
improvements to deployment tooling, rewriting our legacy "scap" tools
to increase the ability to monitor and reason about deployments;
introduction of daily "SWAT" deploys to quickly release fixes, etc.
- Introduction of various infrastructure tools that help us better
analyze/profile issues, including logstash for log analysis, increased
use of graphite for performance metrics collection and various
front-ends for visualizing those metrics
- Shift towards loosely coupled services, addressing the difficulty of
maintaining and improving our highly monolithic codebase (examples
include Parsoid, Citoid, Mathoid, and the new Content API in
development)
- Introduction of Beta Features framework to stage features for early
adopters
5) The changes Lila has pushed for since we started include:
- Greater focus on quarterly prioritization and a "rolling roadmap"
rather than a fiscal year view of the world
- Increased emphasis on understanding the needs of different user
personas at all cycles of software development, including through use
of qualitative and quantitative methods
- Reducing velocity of user-facing changes (esp. on desktop) to
increase focus on foundations (platform/process improvements) that
ultimately will enable us to move faster and more effectively
- Documenting product development methodology on-wiki and establishing
a clearer social contract (to reduce the reliance on RFCs/votes
regarding feature configurations)
- Surveying the needs of current users to more systematically balance
projects that serve future/new users vs. projects that serve the users
we have today
- Improved communication channels for community engagement to make it
easier to understand what major projects are currently in development
and how to provide feedback
This already means, effectively, that the commitments in the Annual
Plan developed during Sue's time should be taken with a big block of
salt at this point in time -- we're slowing down the deployment (not
development) of big user-facing features like Flow and VE as much as
needed to ensure that we incorporate user feedback, data and
qualitative research into the product development process
appropriately and spend sufficient time on the technical foundations
for these projects.
The quarterly prioritization alone has been, IMO, a huge improvement
that's already paying off. In the "Annual Plan" view of the world,
it's unlikely that we would have prioritized a project like HHVM the
way we did, because we were generally stuck on the priorities set for
the whole year. But it was very clear that this project would provide
huge benefits to our users, and I'm glad we were able to call it out
as _the_ top priority for Q1 and give the team the space to really
focus on getting it done (almost there now, starting to serve reader
traffic [4]).
Our draft Q2 top priorities (not yet posted on-wiki, but discussed in
the metrics meeting last week) are consistent with the above, with the
main user-facing push being on mobile web/apps and editing
performance, while the other priorities are more
platform/process-related. Once again, we're continuing to work on VE /
Flow, but focusing more on fundamentals (performance, architecture,
testing, use case analysis, etc.) than accelerating deployments.
My focus over the coming days is to flesh out the details for the Q2
priorities, and then shift to putting more effort in documenting and
refining product development methodologies and processes on-wiki. On
the engineering side, there's plenty of process/infrastructure
improvement to do as well. From my point of view, continued
improvement to test coverage and CI/testing infrastructure, developer
tools, profiling/instrumentation, staged roll-out support and
strengthening of architectural leadership are the big pieces for
coming months, but I'll let Damon speak to his focus areas as he gets
the lay of the land.
Erik
[1]
https://meta.wikimedia.org/wiki/User:Sue_Gardner/Narrowing_focus
[2]
https://lists.wikimedia.org/pipermail/wikimedia-l/2012-November/122663.html
[3]
https://lists.wikimedia.org/pipermail/wikimedia-l/2012-December/123088.html
[4]
https://gerrit.wikimedia.org/r/#/c/165004/
--
Erik Möller
VP of Product & Strategy, Wikimedia Foundation
_______________________________________________
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>