On Thu, Oct 9, 2014 at 10:26 AM, Pine W wiki.pine@gmail.com wrote:
I'm sure a Board member, Lila, or Erik will correct me if I am mistaken, but my understanding is that there is internal agreement at Board level that the Product side of the org needs some systemic changes, that Lila was chosen with the goal of making those changes, and that some changes are already happening.
There's agreement at all levels that we want to continue down the path set by Sue back in 2012 [1] for WMF to truly understand itself as a technology and grantmaking organization. That path led to where we are today:
1) As part of the ED transition, Sue recommended (and the Board accepted the recommendation) to seek an ED with a strong technology/product background, and we hired Lila Tretikov as Sue's successor who matches those requirements.
2) In November 2012, I recommended that we prepare for building out new functions for UX and Analytics, and prepare for dedicated leadership for Engineering and Product. Sue accepted this recommendation. I hired Directors for UX and Analytics in 2013, followed by Community Engagement in 2014, and finally we hired a VP Engineering last week to complete the process.
3) To better account for the need to learn quickly and adjust course as appropriate, we introduced quarterly reviews in December 2012 [3] and increasingly reduced the specificity of Annual Plan level commitments while increasing the focus on metrics and accountability in the reviews.
4) On the technology and product front, many improvements to process and support infrastructure have been implemented in the last couple of years, including but not limited to:
- Development of MediaWiki Vagrant as a standardized dev environment, to reduce failure cases due to developer environment inconsistencies - Improvements to continuous integration infrastructure for PHP unit tests and QUnit JavaScript unit tests, and increased focus (but not nearly enough yet) on automated tests, especially for newly developed features - Introduction and continued improvement of BetaLabs as a staging environment for all commits, increased use of automated end-to-end browser tests and QA testing by humans to catch bugs and regressions prior to production rollouts - Introduction and use of various tools for measuring the impact of features, including EventLogging as a standard instrumentation framework for measuring feature usage, dashboards for visualizing usage, WikiMetrics for analyzing editor cohort behavior, Editor Engagement Vital Signs for understanding system-wide user behavior, analysis of pageview data using Hadoop (just rolled out), etc. - Highly specialized automated testing frameworks for specific projects, e.g. Parsoid round-trip testing and visual diffing (!) to detect dirty diffs or output problems - Introduction of design research as a discipline in the UX team (through hiring of Abbey Ripstra as User Research Lead) and incorporation of user studies in a much more systematic way across products - Community liaisons dedicated to key products, responding to user feedback and helping Product Managers understand more complex community needs - Continued shortening of release/deployment cycles; significant improvements to deployment tooling, rewriting our legacy "scap" tools to increase the ability to monitor and reason about deployments; introduction of daily "SWAT" deploys to quickly release fixes, etc. - Introduction of various infrastructure tools that help us better analyze/profile issues, including logstash for log analysis, increased use of graphite for performance metrics collection and various front-ends for visualizing those metrics - Shift towards loosely coupled services, addressing the difficulty of maintaining and improving our highly monolithic codebase (examples include Parsoid, Citoid, Mathoid, and the new Content API in development) - Introduction of Beta Features framework to stage features for early adopters
5) The changes Lila has pushed for since we started include:
- Greater focus on quarterly prioritization and a "rolling roadmap" rather than a fiscal year view of the world - Increased emphasis on understanding the needs of different user personas at all cycles of software development, including through use of qualitative and quantitative methods - Reducing velocity of user-facing changes (esp. on desktop) to increase focus on foundations (platform/process improvements) that ultimately will enable us to move faster and more effectively - Documenting product development methodology on-wiki and establishing a clearer social contract (to reduce the reliance on RFCs/votes regarding feature configurations) - Surveying the needs of current users to more systematically balance projects that serve future/new users vs. projects that serve the users we have today - Improved communication channels for community engagement to make it easier to understand what major projects are currently in development and how to provide feedback
This already means, effectively, that the commitments in the Annual Plan developed during Sue's time should be taken with a big block of salt at this point in time -- we're slowing down the deployment (not development) of big user-facing features like Flow and VE as much as needed to ensure that we incorporate user feedback, data and qualitative research into the product development process appropriately and spend sufficient time on the technical foundations for these projects.
The quarterly prioritization alone has been, IMO, a huge improvement that's already paying off. In the "Annual Plan" view of the world, it's unlikely that we would have prioritized a project like HHVM the way we did, because we were generally stuck on the priorities set for the whole year. But it was very clear that this project would provide huge benefits to our users, and I'm glad we were able to call it out as _the_ top priority for Q1 and give the team the space to really focus on getting it done (almost there now, starting to serve reader traffic [4]).
Our draft Q2 top priorities (not yet posted on-wiki, but discussed in the metrics meeting last week) are consistent with the above, with the main user-facing push being on mobile web/apps and editing performance, while the other priorities are more platform/process-related. Once again, we're continuing to work on VE / Flow, but focusing more on fundamentals (performance, architecture, testing, use case analysis, etc.) than accelerating deployments.
My focus over the coming days is to flesh out the details for the Q2 priorities, and then shift to putting more effort in documenting and refining product development methodologies and processes on-wiki. On the engineering side, there's plenty of process/infrastructure improvement to do as well. From my point of view, continued improvement to test coverage and CI/testing infrastructure, developer tools, profiling/instrumentation, staged roll-out support and strengthening of architectural leadership are the big pieces for coming months, but I'll let Damon speak to his focus areas as he gets the lay of the land.
Erik
[1] https://meta.wikimedia.org/wiki/User:Sue_Gardner/Narrowing_focus [2] https://lists.wikimedia.org/pipermail/wikimedia-l/2012-November/122663.html [3] https://lists.wikimedia.org/pipermail/wikimedia-l/2012-December/123088.html [4] https://gerrit.wikimedia.org/r/#/c/165004/