Hi,
The report covering Wikimedia engineering activities in October 2013 is now available.
Wiki version: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/October
Blog version: https://blog.wikimedia.org/2013/11/12/engineering-report-october-2013/
We're
also proposing a shorter, simpler and translatable version of this
report that does not assume specialized technical knowledge:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/October/summary
Below is the HTML text of the report.
As always, feedback is appreciated on the usefulness of the report and its summary, and on how to improve them.
------------------------------------------------------------------
Major news in October include:
Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.
Personnel
Are you looking to work for Wikimedia? We have a lot of hiring coming
up, and we really love talking to active community members about these
roles.
Announcements
- Ori Livneh transitioned to the Platform Engineering group as Senior Performance Engineer (announcement).
- Gergő Tisza joined the Platfom Engineering group as Software Engineer on the Multimedia team (announcement).
- Leslie Carr and Ryan Lane were both promoted to the position of Senior Operations Engineer (announcement).
- Rummana Yasmeen joined the Platfom Engineering group as Software
Test Engineer on the QA team, working primarily on VisualEditor (announcement).
- Vibha Bamba was promoted to the position of Senior User Experience Designer (announcement)
Technical Operations
Site infrastructure
- The team continued to heavily refactor the Puppet configuration:
manifests for MySQL, nginx, SSH, puppetmaster, and several others are
now properly organized as modules. Alexandros Kosiaris has made
considerable progress towards supporting multiple puppetmasters on our
cluster, which will greatly improve Puppet performance.
Wikimedia Labs
- Andrew Bogott has been working with Yuvaraj Pandian to get the new
proxy system properly deployed; this will greatly reduce our need to
hand out public IPs to labs users. We're close to hiring a contractor to
help with the upcoming migration of Labs from Tampa to Ashburn.
Editor retention: Editing tools
VisualEditor
In
October, the VisualEditor team continued to improve the stability and
performance of the system, and add new features. The deployed version of
the code was updated five times (
1.22-wmf20,
1.22-wmf21,
1.22-wmf22,
1.23-wmf1 and
1.23-wmf2).
Beyond fixing bugs, the focus of the team's work this month was to make
a number of large structural changes to make the system more dependable
and extensible, and continue to make some usability improvements. For
example, you now need to press the "delete" key twice to delete a
template, reference or image; the first time, they only become selected,
to avoid accidental deletion of infoboxes and similar content. A new
feature, empowering users to switch from editing in VisualEditor to
editing wikitext directly without having to save the page, was also
implemented.
Parsoid
In
October, the Parsoid team continued to refine the parser behavior in
edge cases. Performance was improved by increasing the parallelism of
API requests and separating page updates from dependency-triggered
updates in the job queue. The
round-trip testing server
performance was improved so that we can now run round-trip tests on
160,000 pages over night. Support for private wikis was also added this
month. We also made additional progress on
Rashomon,
the revision storage service based on Apache Cassandra. Rashomon is
initially going to be used for implementing HTML and metadata storage
for Parsoid output. Rashomon was deployed on a test cluster and
import/write tests were performed.
Core Features
Notifications
In October, we
released Notifications worldwide
on about 800 different sites, including most of the remaining Wikipedia
wikis and 'sister projects'. Fabrice Florin and Keegan Peterzell
managed the community outreach for these final releases, while Benny
Situ proceeded with the technical deployments. Community members we've
spoken to generally find this tool helpful, across languages and world
regions. Notifications are also available on mobile devices, and seem to
be adding value on these platforms as well. We now plan a final release
of Notifications on the German and Italian editions of Wikipedia at the
end of November. To learn more, visit our
project hub, read the
help page and join the discussion on the
talk page.
Flow
Growth
Growth
Support
2013 Wikimedia fundraiser
The
fundraising team has been steadily ramping up to the big English
fundraiser, and so the fundraising tech team has been transitioning into
their usual year-end fundraising monitoring and support role. At the
end of October, we went in to a feature freeze for CentralNotice, and
have been concentrating on bugfixes with the intent of increasing
stability on the payments cluster and in the rest of the donation
pipeline. We have also started planning for the international
fundraising campaigns targeted for early 2014.
Wikipedia Education Program
This
month, we fixed several bugs in the Education Program extension and
added one much-needed UI feature. More improvements are in the pipeline.
For now, we're focusing on issues that are not linked to the
extension's lack of integration with ContentHandler, and on features
that might carry over to a new version of the extension. We've also
begun the groundwork for redesigning the UX and broadening the
extension's scope.
Commons App
Many
changes were made to the iOS Commons app in October. Users can now
select the license among the most common ones, and swipe to delete
categories on yet-to-be-uploaded images. A number of visual and
interface improvements have been made, including on the settings page
and the image details slider, notably for iOS 7 compatibility. Icon
consistency has been improved throughout the app. The app now also has
better internationalization support (via Autolayout), as well as better
landscape and iPad support.
Wikipedia Zero
This
month, the team released a wide array of features and enhancements.
Part of the work focused on reducing accidental data charges, for
example by adding interstitials and suppressing UI chrome icons. The
latter change caused a 30-minute outage of JavaScript and CSS assets on
Wikidata.
We started redirecting ineligible access attempts against subdomains
zero.wikipedia.org to a unified warning page; this will reduce residual
content in search engine caches, and reduce charges for users
accidentally visiting zero.wikipedia.org subdomains while off
participating networks.
We added support for partner self-management of configuration for
JSON-oriented users; this will be enhanced with stage gating before any
actual push to production.
Last, we nearly finalized IP address zero-rating lists (with the support
of the Operations team), and we've also started working on Wikipedia
Zero support for all Wikimedia projects.
Mobile web projects
The
mobile web team has been focusing on a variety of new features, as well
as running tests and gathering metrics on new and new-ish mobile
editors. The team has also been working on general design enhancements
to improve the user experience and make the look and feel consistent
across all aspects of the mobile experience.
Feature highlights from the past month include: a better AbuseFilter
support for mobile editing, early experimentation around integrating
VisualEditor with MobileFrontend for tablets, client-side performance
enhancements, the ability to 'thank' from the watchlist, and Captcha
support for mobile editing.
In beta, 'Near this page' is now available, as well as user profiles and
'Keep Going' (a series of calls to action for new editors).
- The Language Engineering team worked on building a new Main Page for
translatewiki.net, the primary translation platform for MediaWiki
projects. This includes a new sign-in screen and project displays. A
special feature is also being introduced for new translators through
which they can provide a limited set of translations for evaluation.
- As part of the enhancements towards Universal Language Selector (ULS), a new font has been developed
for displaying Language Names in their own scripts. This is known as
the Autonym font and contains only the characters needed for the
language names. Significantly small in size, this font will soon be
deployed on Wikimedia projects. The team also worked on fixing a font
inheritance issue for the edit area (bug 53734).
- The monthly release of the MediaWiki Language Extension Bundle (MLEB) was also completed.
- The team prepared for the open-source language summit
to be held in Pune, India on November 18−19 in collaboration with Red
Hat and other open-source internationalization developers.
MediaWiki Core
MediaWiki 1.22
MediaWiki 1.23
Site performance and architecture
We implemented logging, aggregation & graphing of the VisualEditor DOM load & save timing. We also rolled out mw.inspect,
a library for inspecting static asset metrics. We configured stable
URLs and improved cache headers for font resources, and rolled out a
localStorage module caching to test wikis and the beta cluster.
Admin tools development
Search
In
October, CirrusSearch was deployed as a secondary search engine to
Wikidata, all Wikivoyage wikis, and Wikipedia in Bengali. It became the
primary search engine on Wiktionary in Italian, Wikipedia in Catalan and
Wikisource in English. In November, we plan to deploy many more wikis
including some larger than the Catalan Wikipedia. To expand to those
larger wikis, we've negotiated some new hardware that should be deployed
mid month.
Auth systems
We
continued to refine the OAuth UX with the design team, and completed
all major development tasks for the initial OAuth product. The first
third-party application approved to use OAuth, "Gerrit Patch Uploader",
was successfully used by several end users. We plan to finish the OAuth
deployment in November.
Wikimania Scholarship app
A
two-week cleanup sprint began on October 23, with the goal of having a
functioning version of the existing application running in Labs with
major code cleanliness and security concerns addressed. At the close of
October, approximately half the listed tasks were completed, but these
tasks encompass only about one third of the total work. The project
should be in good shape for a progress review around November 8.
Security auditing and response
We responded to several issues reported in core and extensions. An emergency password reset was put into place to address a
private data security issue.
Quality assurance
Quality Assurance
In
October the Quality Assurance project brought on a new software tester
for VisualEditor, Rummana Yasmeen, who became productive quickly,
reporting a number of issues and validating a VE deployment. QA spent a
significant amount of time with the WMF Language team, using automated
browser tests to find issues with the UniversalLanguageSelector and
other Language software. QA continues to expand the utility of the beta
labs test environment, supporting the new Flow extension there, and
increasing the role of beta for deployments. Membership in the QA
mailing list continues to grow, and October saw more contributions to
the automated tests from volunteers. QA will participate in the Google
Code In project as well.
Beta cluster
Continuous integration
October
has been dedicated to consolidating the Jenkins configuration to make
it easier to edit. Most actions are now handled by shell scripts under
integration/jenkins.git:/bin;
editing the scripts doesn't require updating Jenkins jobs. The second
slave server has been added to production and is successfully running
PHPUnit tests. The packaging of dependencies required to upgrade Zuul
has been completed, and Antoine Musso now has a version working in Labs.
Finally, we investigated the possibility of running the browser tests
whenever a change is submitted in Gerrit; that work is still in
progress. Thanks to
Carl Fürstenberg's
work during the Summer, we are now able to build some Debian packages
straight into Jenkins using a dedicated instance and the
Jenkins Debian glue scripts. The jobs are
listed in Jenkins under the Ops-DebGlue view.
Browser testing
October
saw a major change to the architecture of the browser tests. Since they
are now in many different repositories (VisualEditor,
UniversalLanguageSelector, MobileFrontend, Flow, etc.), we consolidated
the code shared among the various tests in all the repositories into a
single library. This allows all the tests, present and future, to use
the best and most up-to-date infrastructure. As for the tests
themselves, we expanded coverage and identified issues for VisualEditor,
Flow, gadgets, Language features, MobileFrontend, and more.
Bug management
Bugzilla now offers a
new guided bug entry form which will make creating good bug reports easier for newcomers. Bugzilla now also
displays metadata changes
of a bug report inline for all logged-in users, so they can see in the
comments who changed a value of a field (without clicking on "History").
Daniel Zahn upgraded Wikimedia Bugzilla to latest version
4.2.7. Legoktm
mass-imported about 400 Pywikibot tickets from Sourceforge to Bugzilla. On a related note, Amir ran a
PyWikibot Bug Triage
resulting in nearly 100 tickets receiving updates. Furthermore, Andre
Klapper investigated Wikimedia Bugzilla's customizations in CSS and code
in order to clean up and sync with the upstream code base, to simplify
current maintenance and also make a potential future upgrade of
Wikimedia Bugzilla from version 4.2 to 4.4 easier.
Mentorship programs
Technical communications
Volunteer coordination and outreach
Multimedia
Multimedia
Kraken
The
team designed and implemented multiple data center configuration
support for Kafka (message bus). A bug involving buffer space allocation
was exposed and fixed in the Varnish module, and infrastructure work
done on automated data ingestion and partitioning. Product Analyst
Oliver Keyes did some exploratory work with Hive and provided feedback
to the Development team on ease of use and use cases.
Wikimetrics
The Threshold metric was implemented, and numerous bugs were fixed.
Data Quality
We
created dashboards for several Wikipedia Zero partners (Orange
Madagascar, Bangalink, Umniah Jordan), and identified and fixed
Wikipedia Zero data issues in collaboration with the Zero team.
Research and Data
This month, we continued to support
Growth and
Mobile as the team's focus areas for this quarter. We published the
results of the latest
GettingStarted test run by the Growth team, we completed the
cohort analysis for Mobile user acquisition and we worked with the Mobile team to prepare the launch a new test for
new user activation, currently underway.
We analyzed active editor trends to determine whether the September 2013 total active editor data
represented an anomalous change from seasonality and the long-term
trend, and concluded that this was not the case. The results of this
analysis call for the need to apply time series analysis and forecasting
methods to other key performance indicators that the Foundation
publishes on a daily or monthly basis.
We continued to work with the analytics engineers to provide requirements for Wikimetrics (with a particular focus on UserMetrics feature parity), and to perform data QA and validate the output of the application for metrics that were recently implemented.
We completed a round of consultations with internal stakeholders to
identify research needs of each team in the organization and determine
their priority. We presented
a review of our activities for Q1 and plans for Q2 at the Analytics
Quarterly meeting. We identified "metric standardization" as one of the
goals the team will focus on in this quarter.
We
organized and
announced
the inaugural Wiki Research Hackathon, a global event hosted in 8
locations in 5 countries, bringing together Wikimedia researchers,
academics and community members to work on wiki research projects. The
hackathon — the first event organized in the context of the
Labs2 initiative — will take place on November 9, 2013.
The Kiwix project is funded and executed by Wikimedia CH.
- We have released a new version (1.4) of Kiwix for Android;
besides other improvements, we have introduced a localized user
interface. We continue our efforts to speed-up the ZIM throughput;
Wikipedia is now available offline in more than 60 languages. The last flash drives from our Deaddrops & Geocaching
pilot were sent; we hope to get them walled up before the end of the
year. We have also submitted a few proposals about Kiwix for Android for
the Google Code-In program.
The Wikidata project is funded and executed by Wikimedia Deutschland.
- At the beginning of the month, Lydia Pintscher took over the product management for Wikidata. In a blog entry,
she elaborated her goals for the future. The developers continued their
work on the numbers datatype, sorting of statements and simple queries.
In addition, a prototype for a multilingual picture dictionary was published, as well as a tool to visualize time and location data from Wikidata and another one to build lists based on data in Wikidata.
- October was also the month to celebrate Wikidata's first birthday. Enjoy some essays, interviews and more from the community.
Future
- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.
--
Guillaume Paumier
Technical Communications Manager — Wikimedia Foundation
https://donate.wikimedia.org