The report covering Wikimedia engineering activities (including upcoming
events, new hires, and open positions) in March 2013 is now available.
Thanks to Guillaume Paumier and Tilman Bayer and the engineers who
helped me put this together. I'm sorry for delaying and sending out this
announcement a few days after the blog posting.
Wiki
version:https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/Ma…
Blog
version:https://blog.wikimedia.org/2013/04/04/engineering-march-2013-report/
Below is the full text of the report, including links.
We're also providing a shorter, simpler and translatable version of this
report that does not assume specialized technical
knowledge:https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/…
As always, feedback is appreciated about the usefulness of the report
and its summary, and on how to improve them.
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation
Major news in March include:
- Open
registration<https://blog.wikimedia.org/2013/03/27/join-the-wikimedia-ha…
the Amsterdam hackathon, including travel sponsorship;
- Lua
scripting<https://blog.wikimedia.org/2013/03/11/lua-templates-faster-mor…
on all WMF wikis, with implications
for performance and structured data
use<https://blog.wikimedia.org/2013/03/14/what-lua-scripting-means-wikim…
;
- A redesign of the Translate
interface<https://blog.wikimedia.org/2013/03/25/redesigning-the-translat…
other
progress<https://blog.wikimedia.org/2013/03/12/new-release-of-the-mediaw…
translation and language-related tools;
- Fresh, friendly instructions on reporting a technical
problem<https://blog.wikimedia.org/2013/03/18/how-to-create-a-good-first…
an
invitation to help prioritize problems to
fix<https://blog.wikimedia.org/2013/03/19/help-wikimedia-squash-software…
;
- Greater ability to upload images to Commons from mobile
phones<https://blog.wikimedia.org/2013/03/11/putting-commons-contributio…s/>,
allowing you to directly add a
photo<https://blog.wikimedia.org/2013/03/28/add-an-image-to-this-article…
a Wikipedia article that has no image;
- Collaboration with the Noun
Project<https://blog.wikimedia.org/2013/03/13/the-noun-project-and-the-w…
creating an "Encyclopedia Collection" of free icons;
- A deeper look at
Parsoid<https://blog.wikimedia.org/2013/03/04/parsoid-how-wikipedia-catc…
the challenge of a better editing interface;
- Wikipedia Zero winning a SXSW Interactive award for
activism<https://blog.wikimedia.org/2013/03/13/wikipedia-zero-wins-2013-…
gaining a
new partner,
Axiata<https://blog.wikimedia.org/2013/03/19/axiata-wikipedia-zero-partn…
.
*Note: We're also providing a shorter, simpler and translatable version of
this
report<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/…
does not assume specialized technical knowledge.
*
Engineering metrics in March:
- Approximately 113 unique committers contributed patchsets of code to
MediaWiki.
- The total number of unresolved
commits<https://gerrit.wikimedia.org/r/#q,status:open+project:%255Emedia…
from about 830 to about 816.
- About 48 shell requests
<https://www.mediawiki.org/wiki/Shell_requests>were
processed<https://bugzilla.wikimedia.org/buglist.cgi?chfieldto=2013-03-3…
.
- Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs> now
hosts 154 projects and 1,103 users; to date 1,641 instances have been
created.
Upcoming events
There are many opportunities for you to *get involved and
contribute<https://www.mediawiki.org/wiki/How_to_contribute>
* to MediaWiki and technical activities to improve Wikimedia sites, both
for coders and contributors with other talents.
For a more complete and up-to-date list, check out the
Project:Calendar<https://www.mediawiki.org/wiki/Project:Calendar>
.
Date Type Event Contact Mar 31–Apr 7 Offline Hackathon
2013<http://www.kiwix.org/wiki/Offline_Hackathon_2013>(Paris(Paris, France)
Apr 1–Apr 7 QA: Fresh bugs
triage<https://www.mediawiki.org/wiki/Bug_management/Triage>- Skin
and page
rendering<https://www.mediawiki.org/wiki/Bug_management/Triage/20130402&…
a dedicated bugday on April 02nd
AKlapper <https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29>,
Valeriej<https://www.mediawiki.org/wiki/User:Valeriej> Apr
6 QA: Testing Americas focuses in the new Account creation user
experience<http://weekendtesting.com/archives/2883>
. Steven Walling <https://www.mediawiki.org/wiki/User:Steven_%28WMF%29>,
Cmcmahon <https://www.mediawiki.org/wiki/User:Cmcmahon%28WMF%29>,
Qgil<https://www.mediawiki.org/wiki/User:Qgil>
. Apr 6 Iconathon with The Noun
Project<https://www.mediawiki.org/wiki/Events/The_Noun_Project_Iconathon…
Francisco, CA, USA) VBamba
(
WMF)<https://www.mediawiki.org/w/index.php?title=User:VBamba_%28WMF%29&a…;redlink=1>,
MAssaf <https://www.mediawiki.org/wiki/User:MAssaf> Apr 10 Language
Engineering team <https://www.mediawiki.org/wiki/Language_portal> office
hour in #wikimedia-office
<irc://irc.freenode.net/wikimedia-office>connect<http://webchat.freenode.net/?channels=#wikimedia-office>
Runa
Bhattacharjee <https://www.mediawiki.org/wiki/User:Runab_WMF> Apr 11 Meetup:
GSoC and other open source internship
programs<http://www.meetup.com/Wikipedia-Engineering-Meetup/events/10909…
Francisco, CA, USA)
Qgil <https://www.mediawiki.org/wiki/User:Qgil> Apr 14 THATCamp
London
2013<https://uk.wikimedia.org/wiki/GLAM-WIKI_2013/Hackathon>and
Europeana hackathon (London, UK) Wikimedia
UK <https://uk.wikimedia.org/wiki/GLAM-WIKI_2013/Hackathon> Apr 15
QA: Old bugs triage
<https://www.mediawiki.org/wiki/Bug_management/Triage>- topic
to be
defined<https://www.mediawiki.org/wiki/Project_talk:WikiProject_Bug_Squa…
AKlapper <https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29>,
Valeriej<https://www.mediawiki.org/wiki/User:Valeriej> Apr
17
QA: Write browser testing
automation<https://www.mediawiki.org/wiki/QA/Browser_testing>scenarios
for the new
Account_creation_user_experience<https://www.mediawiki.org/wiki/Account_…
.
Zeljko.filipin<https://www.mediawiki.org/wiki/User:Zeljko.filipin%28WMF%…29>,
Qgil <https://www.mediawiki.org/wiki/User:Qgil>,
Cmcmahon<https://www.mediawiki.org/wiki/User:Cmcmahon%28WMF%29> Apr
20 Last day to register for the May Amsterdam hackathon and request travel
subsidy <https://www.mediawiki.org/wiki/Amsterdam_Hackathon_2013>
Multichill<https://www.mediawiki.org/wiki/User:Multichill> Apr
30
Last day to submit a talk for Wikimania in
August<https://wikimania2013.wikimedia.org/wiki/Submissions> Wikimania
organizing team <https://wikimania2013.wikimedia.org/wiki/Contact>
Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us>
Are you looking to work for Wikimedia? We are hiring for many positions,
and we really love talking to active community members about these roles.
In Engineering:
- Director of
Analytics<http://hire.jobvite.com/Jobvite/Job.aspx?j=oJriXfw9>
- Software Engineer - Editor
Engagement<http://hire.jobvite.com/Jobvite/Job.aspx?j=ovvXWfwD>
- Software Engineer -
Parser<http://hire.jobvite.com/Jobvite/Job.aspx?j=oIsbXfw2>
- Software Engineer - Language
Engineering<http://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH>
- Software Engineer -
Mobile<http://hire.jobvite.com/Jobvite/Job.aspx?j=o4cKWfwG>
- Software Engineer - Multimedia
Systems<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3>
- Software Engineer - Multimedia User
Interfaces<http://hire.jobvite.com/Jobvite/Job.aspx?j=ohqbXfwz>
- Software Engineer -
Search<http://hire.jobvite.com/Jobvite/Job.aspx?j=ogk1Wfwh>
- Product Manager -
Mobile<http://hire.jobvite.com/Jobvite/Job.aspx?j=oGWJWfw1>
- Director of User
Experience<http://hire.jobvite.com/Jobvite/Job.aspx?j=otv0WfwE>
- Visual Designer <http://hire.jobvite.com/Jobvite/Job.aspx?j=oomJWfw9>
- Dev-Ops Engineer
(
SRE)<http://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf>
- Operations Engineer - Database
Administrator<http://hire.jobvite.com/Jobvite/Job.aspx?j=obMOWfwr>
We have seven additional
openings<https://wikimediafoundation.org/wiki/Work_with_us>in other
Foundation departments.
Announcements
Two new full-time employees started in WMF engineering in March:
- Yuri Astrakhan, Senior Software Engineer in the Mobile group (
announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-March/…
).
- Adam Baso, Senior Software Engineer, Mobile (Engineering)
(
announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-March/…
).
Technical Operations
*Site infrastructure*
This month we saw a few short site glitches that lasted from about a minute
to ten minutes each. The outages did not noticeably affect readers, but
editors and contributors experienced intermittent problems.
- The first incident was triggered by a deployment of Article Feedback
Tool v5 <https://www.mediawiki.org/wiki/Extension:ArticleFeedbackv5>,
and once the code was reverted, the site outage ended. The incident lasted
for about 10 minutes (incident
documentation<https://wikitech.wikimedia.org/wiki/Site_issue_Mar_15_2013…
).
- The other two were jobqueue-related, according to Asher Feldman. The
current MySQL jobqueue implementation is far too costly. In analyzing the
data during that 24-hour period, we see that 75% of all queries that take
over 450ms to run on the English Wikipedia master are related to the
jobqueue, and all major actions result in replicated writes. In fact, the
jobqueue takes 58% of all query execution time when not limiting the
analysis to queries over the slow threshold. If 1 million
*refresh-links*jobs are queued as quickly as possible without paying
attention to
replication lag, that causes the Apache servers to experience time-out due
to the replication lag. MediaWiki depends on reading from slaves to scale,
and avoids lagged ones. If all slaves are lagged, the master is used for
everything, and if this happens to English Wikipedia, the site falls over.
This MySQL jobqueue was identified as a scaling bottleneck a while ago, and
thus we will be switching to Redis very soon. We're currently aiming for
that switch to coincide with the release of 1.22wmf1, but we may be able to
backport to 1.21wmf12 and get this done in early April.
- On March 12, we experienced a ESAM site outage which was probably
caused by packet loss between ESAM and EQIAD. Leslie changed routes from
Esams to Eqiad to fix the packet loss, which caused Esams to recover. While
we still don't clearly understand what caused the outage, we did notice it
coincided with the news release when the new Pope was elected. The election
did trigger a surge in traffic to our web properties.
In March, we had a short security sprint led by Leslie Carr. We patched
servers that needed security upgrades. In addition, we continued to work on
MariaDB migration, Ceph deployment and fixing Varnish bugs.TechOps has
initiated a biweekly meeting with the engineering teams to drive alignment
amongst the various engineering projects and TechOps regarding requirements
and expectations. This is also the process to surface potential deployment
issues (such as capacity demand, new infrastructure and performance).
Meeting minutes are documented on the meeting
Etherpad<http://etherpad.wmflabs.org/pad/p/EMGT-Ops-Projects-25March2013…
.
*Fundraising*
Added logging to fundraising deployment scripts.
*Data Dumps <https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps>*
Work is continuing on tools for import. Setting up a local copy of a wiki
which includes only a subset of the page content has always been
problematic, since this requires use of the notoriously slow and finicky
importDumpphp maintenance script. Under development is a tool to filter the
currently produced SQL table dumps against a list of page IDs of a content
subset; these tables could then be imported into a MySQL database, along
with tables produced from the content subset, bypassing the need for
importDump.php. Additionally, these SQL fles could be shared with other
users who are interested in the same content subset. We hope to be able to
launch this in April.
*Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>*
A schedule has been posted for
LabsDB<https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan>an>.
A number of glusterfs stability work has been done. We've also begun work
on a replacement for project storage. A new feature has been added in
support of tool labs: service users and groups. This feature is per-project
manageable service users and groups. A number of interface fixes were made,
such as adding the admin list to the project page and layout of the
instance pages. Network changes were made to the instances and network
hosts: the network node has 3 bonded 1GB NICs and all instances were
changed to use the virtio network driver, which increases their speed to
the speed of the host. Work on tool labs progressed well this month. Most
of the necessary infrastructure is available for many tools and bots. Features
Engineering <https://www.mediawiki.org/wiki/Wikimedia_Features_engineering>
Editor
retention: Editing tools
*VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>*
In March, the team worked on the major new features that will be added in
the coming months. The objective is for VisualEditor to be the default
editor for all users, capable of letting them edit the majority of content
without needing to use the wikitext editor, in July 2013. This will mean
adding support for references, (at least) basic templates, categories and
images, each of which is a very large piece of work. This month the primary
focus was editing of categories and templates, with draft designs created
and initial code developed. The team undertook its first ever "Quarterly
Review<https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/Q…s>",
whose
slides<https://www.mediawiki.org/wiki/File:VisualEditor-Parsoid_-_2012-1…
these designs, the work done to date and expectations for the near
future. The alpha version of VisualEditor on
mediawiki.org and the English
Wikipedia was updated twice
(
1.21-wmf11<https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf11#VisualEdi…
-wmf12 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf12#VisualEditor>),
adding better input and selection support, fixing a number of bugs, and
restructuring the back-end so that the new features will be simpler to
create.
*Parsoid <https://www.mediawiki.org/wiki/Parsoid>*
In March, the Parsoid team continued with improvements to
internationalization, serialization, and extension handling.
The parser test framework now supports language-specific tests, which
required support for loading language-specific default setting in Parsoid.
The serializer is now fully DOM-based and uses constraint-based newline /
white-space separator handling, which will make the serializer less
sensitive to newlines and whitespace in HTML. Round-trip test
results<http://parsoid.wmflabs.org:8001/>of 82% (pages without any
diffs) and 98% (pages without semantic diffs)
indicates that the new serializer is on par with the old serializer
currently deployed on production.
Extension content is now parsed all the way to DOM, which enforces proper
nesting. The generic support for balanced fragment parsing will later also
be applied to templates. Parsing of transclusion directives (includeonly
and friends) has also been improved and simplified.
The DOM
specification<https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec&…
images and templated / extension content was fleshed out in
preparation
for full editing support.
Late in March, C. Scott Ananian joined us as a contractor. Welcome!
Editor engagement features
*Notifications <https://www.mediawiki.org/wiki/Echo_%28Notifications%29>*
In March, the team continued to deploy new features for the Notifications
project (code-named *Echo*) on
mediawiki.org. Ryan Kaldari developed a
new Thanks
notification<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Th…
lets you express your gratitude to users who make constructive edits
by notifying them that they have been thanked (this feature was designed to
give positive feedback to new editors during their first steps on
Wikipedia). Benny Situ built the User rights
notification<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Us…ts>,
which is sent when your user rights are changed (this feature was requested
by power users on the English Wikipedia). Luke Welling developed new code
to send HTML email
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#H…ts>,
based on designs from Vibha Bamba. Fabrice Florin led discussions about
these new features to serve the needs of both new and current users, then
updated their feature
requirements<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Fi…ns>;
he also co-wrote this metrics
plan<https://www.mediawiki.org/wiki/Echo/Metrics>with Dario
Taraborelli, as well as a socialization plan and new
project
pages<https://en.wikipedia.org/wiki/Wikipedia:Editor_engagement/Echo>…
Oliver Keyes. We are now completing these final features and are
aiming for a first release on the English Wikipedia later this month; in
the meantime, you can help us test the current
version<https://www.mediawiki.org/wiki/Echo/Testing>on
mediawiki.org. To learn more, read this project
update<http://blog.wikimedia.org/2013/02/07/echo-a-new-notification-syst…
the Wikimedia blog.
*Article feedback <https://www.mediawiki.org/wiki/Article_feedback>*
This month, we released an updated version of Article Feedback
v5<https://www.mediawiki.org/wiki/Article_feedback/Version_5>(AFT5) on
the
French <https://en.wikipedia.org/wiki/fr:Sp%C3%A9cial:ArticleFeedbackv5>and
German
Wikipedia<https://en.wikipedia.org/wiki/de:Spezial:Artikelr%C3%BCckmeldu…v5>,
for evaluation by their communities. Developer Matthias Mullie completed
final features for this release, including a new feedback
link<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_R…es>,
auto-archive<https://www.mediawiki.org/wiki/Article_feedback/Version_5/F…
a tool that lets you discuss
useful feedback on article talk
pages<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_…ge>,
based on designs from Pau Giner and suggestions from Oliver Keyes. Product
manager Fabrice Florin worked with Denis Barthel and Sebastian Peisker on
the German
release<https://en.wikipedia.org/wiki/de:Wikipedia:Artikel-Feedback>a…
Benoît Evelin on the French
release<https://en.wikipedia.org/wiki/fr:Wikip%C3%A9dia:Prise_de_d%C3%A9…is>,
and we are very grateful to them and many other community members for their
invaluable contributions. A German community vote is expected in May on the
German Wikipedia, and in September on the French Wikipedia, when they will
decide whether or not to deploy the tool across their entire sites. Due to
data caching issues, the tool was temporarily turned off on the English
Wikipedia, where we expect to re-deploy it on an opt-in basis as soon as
practical, as described on this talk
page<https://en.wikipedia.org/wiki/Wikipedia_talk:Article_Feedback_Tool/…ck>.
After the English Wikipedia re-deployment in April, we plan to monitor
community feedback on all three pilots before making any more updates, but
other projects interested in the tool are invited to read this this 2013
release
plan<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Release_P…
contact us if they would like Article Feedback on their sites later
this year.
*Flow <https://www.mediawiki.org/wiki/Flow>*
Design work continued on Flow. We continued creating a "Portal" that will
engage discussion about Flow at three locations
(
mediawiki.org<https://www.mediawiki.org/wiki/FlowPortal>al>,
meta, and the English Wikipedia), and performing
research<https://www.mediawiki.org/wiki/Flow_Portal/Research>
.
Editor engagement experiments
*Editor engagement
experiments<https://www.mediawiki.org/wiki/Editor_engagement_experiments…
*
In March, the Editor Engagement Experiments team largely placed other
projects – such as guided
tours<https://www.mediawiki.org/wiki/Guided_tours>rs>,
EventLogging <https://www.mediawiki.org/wiki/EventLogging>, and others – on
hold to focus on two key initiatives: the "Getting
Started<https://www.mediawiki.org/wiki/Extension:GettingStarted>"
process for onboarding new
Wikipedians<https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians>ns>,
and on making the redesign of account creation and
login<https://www.mediawiki.org/wiki/Account_creation_user_experience>…
permanent, internationalized part of MediaWiki core.
For the Getting Started project, the team launched a new version on English
Wikipedia, which included a new landing page with additional types of tasks
suggested for brand new editors to try. The list of tasks is now generated
by a basic recommender
system<https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians/Recomme…
by Ori Livneh, which gathers, filters, and delivered a fresh list of
tasks automatically for every editor. This new backend paves the way for
releasing the "getting started" feature on other projects, after we've
completed data analysis and testing to understand which kinds of tasks are
ideal for first time editors. Additionally, Matt Flaschen collaborated with
the Editor Engagement Features team to build
notifications<https://www.mediawiki.org/wiki/Echo>to welcome new
editors and invite them to contribute via the Getting
Started.
For the account creation and login work, S Page, Munaf Assaf, and the rest
of the team rebuilt our design to work with MediaWiki core, and solicited
reviews from outside the team. We currently plan to launch both interface
redesigns on an opt-in basis in April, to have editors test the
localization and other functional aspects of the forms via a URL parameter,
before we enable them as default.
Support
*2012 Wikimedia
fundraiser<https://www.mediawiki.org/wiki/2012_Wikimedia_fundraiser>
*
In March, we wrapped up our 2012/13 non-English international fundraising
efforts, making approximately 5 million USD over the course of the month.
Originally, we had planned to run the non-English international fundraiser
continuously until June, but were forced to accelerate our plans due to
some potential instability at the beginning of April with one of our
crucial payment gateways. At the very end of March, we started publishing
aggregate public fundraising data to
samarium.wikimedia.org.
Language
engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineerin…
*Language tools <https://www.mediawiki.org/wiki/Language_tools>*
Highlights for this month's team progress include:
- The latest translation interface and translation editor
improvements<https://wikimediafoundation.org/wiki/2013/01/11/a-more-effi…
the Translate extension were completed and deployed. The new interface
was made default with MediaWiki 1.22wmf1, later enabled on Meta on
2013-04-03; the old translation editor is planned to be removed completely
in a few weeks. The user
documentation<https://www.mediawiki.org/wiki/Help:Extension:Translate>…
already been updated.
- The Search
improvements<https://www.mediawiki.org/wiki/Translation_UX/Development_p…
completed as well, with a pilot of the new
Special:SearchTranslations<https://translatewiki.net/wiki/Special:Search…
on a Solr backend) that will be extended to Wikimedia projects in a
few days.
- Publication of the language coverage
matrix<https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0Asnp…
.
- Development progress for the
translatewiki.net homepage
redesign<https://www.mediawiki.org/wiki/Translation_UX/Development_plan#…
.
*Milkshake <https://www.mediawiki.org/wiki/Milkshake>*
The language engineering team continued adding more input methods and web
fonts contributions to jQuery.ime and jQuery.webfonts (Milkshake
components). UX designer Pau Giner iterated with Howie Fung and Erik Möller
to incorporate UX changes to handle logged-in use cases for the Universal
Language Selector (jQuery.uls). ULS deployment is targeted for this fiscal
year (by the end of June 2013).
*Language community outreach*:
The Language Engineering team kickstarted its *Language Support Maven*
plan<https://www.mediawiki.org/wiki/Language_tools/Language_Team_Plan>…
getting language tools feedback from Wikimedian community members who
are using internationalisation and localisation tools developed by the
team. The team also held its regular monthly office
hours<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013…
March. The team's outreach coordinator also reported team progress
with
multiple blog
posts<https://blog.wikimedia.org/c/technology/features/internationalizat…
the technology blog. The team plans to restore its bug triage
sessions,
starting in April 2013.
Mobile<https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>
*Commons App <https://www.mediawiki.org/wiki/Apps/Commons>*
Initial version of Commons photo uploader app for Android is available for
download in Google Play. iOS version is still in beta, but should be
available in the store next month.
*Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>*
In March we added new telecom partners (such as Axiata Group
Berhad<https://blog.wikimedia.org/2013/03/19/axiata-wikipedia-zero-partn…)
and fixed some bugs, and brought new staff online. We also won an SXSW
Interactive "Activism"
award<https://blog.wikimedia.org/2013/03/13/wikipedia-zero-wins-2013-sxs…
Wikipedia Zero. In April we aim to start improving the code and the IP
detection.
*OpenStreetMap <https://www.mediawiki.org/wiki/OpenStreetMap>*
Max Semenik, Arthur Richards and Faidon Liambotis held an OSM
mini-hackathon<https://www.mediawiki.org/wiki/Events/Wikimedia_Mapping_E…
Open Source Days 2013 in Copenhagen. During the event, they agreed on
an
implementation strategy for the WMF mapping cluster.
*Mobile Web Photo
Upload<https://www.mediawiki.org/wiki/Mobile_design/Uploads>
*
In March, we added the ability to easily
upload<https://blog.wikimedia.org/2013/03/11/putting-commons-contributio…
lead
image<https://blog.wikimedia.org/2013/03/28/add-an-image-to-this-article…
articles that lack one in the stable version of the mobile site. We
also
deployed a workaround for an issue we discovered with heightened security
features in some newer browsers that make logging in to all the projects
via CentralAuth impossible in certain circumstances; that had prevented a
number of users from being able to upload photos via the mobile site. We
are now well on our way to reach our goal of 1000 unique uploaders/month by
the end of the fiscal year. Check out the mobile app
dashboard<http://mobile-reportcard.wmflabs.org/>to see mobile
contributions via the website and via apps. Also of note:
we've added thumbnails of lead images from articles in the mobile watchlist
view, as well as a "last modified" timestamp on articles in the stable
version of the mobile site. We are currently focusing on some performance
enhancements for the mobile site. In April we will graduate the "uploads
dashboard" feature from beta to stable, will further refine our photo
upload features, and will expose a feature to identify articles on subjects
near your current location to the beta version of the site.
Platform
Engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineerin…
MediaWiki
Core
*MediaWiki 1.21 <https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap>*
1.21wmf11 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf11> and
1.21wmf12 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf12> were
deployed to Wikimedia sites in March. We created the REL1_21 branch on
March 25, with the goal of a release candidate at the end of the month.
*Lua scripting <https://www.mediawiki.org/wiki/Lua_scripting>*
We launched Lua
scripting<https://blog.wikimedia.org/2013/03/11/lua-templates-faster-mor…
all wikis, wrote
about the launch's wider
significance<https://blog.wikimedia.org/2013/03/14/what-lua-scripting-me…e/>,
and held IRC office hours <https://meta.wikimedia.org/wiki/IRC_office_hours>.
In March we also added frame:callParserFunction() and frame:extensionTag(),
improved CPU time accounting, and allowed argument expansion to be
excluded. We have patches outstanding for "text" module including unstrip
functionality, as well as improved debug output. We've also made
significant improvements to templates since the launch.
*Auth systems <https://www.mediawiki.org/wiki/Auth_systems>*
This activity kicked off on March 26. We're planning a minimal OpenID
implementation and OAuth implementation in the coming months. Very
tentative target date is end of May.
*Search <https://www.mediawiki.org/wiki/Search>*
Search deployed to Beta Cluster. Search code instrumented for better
troubleshooting and identification of issues, and work is underway to add
PoolCounter support. Plan for April to make search updates more robust.
*Wikidata deployment <https://www.mediawiki.org/wiki/Wikidata_deployment>*
Phase II deployment will complete on March 27 to several wikis, including
Italian, Hebrew and Hungarian Wikipedias. This phase includes a new Lua
interface to Wikidata so as to make infobox population from Wikidata
possible. The Wikidata section of the monthly report has more details.
*Multimedia <https://www.mediawiki.org/wiki/Multimedia>*
Mostly bugfixing this month, as well as hiring for the two multimedia
positions. Jan Gerber finished work on an API to rotate images (bug
33186<https://bugzilla.wikimedia.org/show_bug.cgi?id=33186>)6>),
which needs a little site configuration work to get deployed. Transcoded
videos have been moved to their own container in the Swift filesystem in
anticipation of video-specific optimizations (bug
43343<https://bugzilla.wikimedia.org/show_bug.cgi?id=43343>)3>).
Jan also improved the user interface in cases where an embedded media
player is too small to display credits and player controls.
TimedMediaHandler<https://www.mediawiki.org/wiki/Extension:TimedMediaHan…
extended so that other MediaWiki extensions can render player elements
using PHP. The Score
extension<https://www.mediawiki.org/wiki/Extension:Score>can now use
TimedMediaHandler or OggHandler to render the audio player (see
bug 43388 <https://bugzilla.wikimedia.org/show_bug.cgi?id=43388>), which
puts the Score extension one step closer to deployment and music staves one
step closer to generation and display in wiki pages (see bug
189<https://bugzilla.wikimedia.org/show_bug.cgi?id=189>
).
*Site performance and
architecture<https://www.mediawiki.org/wiki/Site_performance_and_archite…
*
JobQueueRedis merged <https://gerrit.wikimedia.org/r/#/c/53315/>,
JobQueueAggregatorRedis <https://gerrit.wikimedia.org/r/#/c/48223/> merged
and deployed. These improvements to the jobqueue should help site
performance (see Tech Operations section of the monthly report).
*Admin tools
development<https://www.mediawiki.org/wiki/Admin_tools_development>
*
We've implemented improved support for blocking users coming in through
proxies with GlobalBlocking. Continuing work on identifying accounts that
have not been merged with Single User Login, with the goal of merging those
accounts starting in April.
*Security auditing and
response<https://www.mediawiki.org/wiki/Security_auditing_and_response&g…
*
The fundraising code base review is done. A MediaWiki security release,
1.20.3, was
published<http://lists.wikimedia.org/pipermail/mediawiki-announce/2013-M…
March 4. A review is underway for user
metrics API <https://www.mediawiki.org/wiki/User_Metrics>.
Quality assurance
*Quality Assurance <https://www.mediawiki.org/wiki/QA>*
Finalizing details for new-user features volunteer test
event<https://www.mediawiki.org/wiki/QA/WTA_ACUX_Test_Plan_Apr_6>with
Weekend Testers on April 6. Tested AFTv5 release to German Wikipedia,
French Wikipedia pending. Testing upcoming new features for
Echo<https://www.mediawiki.org/wiki/Echo>
.
*Beta cluster <https://www.mediawiki.org/wiki/Beta_cluster>*
"Phase 1" support on beta for Mobile is complete and Mobile is using the
beta cluster for testing now. We added search to beta, including Mobile.
Lucene instances have been set up to provide search suggestion and ...
search capabilities, but it's a rough base which still needs to be
improved. More automated tests are now targeting beta cluster, and
targeting the test2wiki/production cluster is underway. Jenkins is now
upgrading the database schemas on an hourly basis and deploying changes to
the MediaWiki configuration just after they have been merged. If you are
curious, have a look at the Jenkins dashboard for the beta
project<https://integration.wikimedia.org/ci/view/Beta/>
.
*Continuous
integration<https://www.mediawiki.org/wiki/Continuous_integration>
*
Timo Tijhof implemented a Jenkins job to run the QUnit javascript test for
MediaWiki core. That will definitely help us catch most of the javascript
issues.
The continuous integration site has been moved from
integration.mediawiki.org to
integration.wikimedia.org and is now always on
HTTPS. The index page has been rewritten based on Twitter Bootstrap (see
integration.wikimedia.org).
Antoine Musso has given our Zuul status
page<https://integration.wikimedia.org/zuul/>an overhaul. It features
live reloading through ajax and contains direct
links to the Gerrit changesets and Jenkins jobs. A big improvement
over the plain
text <https://integration.wikimedia.org/zuul/status> version.
Antoine Musso and Timo Tijhof set up the new
doc.wikimedia.org
portal<https://doc.wikimedia.org>.
The MediaWiki core (Doxygen-generated) PHP documentation has been moved
here (
svn.wikimedia.org/doc is now a redirect). We're currently working on
packaging jsduck and writing Jenkins jobs to generate JavaScript
documentation with JSDuck.
We've packaged various Python modules for the Debian project, which will in
turn let us simplify deployment. Meanwhile, we're experimenting with having
our Debian/Ubuntu packages built by Jenkins directly.
This month we've continued to extend Jenkins coverage for Gerrit
repositories. We're happy to announce that almost all repositories for
MediaWiki extensions in Gerrit now have Jenkins integration.
*Browser testing <https://www.mediawiki.org/wiki/QA/Browser_testing>*
We held an exercise to build the test backlog using Search as an
example<https://www.mediawiki.org/wiki/QA/Browser_testing/Search_feature…
did build the test backlog for search; it now contains wide variety
of regression tests for automation. But the exercise didn't attract as many
volunteers as we'd like; we're
looking<https://www.mediawiki.org/wiki/QA/Strategy>for ways to address
that in future exercises. Browser tests continue to
identify significant regression issues, with new features in particular. We
are extending browser test coverage to move-page functions, PDFs, and
Search with a volunteer from the March exercise.
Analytics
*Visualization, Reporting & Applications*
In order to support mobile initiatives--including the Mobile Website,
Mobile Apps, and Wikipedia Zero--we focused our attention on providing data
extracts and visualizations with this focus. New visualizations
include the Mobile
app dashboard <http://mobile-reportcard.wmflabs.org>. In addition, we
updated the report card <http://reportcard.wmflabs.org> for the March
Metrics Meeting, improved the robustness of the reportcard
infrastructure<https://mingle.corp.wikimedia.org/projects/analytics/card…26>,
added target
bars<https://mingle.corp.wikimedia.org/projects/analytics/cards/123>a…
added links
to the metric
definitions<https://mingle.corp.wikimedia.org/projects/analytics/cards/1…
.
*Wikistats*
We are currently working on a new mobile pageview report.
*Services & Access Points*
In March, we saw the launch of the User Metrics
API<https://www.mediawiki.org/wiki/User_Metrics>cs>,
a service <http://metrics.wikimedia.org> that allows researchers to
perform cohort
analysis <https://en.wikipedia.org/wiki/Cohort_study> on various data sets,
making it easier to measure the effects of programs and platform
experiments among discrete sets of users. We are currently working on
improving the web-based user interface to make it available for use outside
of Wikimedia Foundation staff in the coming months.
*Analytics Infrastructure*
Our big-data cluster known as
Kraken<https://www.mediawiki.org/wiki/Analytics/Kraken>has been
undergone no major changes in capability, but we have been working
to make it more
robust<https://mingle.corp.wikimedia.org/projects/analytics/cards/461>…
improve security. Our udp2log
monitoring <https://mingle.corp.wikimedia.org/projects/analytics/cards/155>has
become more accurate, and Limn
can be
installed<https://mingle.corp.wikimedia.org/projects/analytics/cards/52&…
both production and Labs instances.
*Misc*: *Defects Closed*
Fixed the Space characters in pagecounst-raw
titles<https://bugzilla.wikimedia.org/show_bug.cgi?id=45178>buggt;bug.
*Misc*: *Management & Communication*
The Analytics team has started to use
Mingle<https://mingle.corp.wikimedia.org/projects/analytics>to manage
its work more effectively day-to-day.
Bugzilla<https://bugzilla.wikimedia.org/buglist.cgi?list_id=190848&q…
our primary interface for managing defects with respect to
communicating their priority and status. Finally, we had our Analytics
Reboot<https://www.mediawiki.org/wiki/Analytics/Roadmap/PlanningMeetings…eeting,
where all internal WMF Analytics stakeholders convened and we
surveyed what customer opportunities were out there, what Analytics models
are currently available, and how to improve inter-team communication.
Engineering
community team
*Bug management <https://www.mediawiki.org/wiki/Bug_management>*
In Bugzilla, a way to mark bugfixes to copy from the development branch to
stable
branches<http://lists.wikimedia.org/pipermail/wikitech-l/2013-March/0675…
introduced to easier identify important bugfixes to include in tarball
releases.
Two bugdays took place as part of the QA Weekly
Goals<https://www.mediawiki.org/wiki/QA/Weekly_goals>ls>:
cleaning up and retesting General MediaWiki
reports<https://www.mediawiki.org/wiki/Bug_management/Triage/20130307>…
a
bugday concentrating on the LiquidThreads
extension<https://www.mediawiki.org/wiki/Bug_management/Triage/20130318&…18>.
For the latter, 76 out of 218 open reports received updates.
Valerie<https://www.mediawiki.org/wiki/User:Valeriej>analyzed which
important Wikimedia feedback channels link to each other and
Bugzilla, and created a diagram of the current
situation<https://commons.wikimedia.org/wiki/File:Steps_to_Bugzilla.svg&…vg>.
Valerie also published two blogposts explaining how to create a good first
bug
report<https://blog.wikimedia.org/2013/03/18/how-to-create-a-good-first-…
how
to help Wikimedia squash software
bugs<http://blog.wikimedia.org/2013/03/19/help-wikimedia-squash-software…s/>.
Andre <https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29>
improved the Bugzilla
Weekly
Report<http://lists.wikimedia.org/pipermail/wikitech-l/2013-March/067486…
to the wikitech mailing list. On most open bug reports with a target
milestone set to future MediaWiki version 1.21.0, reminder comments were
added for developers. Andre and Valerie also held the first IRC Office Hour
on Bugzilla and Bug
management<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours…
those interested in discussing problems and improvements with
Wikimedia's bug management. In Bugzilla's internal product and component
taxonomy, several Mobile application products were merged into a single
"Wikipedia App"
product<https://bugzilla.wikimedia.org/show_bug.cgi?id=41922>and two
Search components were
merged<https://bugzilla.wikimedia.org/show_bug.cgi?id=46542>42>,
to simplify finding information for developers and reporters.
Also, the bug management task
list<https://www.mediawiki.org/wiki/Bug_management/Task_list>received
a major cleanup, making it clearer what is being worked on and
what you can help with.
*Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>*
Quim Gil <https://www.mediawiki.org/wiki/User:Qgil> focused on:
- Completing Round 5 of Outreach Program for
Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women>with
casual evaluation meetings with the 6 intern/mentors teams. To be
summarized with a blog post in early April.
- Wikimedia's application for Google Summer of Code
2013<https://www.mediawiki.org/wiki/Summer_of_Code_2013>and a new
round of Outreach
Program for
Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women>inclu…
17 common project ideas with mentors.
- Making Possible
projects<https://www.mediawiki.org/wiki/Mentorship_programs/Possible_pro…
reference list of big tasks to potential contributors. Featured
project
ideas<https://www.mediawiki.org/wiki/Mentorship_programs/Possible_projec…
go through a reality check considering project feasibility for
newcomers, availability of mentors and community/maintainers buy-in, which
we check through related feature requests filed in
Bugzilla<https://www.mediawiki.org/wiki/Bugzilla>
.
- Announcing a GSoC and other open source internship
programs<http://www.meetup.com/Wikipedia-Engineering-Meetup/events/10909…
in San Francisco on April 11.
*Technical
communications<https://www.mediawiki.org/wiki/Technical_communications&g…
*
Guillaume Paumier <https://www.mediawiki.org/wiki/User:Guillom> reviewed,
published and advertised several tech blog posts, and did communications
support for the deployment of
Extension:Scribunto<https://www.mediawiki.org/wiki/Extension:Scribunto&g…to>;
this included reviewing blog posts, organizing IRC office hours, and
announcing the deployment on wikis and
lists<http://lists.wikimedia.org/pipermail/wikimediaannounce-l/2013-Marc…ml>.
He modified a template <https://www.mediawiki.org/wiki/Template:Event> so
that events from the MediaWiki
calendar<https://www.mediawiki.org/wiki/Project:Calendar>show as
bulleted
items<https://www.mediawiki.org/wiki/Template_talk:New_opportunities/Con…
the home
page <https://www.mediawiki.org/wiki/MediaWiki>. He continued to
investigate SugarCRM, and expanded *Product
development<https://www.mediawiki.org/wiki/Product_development>
* to facilitate the involvement of contributors, and redesigned,
reorganized and simplified the *How to
contribute<https://www.mediawiki.org/wiki/How_to_contribute>
* landing page. He added "Open tasks" sections to pages of activities in
need of Product help, and linked to them from the Product hub. He also met
with Quim Gil to discuss his contributors
plan<https://www.mediawiki.org/wiki/User:Qgil/Contributors>rs>,
and followed up on the gerrit tagging
proposal<http://thread.gmane.org/gmane.science.linguistics.wikipedia.tec…83>.
He researched ways to create and maintain translations of the Wikimedia
Glossary <https://meta.wikimedia.org/wiki/Glossary>, and started a discussion
on the lists <http://thread.gmane.org/gmane.org.wikimedia.wikidata/1856>about
the best way to move forward with glossaries scattered across wikis.
Last, he started to plan for the centralization of mobile
documentation<https://www.mediawiki.org/wiki/Technical_communications/Mo…on>,
and drafted thoughts about a consolidation of technical
communities<https://www.mediawiki.org/wiki/Technical_communications/Dev_…
.
*Volunteer coordination and
outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outre…
*
Quim Gil <https://www.mediawiki.org/wiki/User:Qgil> focused on:
- Drafting Wikitech
contributors<https://www.mediawiki.org/wiki/Requests_for_comment/Wikitec…rs>,
a proposal to attract technical volunteers and connect them with
interesting people and activities in a single site:
wikitech.wikimedia.org <https://wikitech.wikimedia.org/wiki/Main_Page>.
Helping define the proposal for distribution of content between Wikitech
and
mediawiki.org<https://www.mediawiki.org/wiki/Technical_communications/De…
.
- (Re)defining factors for measuring
success<https://www.mediawiki.org/wiki/QA/Strategy#Measuring_success>…
QA activities.
- QA weekly goals <https://www.mediawiki.org/wiki/QA/Weekly_goals>:
supporting and promoting our first browser automation activity: Browser
automation testing for Wikipedia
Search<https://www.mediawiki.org/wiki/QA/Browser_testing/Search_features…es>.
Also the general MediaWiki reports Bug
Triage<https://www.mediawiki.org/wiki/Bug_management/Triage/20130307>…
the
LiquidThreads Bug
Triage<https://www.mediawiki.org/wiki/Bug_management/Triage/20130318>
.
- San Francisco
meetups<http://www.meetup.com/Wikipedia-Engineering-Meetup/>p/>:
organized Lua meets
Wikipedia<http://www.meetup.com/Wikipedia-Engineering-Meetup/events/1060…meetup.
- Helping the Security for developers
training<https://www.mediawiki.org/wiki/Security_for_developers/Training…
and creating a wiki page to be recycled for future editions.
- Drafting criteria to become an official Wikimedia mobile
app<https://meta.wikimedia.org/wiki/Wikimedia_Apps/proposal#Main_criteri…
sync with Mobile Programs & Engineering teams.
Kiwix <http://www.kiwix.org>
*The Kiwix project is funded and executed by Wikimedia
CH<https://meta.wikimedia.org/wiki/Wikimedia_CH>
.*
Work on the 0.rc3 release of Kiwix is ongoing, mostly consisting of bug
fixing and a few UI
improvements<https://sourceforge.net/p/kiwix/kiwix/ci/master/tree/CHANGE…OG>.
The release date is in around one month. For the first time, a ZIM file of
Wikisource (in
French)<http://download.kiwix.org/zim/wikisource_fr_all.zim.torrent>w…
done, within the scope of the
Afripedia<http://blog.wikimedia.org/2013/01/24/afripedia-project-increas…roject.
Wikidata <https://meta.wikimedia.org/wiki/Wikidata>
*The Wikidata project is funded and executed by Wikimedia
Deutschland<https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en>
.*
Denny Vrandečić and Lydia Pintscher gave a short update on Wikidata's
status at the metrics and activities
meeting<https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/…07>.
A more detailed analysis can be found in our blog
post<http://blog.wikimedia.de/2013/03/19/some-data-on-wikidata/>.
In addition, Wikidata phase 1 (language links) has been activated on the
remaining 282
Wikipedias<http://blog.wikimedia.de/2013/03/06/wikidata-now-live-on-all-wikipedias/>.
This means that all Wikipedias now get their language links from Wikidata.
Not too long after that, phase 2 (infoboxes) was activated on the first 11
Wikipedias <http://blog.wikimedia.de/2013/03/27/you-can-have-all-the-data/>.
They can now make use of shared structured data from Wikidata in their
articles. On Wikidata itself we introduced a new data type (string),
extended references in statements (they can now have multiple values), and
improved the search box. We have written down how we envision
queries<https://meta.wikimedia.org/wiki/Wikidata/Development/Queries>…
Wikidata and would appreciate your feedback. As
a nice demonstration of the potential of Wikidata we've seen two new
projects this month: Wiri
<http://toolserver.org/%7Emagnus/thetalkpage/>and a tree
of life <http://simia.net/treeoflife/>. Future The engineering management
team continues to update the
*Deployments<https://wikitech.wikimedia.org/wiki/Deployments>
* page weekly, providing up-to-date information on the upcoming deployments
to Wikimedia sites, as well as the *engineering
roadmap<https://www.mediawiki.org/wiki/Roadmap>
*, listing ongoing and future Wikimedia engineering efforts.