[Wikimedia-l] [Wikimedia Announcements] Wikimedia engineering May 2013 report

Guillaume Paumier gpaumier at wikimedia.org
Mon Jun 10 13:16:47 UTC 2013 

Hi,

The report covering Wikimedia engineering activities in May 2013 is now
available.

Wiki version:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May
Blog version:
https://blog.wikimedia.org/2013/06/10/wikimedia-engineering-may-2013-report/

We're also proposing a shorter, simpler and translatable version of this
report that does not assume specialized technical knowledge:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May/summary

Below is the full HTML text of the report.

As always, feedback is appreciated on the usefulness of the report and its
summary, and on how to improve them.

------------------------------------------------------------------

Major news in May include:

   - An invitation<https://blog.wikimedia.org/2013/05/01/apply-for-an-internship-with-the-language-engineering-team/>from
the Language engineering team to collaborate on language-related
   projects;
   - A new Notifications
system<https://blog.wikimedia.org/2013/05/02/notifications-launch-english-wikipedia/>enabled
on the English Wikipedia;
   - Recent developments<https://blog.wikimedia.org/2013/05/08/updates-from-language-engineering-changes-to-the-language-selector-new-extension-bundle-release/>in
language engineering, and the upcoming deployment of the Universal
   language selector<https://blog.wikimedia.org/2013/05/22/getting-ready-for-uls-everywhere-2/>on
all wikis;
   - The start of a discussion around
Flow<https://blog.wikimedia.org/2013/05/20/flow-next-generation-discussion-system/>,
   a proposed discussion system for Wikimedia sites;
   - A call for
proposals<https://blog.wikimedia.org/2013/05/21/request-for-proposals-mediawiki-release-management/>to
manage the MediaWiki release cycle;
   - An experience-sharing
exercise<https://blog.wikimedia.org/2013/05/28/developing-distributedly-part-1-tools-for-remote-collaboration/>by
the Mobile engineering team about distributed collaboration;
   - "Nearby <https://blog.wikimedia.org/2013/05/29/wikipedia-nearby-beta/>",
   a feature showing Wikipedia articles about nearby places on location-aware
   devices;
   - Tool Labs, which is now operational and ready to host
tools<https://blog.wikimedia.org/2013/05/30/preparing-for-the-migration-from-the-wikimedia-toolserver-to-tool-labs/>migrated
from the Toolserver;
   - A test wiki<https://blog.wikimedia.org/2013/05/30/test-features-in-a-right-to-left-language-environment/>to
try out new features in right-to-left languages
   - Tech news <https://meta.wikimedia.org/wiki/Tech/News>, a weekly tech
   newsletter to help users stay informed of technical changes going to impact
   them.

*Note: We're also providing a shorter, simpler and translatable version of
this report<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May/summary>that
does not assume specialized technical knowledge.
*
Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us>

Are you looking to work for Wikimedia? We have a lot of hiring coming up,
and we really love talking to active community members about these roles.

   - Software Engineer -
Parser<http://hire.jobvite.com/Jobvite/Job.aspx?j=oIsbXfw2>
   - Software Engineer -
Fundraising<http://hire.jobvite.com/Jobvite/Job.aspx?j=oawpXfwM>
   - Software Engineer - Language
Engineering<http://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH>
   - Software Engineer -
Mobile<http://hire.jobvite.com/Jobvite/Job.aspx?j=o4cKWfwG>
   - Software Engineer - Multimedia
Systems<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3>
   - Software Engineer - Multimedia User
Interfaces<http://hire.jobvite.com/Jobvite/Job.aspx?j=ohqbXfwz>
   - Senior Software Engineer -
Platform<http://hire.jobvite.com/Jobvite/Job.aspx?j=ouLnWfwi>
   - UX Designer <http://hire.jobvite.com/Jobvite/Job.aspx?j=onImXfw8>
   - Research Analyst <http://hire.jobvite.com/Jobvite/Job.aspx?j=oTqrXfwr>
   - Product Manager -
Platform<http://hire.jobvite.com/Jobvite/Job.aspx?j=o3vtXfwI>
   - Dev-Ops Engineer - SRE<http://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf>
   - MySQL Database
Administrator<http://hire.jobvite.com/Jobvite/Job.aspx?j=obMOWfwr>
   - Director of Technical
Operations<http://hire.jobvite.com/Jobvite/Job.aspx?j=orXoXfwt>

Announcements

   - Alexandros Kosiaris joined the Technical Operations team as Operations
   Engineer (announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069370.html>
   ).
   - May Galloway joined the Product Development team as Visual Designer (
   announcement<http://lists.wikimedia.org/pipermail/design/2013-May/000518.html>
   ).
   - Jared Zimmerman joined the Engineering Department as Director of User
   Experience (announcement<http://lists.wikimedia.org/pipermail/wikimediaannounce-l/2013-May/000647.html>
   ).
   - Nik Everett joined the Platform engineering team as Senior Software
   Engineer specializing in Search
(announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069668.html>
   ).
   - Aarti Dwivedi <https://www.mediawiki.org/wiki/User:Rtdwivedi>, Anubhav
   Agarwal <https://www.mediawiki.org/wiki/User:Anubhav_iitr>, Harsh
Kothari<https://www.mediawiki.org/wiki/User:Harsh4101991>,
   Himeshi De Silva <https://www.mediawiki.org/wiki/User:Himeshi>,
Jiabao Wu<https://www.mediawiki.org/wiki/User:Jiabao_wu>,
   Kiran Mathew Koshy<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993>,
   Liangent <https://www.mediawiki.org/wiki/User:Liangent>, Molly
White<https://www.mediawiki.org/wiki/User:GorillaWarfare>,
   Moriel Schottlender <https://www.mediawiki.org/wiki/User:Mooeypoo>, Nazmul
   Chowdhury <https://www.mediawiki.org/wiki/User:Rasel160>, Nilesh
   Chakraborty <https://www.mediawiki.org/wiki/User:Nilesh.c>, Or
Sagi<https://www.mediawiki.org/wiki/User:Orsagi>,
   Petr Onderka <https://www.mediawiki.org/wiki/User:Svick>, Pragun
Bhutani<https://www.mediawiki.org/wiki/User:Pragunbhutani>,
   Praveen Singh <https://www.mediawiki.org/wiki/User:Prageck>, Rahul
   Maliakkal <https://www.mediawiki.org/wiki/User:Rahul21>, Richa
Jain<https://www.mediawiki.org/wiki/User:Rjain>,
   Rohan Verma <https://www.mediawiki.org/wiki/User:Clancer>, Tongbo
Sui<https://www.mediawiki.org/wiki/User:Beanixster>and Yevheniy
   Vlasenko <https://www.mediawiki.org/wiki/User:Zhenya> joined the
   Wikimedia tech community as part of Summer of Code
2013<https://www.mediawiki.org/wiki/Summer_of_Code_2013>.
   Katie Cunningham <https://www.mediawiki.org/wiki/User:KatieIreneC>
and Rachel
   Thomas <https://www.mediawiki.org/wiki/User:Rachel99> joined as part of
   the Outreach Program for
Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_6>
   .

Technical Operations

*Site infrastructure*
The migration to MariaDB continued, with Wikimedia Commons now fully moved
over. Additional database infrastructure work was completed in support of
the Tool Labs, producing a row-based replication stream with all
PII<https://en.wikipedia.org/wiki/en:Personally_identifiable_information>removed
for the publicly accessible Tool Labs databases.We'll
be upgrading to Request Tracker version 4 and migrating the service to
Eqiad soon. Most of the ground work is laid for this; RT4 is puppetized and
we've done a dry run.Brandon Black made improvements to our Varnish cache
invalidation code. Together with Mark Bergsma and Faidon Liambotis, they
wrote a replacement for varnishhtcpd called
vhtcpd<https://gerrit.wikimedia.org/r/gitweb?p=operations%2Fsoftware%2Fvarnish%2Fvhtcpd.git;a=summary>,
and deployed it to the production Varnish machines. The new daemon is ~50x
more efficient at the same basic job. This buys us some performance on
Varnish machines in general, but more importantly it gets rid of
invalidation failures due to network buffer overruns when the old daemon
couldn't keep up. This should also rid us from the random software failures
in the old daemon that resulted in missing some or all cache purge requests
for extended periods of time. The initial deployment has just been a basic
swap of the two daemons. Further near-future improvements include turning
on the new daemon's HTCP regex filtering configuration for more efficiency
gains, and tying its HTCP packet statistics back into our normal monitoring
and analysis infrastructure, so that we can better see any further
multicast invalidation delivery issues that may arise.Daniel Zahn led the
OS patch release train; the team has started to upgrade about 50% of the
servers to-date and will finish the rest in June.Faidon also worked with
Aaron Schulz and made Ceph at Eqiad the primary media storage back-end,
leaving Swift at Tampa as a secondary fail-over back-end. This means that
MediaWiki application servers were writing to & reading from there for
about four & two weeks respectively. More recently, the cluster encountered
some problems and traffic was temporarily reverted back to Tampa. Faidon is
now upgrading Ceph software to the newer upstream version, Cuttlefish,
which addresses a number of problems that were encountered during this time
and were successfully reported to Ceph developers. Testing will resume
after that happens, both by re-enabling Ceph as a MediaWiki back-end and by
switching a portion of Varnish traffic to Ceph.Mark and Faidon started
working on migrating the Text Squid servers to use Varnish instead. In
Mark's preliminary tests on 1 Esams and 4 Eqiad servers, the results look
encouraging. He will start adding more traffic to them in the coming weeks.

*Data Dumps <https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps>*
The routine dumps of Wikidata ran into two roadblocks, one of them related
to the way that Recent changes patrol and autopatrolling are handled in
MediaWiki. While a local workaround is in place, there has been discussion
of revamping the patrolling mechanism including changes to the
database [1]<https://bugzilla.wikimedia.org/show_bug.cgi?id=17237>.
The other issue, affecting the full history content dumps, also has a
temporary workaround until we can decide what the meaning of the
rev_lenfield in the database for revisions really means.The
mwxml2sql utils have been through some testing and bug fixes, and a
volunteer is interested in packaging them for Debian to accompany his code
for local installation of a mirror of Wikipedia (or the project of one's
choice).Incremental dumps will be developed this summer by
User:Svick<https://www.mediawiki.org/wiki/User:Svick/Incremental_dumps>as
part of this year's GSoC program. We're happy to have him and can't
wait
for the finished code!

*Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>*
This month was mostly dedicated to bringing Tool Labs online, but a number
of other changes occurred as well. Work has progressed on AJAXification of
the OpenStackManager interface. Instance reboot actions are now using this
and there are gerrit changes awaiting review for instance console output,
instance deletion, and some IP address actions. The custom virtual machine
image had a number of fixes this month improving reliability and boot
speed. We expect further improvements with upgrades to the OpenStack
grizzly or havana releases. OpenStack was upgraded from the essex release
to the folsom release, increasing speed of operations and bringing some new
features (such as instance resizing). We'll be making these features
available for use soon. All virtualization hosts and all instances were
upgraded for a kernel security vulnerability and were rebooted this month,
causing roughly an hour of scheduled downtime. Work has also been
progressing on creating instances pre-configured for doing MediaWiki
development; this has been working in the past, but it is now more
reliable, faster, and meets our legal team's requirements for MediaWiki
installations in Labs. Work has also progressed on a more reliable
development environment for Labs itself. Soon it should be possible to
install a pre-configured instance ready for making changes to the Labs
infrastructure.Tool Labs is now operational, with roughly 150 tools already
migrated. With the completion of the basics of replication (all but s7 are
operational), there remains no roadblocks for migration. During the week
since the Amsterdam Hackaton, a fair number of minor issues have been found
and fixed, and the general consensus from users is that the environment is
functional. On the roadmap for the next month is cleaning up some of the
management for replication (so that it is more generalized), finish s7, and
help users with their migration issues. Tool Labs work has also added a new
feature available to all of Labs: service groups. Service groups are a
user/group combination available locally within a project. Service group
membership allows regular users to sudo to the service groups, allowing
per-project service users, rather than needing to create local users via
puppet, or create global users via wikitech. Work progresses on puppetizing
more OpenStack services and the OpenStackManager extension. Currently OSM
development is hampered by a lack of test installs; soon we should have the
ability to easily create new labs instances running Openstack and
OpenStackManager for testing and development. Features
Engineering<https://www.mediawiki.org/wiki/Wikimedia_Features_engineering>
Editor
retention: Editing tools

*VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>*
In May, the VisualEditor team worked to complete the major new features we
have prioritised over the past few months. Our objective is for
VisualEditor to be the default editor for all Wikipedia users, capable of
letting them edit the majority of content without needing to use the
wikitext editor, in July 2013. We have focussed on four areas of new
functionality: adding and editing inclusions of references, templates,
categories and media items. Our main area of work over the past month has
been on references and templates, and we now have implemented editing them
in our experimental code; category editing is nearly complete and should be
made available very soon. The deployed alpha version of VisualEditor was
updated twice (1.22-wmf4<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf4#VisualEditor>and
1.22-wmf5 <https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf5#VisualEditor>),
adding a number of user interface improvements, including further work on
the back-end to better support the new features, and fixing a number of
bugs.

*Parsoid <https://www.mediawiki.org/wiki/Parsoid>*
In May, the Parsoid <https://www.mediawiki.org/wiki/Parsoid> team
implemented several new features, as well as important performance
optimizations in preparation for the July VisualEditor release.

A major image handling overhaul enabled rendering and editing of all
image-related parameters with a relatively simple DOM
structure<https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec#Images>.
Template and extension editing was improved to support editing of templates
within extensions. This lets editors modify and add templated citations in
VisualEditor, an important feature to improve the quality of articles in
Wikipedia.

On the performance front, we are now reusing expensive template, extension
and image expansions from our own previous output to avoid most API queries
after an edit. This is necessary to avoid overloading the API when tracking
all edits on Wikimedia projects. A cache infrastructure with appropriate
purging was set up and will be tested at full load through June.
At the Amsterdam hackathon, we helped developers leverage our rich
HTML+RDFa DOM output<https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec>for
projects like a Wikipedia-to-SMS service or the Kiwix
offline Wikipedia reader <http://www.kiwix.org/wiki/Main_Page>.
Editor engagement features

*Notifications <https://www.mediawiki.org/wiki/Echo_%28Notifications%29>*
In May, we released new features and bug fixes for Notifications on the
English Wikipedia and mediawiki.org. Ryan Kaldari, Vibha Bamba and Fabrice
Florin collaborated with community members to develop a 'new message
indicator'<https://en.wikipedia.org/wiki/Wikipedia:Notifications/New_message_indicator>,
to inform users when someone posts on their talk page. The team also
released a new 'Thanks
notification<https://en.wikipedia.org/wiki/Wikipedia:Notifications/Thanks>'
that lets editors show their appreciation to users who make helpful edits,
and offers a quick way to give positive feedback on Wikipedia. Benny Situ
developed a feature that marks messages as read when you visit your talk
page and expanded our first metrics
dashboards<http://toolserver.org/%7Edartar/en/echo/>,
in collaboration with Dario Taraborelli and Aaron Halfaker. New team member
Erik Bernhardson developed several new features, such as updating talk page
notifications to link directly to their sections. Luke Welling continued to
develop HTML Email
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#HTML_single_email_notifications>.
Fabrice Florin worked with Oliver Keyes to discuss this product with a wide
range of community members, and led the team to plan the next
steps<https://docs.google.com/a/wikimedia.org/presentation/d/1LyqKRVH1mWs3OMWOCIL5DdC5DO84sti5udvjeJAFpNw/edit#slide=id.gb26ea508_2_6>for
Notifications and other editor engagement features. To learn more,
visit the project
portal<https://en.wikipedia.org/wiki/Wikipedia:Notifications>,
read the FAQ page<https://en.wikipedia.org/wiki/Wikipedia:Notifications/FAQ>and
join the discussion on the talk
page <https://en.wikipedia.org/wiki/Wikipedia_talk:Notifications>.

*Article feedback <https://www.mediawiki.org/wiki/Article_feedback>*
In May, we developed a few last features and bug fixes on the
English<https://en.wikipedia.org/wiki/Special:ArticleFeedbackv5>,
French <https://en.wikipedia.org/wiki/fr:Sp%C3%A9cial:ArticleFeedbackv5>and
German<https://en.wikipedia.org/wiki/de:Spezial:Artikelr%C3%BCckmeldungen_v5>Wikipedias.
As requested by community members, Matthias Mullie developed a
new opt-in feature that makes it easier to enable or disable feedback on a
page<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Enable.2Fdisable_feedback_on_a_page>,
as well as new UI improvements to simplify the feedback page, based on
designs by Pau Giner and specifications by Fabrice Florin. The team also
released new feedback
links<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Feedback_link_on_article_pages>and
tested the
auto-archive<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Auto-archive_comments>feature
on prototype. For tips on how to use Article feedback, visit the
testing
page <https://www.mediawiki.org/wiki/Article_feedback/Version_5/Testing>,
and let us know what you think on this talk
page<https://en.wikipedia.org/wiki/Wikipedia_talk:Article_Feedback_Tool/Version_5>.
Feature development has now ended for this project, and we will determine
our next steps based on the upcoming community vote on the German Wikipedia
in coming weeks.

*Flow <https://www.mediawiki.org/wiki/Flow_Portal/Project_information>*
Discussion portals were
announced<https://blog.wikimedia.org/2013/05/20/flow-next-generation-discussion-system/>and
opened on three wikis: the English
Wikipedia <https://en.wikipedia.org/wiki/en:Wikipedia:Flow>,
Meta-Wiki<https://meta.wikimedia.org/wiki/Flow>,
and mediawiki.org <https://www.mediawiki.org/wiki/Flow_Portal>. An interactive
prototype <http://unicorn.wmflabs.org/flow/> was released to the public for
discussion. Discussion is on-going, and the definition of the "minimum
viable product" is being worked on.
Editor engagement experiments

*Editor engagement
experiments<https://www.mediawiki.org/wiki/Editor_engagement_experiments>
*
In May, the Editor Engagement Experiments team (E3) launched its redesigns
of account creation and
login<https://www.mediawiki.org/wiki/Account_creation_user_experience>forms,
after numerous bug fixes and working with local communities to
customize the interface as needed. The initial rollout was to 30 of the
largest Wikimedia projects in English and other languages, including
Wikipedia, Wikimedia Commons, Wikidata, Wikivoyage, Wikispecies, Wikiquote,
Wiktionary, Wikibooks, Wikiversity and Wikisource. Complete deployment as
the default for all remaining projects was enabled early in June.

Also this month, the team launched and tested a major revamp of the "Getting
Started" <https://www.mediawiki.org/wiki/Extension:GettingStarted>interface
for onboarding
new Wikipedians
<https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians>(in
English). This included a redesigned landing page, a refactor of the
backend to increase speed and stability, a new navigation toolbar on
articles that new users were given as their first editing task, and a guided
tour <https://www.mediawiki.org/wiki/Guided_tour> to help them complete
their first edit. The results of A/B
testing<https://meta.wikimedia.org/wiki/R:OB5>of this new version
showed the largest increase in click-through rates for
the landing page – up to 32%, a large increase over the 10-12%
click-through rate of previous versions. Overall, it also showed a small
but statistically significant increase in the rate of first time edits
(+1.7%) by new English Wikipedians invited to participate in Getting
Started.
Last but not least, the PostEdit
extension<https://www.mediawiki.org/wiki/Extension:PostEdit>(previously
enabled on most of the top Wikimedia projects by size) was
migrated to MediaWiki core. With this change, the post-edit confirmation
message <https://www.mediawiki.org/wiki/Confirmation_message> will be
available on all projects, and will be more easily integrated in to
VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>.
Support

*2012 Wikimedia
fundraiser<https://www.mediawiki.org/wiki/2012_Wikimedia_fundraiser>
*
The Operations team moved db1025 into the firewalled fundraising cluster
(frack), rebuilt it on Precise with MariaDB. RAID monitoring tools were
updated to support RAID controllers used in frack. We've mostly finished
building/puppetizing the new payments listener (thulium) as well as a new
CiviCRM host (barium), both of which are in frack.
Language engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineering>

*Language tools <https://www.mediawiki.org/wiki/Language_tools>*
The last round of major bugs have been fixed in the Universal Language
Selector (ULS) before we start phased its deployment over the month of June
across hundreds of Wikimedia websites. Communications announcements have
been started for the first phase of deployment which includes removal of
Narayam and WebFonts from sites before ULS is rolled out with the same
integrated feature set. Test scenarios for ULS have been identified in
detail. Implementation of automated tests is in progress. A combination of
ULS integration and cross browser testing is in progress. Final performance
tweaks are in progress before we get ready to launch the redesigned home
page of Translatewiki.net in June. The monthly version of the Mediawiki
Language Extension Bundle (MLEB) was released on May 29.

*Milkshake <https://www.mediawiki.org/wiki/Milkshake>*
jQuery.webfonts and jQuery.IME continue to be in maintenance mode with new
input methods added to our repository this month. jQuery.ULS is being
actively updated to reflect design changes suggested by the Product team as
well as bug fixes.

*Language Engineering Communications and
Outreach<https://www.mediawiki.org/wiki/Language_engineering_communications_and_outreach>
*
The Language Mavens met and discussed areas they can contribute to for
Language tools. As part of GSoC 2013, all developers on the team are set to
mentor 4 Language Engineering projects supporting jQuery.IME, Language
Coverage Matrix, RTL support for Visual Editor and Translate mobile app.
Monthly office hours and bug triage were held. The team continued to report
and blog on its activities. Hackathon organization and participation at
Amsterdam and Tel Aviv were very successful.
Mobile <https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>

*Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>*
This month, the team launched Wikipedia Zero with Mobilink in Pakistan,
refactored legacy codebase, migrated configuration from monolithic wiki
articles to per-carrier JSON configuration blobs, generated utility
scripts, patched legacy hyperlink redirect and content rendering bugs, and
supported partner on-boarding.

*Mobile Web Photo Upload<https://www.mediawiki.org/wiki/Mobile_design/Uploads>
*
This month, we've pushed the 'Nearby' view to the stable version of the
mobile site. Now, with a location-aware device, you can easily identify
articles close to your current location. We've also made ongoing
improvements to photo uploads and the photo upload experience, including
improved messaging around image quality and copyright requirements for new
uploaders in the beta version of the mobile site. The upload features in
general have been a great success, as we've sustained over 1000 unique
uploaders <http://mobile-reportcard.wmflabs.org/graphs/unique-uploaders>over
the last two months. We are continuing to experiment in the beta
version of the site with improvements to article editing, an improved
reorganization of site navigation, Echo notifications, talk pages, and
simplifying discovery of article actions (eg editing and watching). We plan
to launch these beta features over the course of the next month.
Platform Engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering>
MediaWiki
Core

*MediaWiki 1.21 <https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap>*
MediaWiki 1.21.0 <https://www.mediawiki.org/wiki/MediaWiki_1.21> was
released, followed by a MediaWiki 1.21.1 maintenance release.

*MediaWiki 1.22 <https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap>*
MediaWiki 1.22wmf3 <https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf3>,
wmf4 <https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf4>, and
wmf5<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf5>were deployed
to Wikimedia sites during the month of May. This included one
breaking change, gerrit change
49364<https://gerrit.wikimedia.org/r/#q,49364,n,z>
.

*Multimedia <https://www.mediawiki.org/wiki/Multimedia>*
After deploying Score <https://www.mediawiki.org/wiki/Extension:Score> in
April, various performance improvements and fixes were merged in May. To
improve scaling of large images, the VipsScaler
extension<https://www.mediawiki.org/wiki/Extension:VipsScaler>was
prepared on the cluster and is ready for deployment in the next weeks.
TimedMediaHandler<https://www.mediawiki.org/wiki/Extension:TimedMediaHandler>had
various bug fixes and the libraries used to encode WebM videos were
updated to improve quality and address encoding issues.

*Wikidata deployment <https://www.mediawiki.org/wiki/Wikidata_deployment>*
Wikidata continued to roll out updates to the client (what lives on the
various project wikis) and repository (what powers wikidata.org) software;
see the wikidata newsletter for more information: May
3rd<https://meta.wikimedia.org/wiki/Wikidata/Status_updates/2013_05_03>and
May
24th <https://meta.wikimedia.org/wiki/Wikidata/Status_updates/2013_05_24>.
Additionally, there was Operations-related work to create helpful URL
redirects for entity matching along with fixes to the logging of
autopatrolled edits (which was causing massive database usage and issues
when doing archival dumps). These URL redirects and autopatrol fixes should
go out in June.

*Site performance and
architecture<https://www.mediawiki.org/wiki/Site_performance_and_architecture>
*
Many MediaWiki developers met in
Amsterdam<http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069569.html>to
discuss architectural
principles <https://www.mediawiki.org/wiki/Architecture_guidelines> and the
RFC process.

*Admin tools development<https://www.mediawiki.org/wiki/Admin_tools_development>
*
This month the team worked on Single User Login
finalisation<https://www.mediawiki.org/wiki/Admin_tools_development/SUL_Audit>,
which will mean that all user accounts will be global across all of
Wikimedia's public wikis, and so allowing for cross-wiki notifications and
better tools for editors. This will require all user accounts to be
uniquely named and not conflict with other accounts. Some more work was
done on the global account renaming
tool<https://www.mediawiki.org/wiki/Admin_tools_development/Global_Rename>.
The team attended the Wikimedia Hackathon 2013 in
Amsterdam<https://www.mediawiki.org/wiki/Amsterdam_Hackathon_2013>where
we discussed issues with admin tools and plans for the future with a
number of volunteer developers.

*Security auditing and
response<https://www.mediawiki.org/wiki/Security_auditing_and_response>
*
We released MediaWiki 1.20.6/1.19.7 and provided security training for
developers at the Amsterdam Hackathon.

*HipHop deployment <https://www.mediawiki.org/wiki/HipHop_deployment>*
Several engineers at the Wikimedia Foundation met with Facebook engineers
to discuss potential deployment of HHVM in 2013
(summary<http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069264.html>).
We formed the HipHop mailing
list<https://lists.wikimedia.org/mailman/listinfo/hiphop>to discuss
moving forward with this work.
Quality assurance

*Quality Assurance <https://www.mediawiki.org/wiki/Quality_Assurance>*
In May, QA worked with a number of parties both in and outside the
Wikimedia Foundation to test Echo, VisualEditor, Universal Language
Selector, and other projects. We began an official QA mailing
list<https://lists.wikimedia.org/mailman/listinfo/qa>
.

*Beta cluster <https://www.mediawiki.org/wiki/Beta_cluster>*
In May, Ariel Glenn helped out setting up missing bits of infrastructure to
the beta cluster. He added a Redis instance (that holds the job
information) and HTTPS support <https://en.wikipedia.beta.wmflabs.org>.
HTTPS will let us write scenarios related to logging in on the wiki or via
a mobile device; it will also let us test out OAuth.

udp2log archiving is now working reliably. Max Semenik has set up an
EventLogging infrastructure on beta, and Niklas Laxström enabled Universal
Language Selector. The Job processing was malfunctioning but was restored.

Since April 30, the MediaWiki instances are using NFS, which is much faster
than the previous GlusterFS share; pages serving time went from 560 ms to
260 ms.

Roan Kattouw has deployed Parsoid and VisualEditor on beta. Just like in
production, users can enable it in your their preferences under 'Editing'.

*Continuous integration<https://www.mediawiki.org/wiki/Continuous_integration>
*
In the beginning of May, Jenkins/Zuul faced overload for a few days; this
was resolved by upgrading Zuul and tweaking some time-expensive part of the
code. Zuul now lets us define which jobs it triggers by using a predefined
template which makes it easier to add new projects. Zuul is now faster to
report changes back to Gerrit, which was a complaint during rush hours.

The Wikibase client and repo components are now tested in Jenkins. All
puppet repositories are now verifying the puppet manifests and erb
templates for syntax validity. The Qunit tests being run for MediaWiki core
and VisualEditor seem to be in good shape.
PHP CodeSniffer has been upgraded as well as the standard for MediaWiki
code. We have yet to enforce it though, since the current code base does
not pass the standard.

*Browser testing<https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing>
*
In May, we added beta labs as a target for automated browser tests, which
allowed us to create tests for the Universal Language Selector at the
Hackathon in Amsterdam. We shored up a lot of Jenkins builds at the
hackathon as well. We created our first test for VisualEditor, and we are
looking forward to working with Rachel Thomas (as part of the Outreach
Program for Women) to create more.
Analytics

*Analytics infrastructure<https://www.mediawiki.org/wiki/Analytics/Infrastructure>
*
We continued our efforts of increasing our monitor coverage of the
different webrequest dataflows. On the udp2log side, we added monitoring
per DC/server role<http://ganglia.wikimedia.org/latest/?c=Miscellaneous%20pmtpa&h=emery.wikimedia.org&m=cpu_report&r=hour&s=descending&hc=4&mc=2>.
Every month, we work on improving the robustness and security of the
analytics-related servers that we run: we moved the multicast relay from
Oxygen to Gadolinium, we upgraded Oxygen to Ubuntu Precise, and we moved
all the Limn-based dashboards from the Kripke labs instance to the Limn0
labs instance. Continous integration for webstatscollector, wikistats and
udp-filters now works. The puppet module for
Hadoop<https://gerrit.wikimedia.org/r/#/c/61710/>has been merged in
the Operations reposotiry; this is a big step forward in
moving Kraken from beta to production status. Magnus Edenhill demonstrated
varnishkafka based on Kafka 0.8; on a local machine varniskafka was able to
process 140k msgs/s and we are planning to do production testing mid June.
Last, we separated the Kraken machines from the other production servers by
installing network ACLs.

*Analytics Visualization, Reporting &
Applications<https://www.mediawiki.org/wiki/Analytics/Visualization,_Reporting_%26_Applications>
*
For the mobile team, we started collecting pageview
counts<http://stats.wikimedia.org/kraken-public/webrequest/mobile/platform/>for
both official and non-official Wikipedia apps. We changed our Kafka
import configuration so that the raw webrequest folders are directly
queryable using Hive. The decision was made to re-platform the UMAPI
codebase; we have spent quite some time specifying user
stories<https://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=764&view=Features+User+Metrics+API>and
had productive discussions about the architecture during the Amsterdam
Hackathon. On the development side, the 'page count' metric was introduced.
We adapted Ori Livneh's Mediawiki Vagrant VM to also support UMAPI in
combination with test data. This will make it much easier to debug issues
and open development up to community members. We also fixed numerous
stability bugs.
Engineering community team

*Bug management <https://www.mediawiki.org/wiki/Bug_management>*
A Wikimedia Labs instance <http://boogs.wmflabs.org/> to test Bugzilla
software changes and a patch to puppetize
Bugzilla<https://gerrit.wikimedia.org/r/#/c/62404/>are now available
(thanks to Ori
Livneh <https://en.wikipedia.org/wiki/en:User:Ori.livneh>). This will make
updating Bugzilla to newer versions and reapplying Wikimedia's custom
patches much easier. A MediaWiki Installer
bugday<https://www.mediawiki.org/wiki/Bug_management/Triage/20130503>took
place in preparation of the MediaWiki 1.21 release, as well as an IRC
office hour<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013-05-13>.
Andre Klapper worked on a proposal for a Bugzilla admin
policy<https://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29/BugzillaAdminPolicy>.
In Bugzilla's configuration, the number of Bugzilla
administrators<https://meta.wikimedia.org/wiki/System_administrators/table>has
been decreased in order to improve coordination. As a side-effect of
his investigation, Andre
documented<http://blogs.gnome.org/aklapper/2013/05/28/understanding-bugzilla-groups-and-admin-rights/>the
meaning of Bugzilla admin rights.

*Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>*
We selected 20 Google Summer of
Code<https://www.mediawiki.org/wiki/Summer_of_Code_2013>and 2 Outreach
Program for Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_6>projects
that will be mentored by a total of 32 volunteers. This represents
more than double the amount of projects we had last year. We received 69
applications from 60 students for Google Summer of Code 2013, from which 9
were also applying to OPW, and 4 OPW-only individual applications. Google
allocated the 21 slots we requested, but we decided to give one back in
order to keep a standard on project feasibility.

*Technical communications<https://www.mediawiki.org/wiki/Technical_communications>
*
In May, Guillaume Paumier <https://www.mediawiki.org/wiki/User:Guillom>'s
major focus was on supporting Tech
ambassadors<https://meta.wikimedia.org/wiki/Tech/Ambassadors>and
setting up Tech
news <https://meta.wikimedia.org/wiki/Tech/News>, an initiative aiming to
collaboratively monitor recent software changes likely to impact
Wikimedians, and distribute a weekly summary, free of technical jargon, to
subscribers on their talk page. Two issues of this weekly summary were
published this month; starting with the second issue, the content is now
distributed in several languages if translations are available. Guillaume
also continued to review technical blog posts, and executed the move of the
Mobile documentation<https://www.mediawiki.org/wiki/Technical_communications/Mobile_documentation_consolidation>from
Meta-Wiki to
mediawiki.org.

*Volunteer coordination and
outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach>
*
Quim Gil <https://www.mediawiki.org/wiki/User:Qgil> has been preparing a
proposal to get automated community
metrics<https://www.mediawiki.org/wiki/Community_metrics>based on
vizGrimoire <http://vizgrimoire.bitergia.org/> and provided by their
maintainers, Bitergia <http://bitergia.com/>. It is currently being
discussed with Sumana
Harihareswara<https://www.mediawiki.org/wiki/User:Sharihareswara_%28WMF%29>and
Rob
Lanphier <https://www.mediawiki.org/wiki/User:RobLa-WMF> for budget
approval. Quim also worked on a user-friendly template for the landing page
of the wikitech-announce<https://lists.wikimedia.org/mailman/listinfo/wikitech-announce>mailing
list that can be used for other Wikimedia lists (source
code <https://www.mediawiki.org/wiki/User:Qgil/wikitech-announce>). He also
created a landing page for
organizations<https://www.mediawiki.org/wiki/Mentorship_programs/Organizations>willing
to collaborate with Wikimedia to co-organize technical activities.
Kiwix <http://www.kiwix.org>

*The Kiwix project is funded and executed by Wikimedia
CH<https://meta.wikimedia.org/wiki/Wikimedia_CH>
.*
With openZIM <http://www.openzim.org>, we have finally released the first
version of its standard implementation code: the
zimlib<http://www.openzim.org/wiki/Zimlib>.
Kiwix was introduced in Debian
testing<http://packages.debian.org/search?suite=all&arch=any&searchon=names&keywords=kiwix>.
A new release of Kiwix for
Android<https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile>,
with a few bug fixes and improvements, was released. Our first GSoC project
(ZIM incremental updates for
Kiwix<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993/ZIM_incremental_updates_for_Kiwix>)
was prepared and accepted; work has already started with
Kiran<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993>,
the Indian student responsible for this project.
Wikidata<https://meta.wikimedia.org/wiki/Wikidata>

*The Wikidata project is funded and executed by Wikimedia
Deutschland<https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en>
.*
The Wikidata team worked on 2 major topics in May: the ability to access
data from Wikidata in a Wikipedia article by its label and not just its ID,
and the ability to enter points in time into
Wikidata<https://www.wikidata.org/wiki/Wikidata:Project_chat#Time_datatype_available_and_short_overview_of_next_steps>,
which for example now makes it possible to enter the date of birth of a
person. Magnus Manske blogged about the tool
ecosystem<http://blog.wikimedia.de/2013/05/06/the-wikidata-tool-ecosystem/>that
is building around Wikidata. During the next 3 months, the team will
be working<http://lists.wikimedia.org/pipermail/wikidata-l/2013-May/002293.html>with
3 Google Summer of Code students, and 2 other students will be working
with other organizations on Wikidata-related projects. The codebase has
been reviewed<http://lists.wikimedia.org/pipermail/wikidata-tech/2013-May/000006.html>by
Qafoo.
Wikidata-tech <https://lists.wikimedia.org/mailman/listinfo/wikidata-tech>,
a new mailing list for discussions related to the development around
Wikidata, was created. Additionally, the team attended the hackathon in
Amsterdam to give tutorials and meet other developers. A prototype of a
multilingual map <http://4thmain.github.io/d3hacks/wiki-atlas.html> was
built there, among other things. Future The engineering management team
continues to update the
*Deployments<https://wikitech.wikimedia.org/wiki/Deployments>
* page weekly, providing up-to-date information on the upcoming deployments
to Wikimedia sites, as well as the *engineering
roadmap<https://www.mediawiki.org/wiki/Roadmap>
*, listing ongoing and future Wikimedia engineering efforts.
--
Guillaume Paumier
Technical Communications Manager — Wikimedia Foundation
https://donate.wikimedia.org
-------------- next part --------------
_______________________________________________
Please note: all replies sent to this mailing list will be immediately directed to Wikimedia-l, the public mailing list of the Wikimedia community. For more information about Wikimedia-l:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
_______________________________________________
WikimediaAnnounce-l mailing list
WikimediaAnnounce-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimediaannounce-l

More information about the Wikimedia-l mailing list