Hi,
The report covering Wikimedia engineering activities in May 2013 is now available.
Wiki version: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May Blog version: https://blog.wikimedia.org/2013/06/10/wikimedia-engineering-may-2013-report/
We're also proposing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May/summary
Below is the full HTML text of the report.
As always, feedback is appreciated on the usefulness of the report and its summary, and on how to improve them.
------------------------------------------------------------------
Major news in May include:
- An invitationhttps://blog.wikimedia.org/2013/05/01/apply-for-an-internship-with-the-language-engineering-team/from the Language engineering team to collaborate on language-related projects; - A new Notifications systemhttps://blog.wikimedia.org/2013/05/02/notifications-launch-english-wikipedia/enabled on the English Wikipedia; - Recent developmentshttps://blog.wikimedia.org/2013/05/08/updates-from-language-engineering-changes-to-the-language-selector-new-extension-bundle-release/in language engineering, and the upcoming deployment of the Universal language selectorhttps://blog.wikimedia.org/2013/05/22/getting-ready-for-uls-everywhere-2/on all wikis; - The start of a discussion around Flowhttps://blog.wikimedia.org/2013/05/20/flow-next-generation-discussion-system/, a proposed discussion system for Wikimedia sites; - A call for proposalshttps://blog.wikimedia.org/2013/05/21/request-for-proposals-mediawiki-release-management/to manage the MediaWiki release cycle; - An experience-sharing exercisehttps://blog.wikimedia.org/2013/05/28/developing-distributedly-part-1-tools-for-remote-collaboration/by the Mobile engineering team about distributed collaboration; - "Nearby https://blog.wikimedia.org/2013/05/29/wikipedia-nearby-beta/", a feature showing Wikipedia articles about nearby places on location-aware devices; - Tool Labs, which is now operational and ready to host toolshttps://blog.wikimedia.org/2013/05/30/preparing-for-the-migration-from-the-wikimedia-toolserver-to-tool-labs/migrated from the Toolserver; - A test wikihttps://blog.wikimedia.org/2013/05/30/test-features-in-a-right-to-left-language-environment/to try out new features in right-to-left languages - Tech news https://meta.wikimedia.org/wiki/Tech/News, a weekly tech newsletter to help users stay informed of technical changes going to impact them.
*Note: We're also providing a shorter, simpler and translatable version of this reporthttps://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/May/summarythat does not assume specialized technical knowledge. * Personnel Work with us https://wikimediafoundation.org/wiki/Work_with_us
Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- Software Engineer - Parserhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oIsbXfw2 - Software Engineer - Fundraisinghttp://hire.jobvite.com/Jobvite/Job.aspx?j=oawpXfwM - Software Engineer - Language Engineeringhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH - Software Engineer - Mobilehttp://hire.jobvite.com/Jobvite/Job.aspx?j=o4cKWfwG - Software Engineer - Multimedia Systemshttp://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3 - Software Engineer - Multimedia User Interfaceshttp://hire.jobvite.com/Jobvite/Job.aspx?j=ohqbXfwz - Senior Software Engineer - Platformhttp://hire.jobvite.com/Jobvite/Job.aspx?j=ouLnWfwi - UX Designer http://hire.jobvite.com/Jobvite/Job.aspx?j=onImXfw8 - Research Analyst http://hire.jobvite.com/Jobvite/Job.aspx?j=oTqrXfwr - Product Manager - Platformhttp://hire.jobvite.com/Jobvite/Job.aspx?j=o3vtXfwI - Dev-Ops Engineer - SREhttp://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf - MySQL Database Administratorhttp://hire.jobvite.com/Jobvite/Job.aspx?j=obMOWfwr - Director of Technical Operationshttp://hire.jobvite.com/Jobvite/Job.aspx?j=orXoXfwt
Announcements
- Alexandros Kosiaris joined the Technical Operations team as Operations Engineer (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069370.html ). - May Galloway joined the Product Development team as Visual Designer ( announcementhttp://lists.wikimedia.org/pipermail/design/2013-May/000518.html ). - Jared Zimmerman joined the Engineering Department as Director of User Experience (announcementhttp://lists.wikimedia.org/pipermail/wikimediaannounce-l/2013-May/000647.html ). - Nik Everett joined the Platform engineering team as Senior Software Engineer specializing in Search (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069668.html ). - Aarti Dwivedi https://www.mediawiki.org/wiki/User:Rtdwivedi, Anubhav Agarwal https://www.mediawiki.org/wiki/User:Anubhav_iitr, Harsh Kotharihttps://www.mediawiki.org/wiki/User:Harsh4101991, Himeshi De Silva https://www.mediawiki.org/wiki/User:Himeshi, Jiabao Wuhttps://www.mediawiki.org/wiki/User:Jiabao_wu, Kiran Mathew Koshyhttps://www.mediawiki.org/wiki/User:Kiran_mathew_1993, Liangent https://www.mediawiki.org/wiki/User:Liangent, Molly Whitehttps://www.mediawiki.org/wiki/User:GorillaWarfare, Moriel Schottlender https://www.mediawiki.org/wiki/User:Mooeypoo, Nazmul Chowdhury https://www.mediawiki.org/wiki/User:Rasel160, Nilesh Chakraborty https://www.mediawiki.org/wiki/User:Nilesh.c, Or Sagihttps://www.mediawiki.org/wiki/User:Orsagi, Petr Onderka https://www.mediawiki.org/wiki/User:Svick, Pragun Bhutanihttps://www.mediawiki.org/wiki/User:Pragunbhutani, Praveen Singh https://www.mediawiki.org/wiki/User:Prageck, Rahul Maliakkal https://www.mediawiki.org/wiki/User:Rahul21, Richa Jainhttps://www.mediawiki.org/wiki/User:Rjain, Rohan Verma https://www.mediawiki.org/wiki/User:Clancer, Tongbo Suihttps://www.mediawiki.org/wiki/User:Beanixsterand Yevheniy Vlasenko https://www.mediawiki.org/wiki/User:Zhenya joined the Wikimedia tech community as part of Summer of Code 2013https://www.mediawiki.org/wiki/Summer_of_Code_2013. Katie Cunningham https://www.mediawiki.org/wiki/User:KatieIreneC and Rachel Thomas https://www.mediawiki.org/wiki/User:Rachel99 joined as part of the Outreach Program for Womenhttps://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_6 .
Technical Operations
*Site infrastructure* The migration to MariaDB continued, with Wikimedia Commons now fully moved over. Additional database infrastructure work was completed in support of the Tool Labs, producing a row-based replication stream with all PIIhttps://en.wikipedia.org/wiki/en:Personally_identifiable_informationremoved for the publicly accessible Tool Labs databases.We'll be upgrading to Request Tracker version 4 and migrating the service to Eqiad soon. Most of the ground work is laid for this; RT4 is puppetized and we've done a dry run.Brandon Black made improvements to our Varnish cache invalidation code. Together with Mark Bergsma and Faidon Liambotis, they wrote a replacement for varnishhtcpd called vhtcpdhttps://gerrit.wikimedia.org/r/gitweb?p=operations%2Fsoftware%2Fvarnish%2Fvhtcpd.git;a=summary, and deployed it to the production Varnish machines. The new daemon is ~50x more efficient at the same basic job. This buys us some performance on Varnish machines in general, but more importantly it gets rid of invalidation failures due to network buffer overruns when the old daemon couldn't keep up. This should also rid us from the random software failures in the old daemon that resulted in missing some or all cache purge requests for extended periods of time. The initial deployment has just been a basic swap of the two daemons. Further near-future improvements include turning on the new daemon's HTCP regex filtering configuration for more efficiency gains, and tying its HTCP packet statistics back into our normal monitoring and analysis infrastructure, so that we can better see any further multicast invalidation delivery issues that may arise.Daniel Zahn led the OS patch release train; the team has started to upgrade about 50% of the servers to-date and will finish the rest in June.Faidon also worked with Aaron Schulz and made Ceph at Eqiad the primary media storage back-end, leaving Swift at Tampa as a secondary fail-over back-end. This means that MediaWiki application servers were writing to & reading from there for about four & two weeks respectively. More recently, the cluster encountered some problems and traffic was temporarily reverted back to Tampa. Faidon is now upgrading Ceph software to the newer upstream version, Cuttlefish, which addresses a number of problems that were encountered during this time and were successfully reported to Ceph developers. Testing will resume after that happens, both by re-enabling Ceph as a MediaWiki back-end and by switching a portion of Varnish traffic to Ceph.Mark and Faidon started working on migrating the Text Squid servers to use Varnish instead. In Mark's preliminary tests on 1 Esams and 4 Eqiad servers, the results look encouraging. He will start adding more traffic to them in the coming weeks.
*Data Dumps https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps* The routine dumps of Wikidata ran into two roadblocks, one of them related to the way that Recent changes patrol and autopatrolling are handled in MediaWiki. While a local workaround is in place, there has been discussion of revamping the patrolling mechanism including changes to the database [1]https://bugzilla.wikimedia.org/show_bug.cgi?id=17237. The other issue, affecting the full history content dumps, also has a temporary workaround until we can decide what the meaning of the rev_lenfield in the database for revisions really means.The mwxml2sql utils have been through some testing and bug fixes, and a volunteer is interested in packaging them for Debian to accompany his code for local installation of a mirror of Wikipedia (or the project of one's choice).Incremental dumps will be developed this summer by User:Svickhttps://www.mediawiki.org/wiki/User:Svick/Incremental_dumpsas part of this year's GSoC program. We're happy to have him and can't wait for the finished code!
*Wikimedia Labs https://www.mediawiki.org/wiki/Wikimedia_Labs* This month was mostly dedicated to bringing Tool Labs online, but a number of other changes occurred as well. Work has progressed on AJAXification of the OpenStackManager interface. Instance reboot actions are now using this and there are gerrit changes awaiting review for instance console output, instance deletion, and some IP address actions. The custom virtual machine image had a number of fixes this month improving reliability and boot speed. We expect further improvements with upgrades to the OpenStack grizzly or havana releases. OpenStack was upgraded from the essex release to the folsom release, increasing speed of operations and bringing some new features (such as instance resizing). We'll be making these features available for use soon. All virtualization hosts and all instances were upgraded for a kernel security vulnerability and were rebooted this month, causing roughly an hour of scheduled downtime. Work has also been progressing on creating instances pre-configured for doing MediaWiki development; this has been working in the past, but it is now more reliable, faster, and meets our legal team's requirements for MediaWiki installations in Labs. Work has also progressed on a more reliable development environment for Labs itself. Soon it should be possible to install a pre-configured instance ready for making changes to the Labs infrastructure.Tool Labs is now operational, with roughly 150 tools already migrated. With the completion of the basics of replication (all but s7 are operational), there remains no roadblocks for migration. During the week since the Amsterdam Hackaton, a fair number of minor issues have been found and fixed, and the general consensus from users is that the environment is functional. On the roadmap for the next month is cleaning up some of the management for replication (so that it is more generalized), finish s7, and help users with their migration issues. Tool Labs work has also added a new feature available to all of Labs: service groups. Service groups are a user/group combination available locally within a project. Service group membership allows regular users to sudo to the service groups, allowing per-project service users, rather than needing to create local users via puppet, or create global users via wikitech. Work progresses on puppetizing more OpenStack services and the OpenStackManager extension. Currently OSM development is hampered by a lack of test installs; soon we should have the ability to easily create new labs instances running Openstack and OpenStackManager for testing and development. Features Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Features_engineering Editor retention: Editing tools
*VisualEditor https://www.mediawiki.org/wiki/VisualEditor* In May, the VisualEditor team worked to complete the major new features we have prioritised over the past few months. Our objective is for VisualEditor to be the default editor for all Wikipedia users, capable of letting them edit the majority of content without needing to use the wikitext editor, in July 2013. We have focussed on four areas of new functionality: adding and editing inclusions of references, templates, categories and media items. Our main area of work over the past month has been on references and templates, and we now have implemented editing them in our experimental code; category editing is nearly complete and should be made available very soon. The deployed alpha version of VisualEditor was updated twice (1.22-wmf4https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf4#VisualEditorand 1.22-wmf5 https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf5#VisualEditor), adding a number of user interface improvements, including further work on the back-end to better support the new features, and fixing a number of bugs.
*Parsoid https://www.mediawiki.org/wiki/Parsoid* In May, the Parsoid https://www.mediawiki.org/wiki/Parsoid team implemented several new features, as well as important performance optimizations in preparation for the July VisualEditor release.
A major image handling overhaul enabled rendering and editing of all image-related parameters with a relatively simple DOM structurehttps://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec#Images. Template and extension editing was improved to support editing of templates within extensions. This lets editors modify and add templated citations in VisualEditor, an important feature to improve the quality of articles in Wikipedia.
On the performance front, we are now reusing expensive template, extension and image expansions from our own previous output to avoid most API queries after an edit. This is necessary to avoid overloading the API when tracking all edits on Wikimedia projects. A cache infrastructure with appropriate purging was set up and will be tested at full load through June. At the Amsterdam hackathon, we helped developers leverage our rich HTML+RDFa DOM outputhttps://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_specfor projects like a Wikipedia-to-SMS service or the Kiwix offline Wikipedia reader http://www.kiwix.org/wiki/Main_Page. Editor engagement features
*Notifications https://www.mediawiki.org/wiki/Echo_%28Notifications%29* In May, we released new features and bug fixes for Notifications on the English Wikipedia and mediawiki.org. Ryan Kaldari, Vibha Bamba and Fabrice Florin collaborated with community members to develop a 'new message indicator'https://en.wikipedia.org/wiki/Wikipedia:Notifications/New_message_indicator, to inform users when someone posts on their talk page. The team also released a new 'Thanks notificationhttps://en.wikipedia.org/wiki/Wikipedia:Notifications/Thanks' that lets editors show their appreciation to users who make helpful edits, and offers a quick way to give positive feedback on Wikipedia. Benny Situ developed a feature that marks messages as read when you visit your talk page and expanded our first metrics dashboardshttp://toolserver.org/%7Edartar/en/echo/, in collaboration with Dario Taraborelli and Aaron Halfaker. New team member Erik Bernhardson developed several new features, such as updating talk page notifications to link directly to their sections. Luke Welling continued to develop HTML Email notificationshttps://www.mediawiki.org/wiki/Echo/Feature_requirements#HTML_single_email_notifications. Fabrice Florin worked with Oliver Keyes to discuss this product with a wide range of community members, and led the team to plan the next stepshttps://docs.google.com/a/wikimedia.org/presentation/d/1LyqKRVH1mWs3OMWOCIL5DdC5DO84sti5udvjeJAFpNw/edit#slide=id.gb26ea508_2_6for Notifications and other editor engagement features. To learn more, visit the project portalhttps://en.wikipedia.org/wiki/Wikipedia:Notifications, read the FAQ pagehttps://en.wikipedia.org/wiki/Wikipedia:Notifications/FAQand join the discussion on the talk page https://en.wikipedia.org/wiki/Wikipedia_talk:Notifications.
*Article feedback https://www.mediawiki.org/wiki/Article_feedback* In May, we developed a few last features and bug fixes on the Englishhttps://en.wikipedia.org/wiki/Special:ArticleFeedbackv5, French https://en.wikipedia.org/wiki/fr:Sp%C3%A9cial:ArticleFeedbackv5and Germanhttps://en.wikipedia.org/wiki/de:Spezial:Artikelr%C3%BCckmeldungen_v5Wikipedias. As requested by community members, Matthias Mullie developed a new opt-in feature that makes it easier to enable or disable feedback on a pagehttps://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Enable.2Fdisable_feedback_on_a_page, as well as new UI improvements to simplify the feedback page, based on designs by Pau Giner and specifications by Fabrice Florin. The team also released new feedback linkshttps://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Feedback_link_on_article_pagesand tested the auto-archivehttps://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Auto-archive_commentsfeature on prototype. For tips on how to use Article feedback, visit the testing page https://www.mediawiki.org/wiki/Article_feedback/Version_5/Testing, and let us know what you think on this talk pagehttps://en.wikipedia.org/wiki/Wikipedia_talk:Article_Feedback_Tool/Version_5. Feature development has now ended for this project, and we will determine our next steps based on the upcoming community vote on the German Wikipedia in coming weeks.
*Flow https://www.mediawiki.org/wiki/Flow_Portal/Project_information* Discussion portals were announcedhttps://blog.wikimedia.org/2013/05/20/flow-next-generation-discussion-system/and opened on three wikis: the English Wikipedia https://en.wikipedia.org/wiki/en:Wikipedia:Flow, Meta-Wikihttps://meta.wikimedia.org/wiki/Flow, and mediawiki.org https://www.mediawiki.org/wiki/Flow_Portal. An interactive prototype http://unicorn.wmflabs.org/flow/ was released to the public for discussion. Discussion is on-going, and the definition of the "minimum viable product" is being worked on. Editor engagement experiments
*Editor engagement experimentshttps://www.mediawiki.org/wiki/Editor_engagement_experiments * In May, the Editor Engagement Experiments team (E3) launched its redesigns of account creation and loginhttps://www.mediawiki.org/wiki/Account_creation_user_experienceforms, after numerous bug fixes and working with local communities to customize the interface as needed. The initial rollout was to 30 of the largest Wikimedia projects in English and other languages, including Wikipedia, Wikimedia Commons, Wikidata, Wikivoyage, Wikispecies, Wikiquote, Wiktionary, Wikibooks, Wikiversity and Wikisource. Complete deployment as the default for all remaining projects was enabled early in June.
Also this month, the team launched and tested a major revamp of the "Getting Started" https://www.mediawiki.org/wiki/Extension:GettingStartedinterface for onboarding new Wikipedians https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians(in English). This included a redesigned landing page, a refactor of the backend to increase speed and stability, a new navigation toolbar on articles that new users were given as their first editing task, and a guided tour https://www.mediawiki.org/wiki/Guided_tour to help them complete their first edit. The results of A/B testinghttps://meta.wikimedia.org/wiki/R:OB5of this new version showed the largest increase in click-through rates for the landing page – up to 32%, a large increase over the 10-12% click-through rate of previous versions. Overall, it also showed a small but statistically significant increase in the rate of first time edits (+1.7%) by new English Wikipedians invited to participate in Getting Started. Last but not least, the PostEdit extensionhttps://www.mediawiki.org/wiki/Extension:PostEdit(previously enabled on most of the top Wikimedia projects by size) was migrated to MediaWiki core. With this change, the post-edit confirmation message https://www.mediawiki.org/wiki/Confirmation_message will be available on all projects, and will be more easily integrated in to VisualEditor https://www.mediawiki.org/wiki/VisualEditor. Support
*2012 Wikimedia fundraiserhttps://www.mediawiki.org/wiki/2012_Wikimedia_fundraiser * The Operations team moved db1025 into the firewalled fundraising cluster (frack), rebuilt it on Precise with MariaDB. RAID monitoring tools were updated to support RAID controllers used in frack. We've mostly finished building/puppetizing the new payments listener (thulium) as well as a new CiviCRM host (barium), both of which are in frack. Language engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Language_engineering
*Language tools https://www.mediawiki.org/wiki/Language_tools* The last round of major bugs have been fixed in the Universal Language Selector (ULS) before we start phased its deployment over the month of June across hundreds of Wikimedia websites. Communications announcements have been started for the first phase of deployment which includes removal of Narayam and WebFonts from sites before ULS is rolled out with the same integrated feature set. Test scenarios for ULS have been identified in detail. Implementation of automated tests is in progress. A combination of ULS integration and cross browser testing is in progress. Final performance tweaks are in progress before we get ready to launch the redesigned home page of Translatewiki.net in June. The monthly version of the Mediawiki Language Extension Bundle (MLEB) was released on May 29.
*Milkshake https://www.mediawiki.org/wiki/Milkshake* jQuery.webfonts and jQuery.IME continue to be in maintenance mode with new input methods added to our repository this month. jQuery.ULS is being actively updated to reflect design changes suggested by the Product team as well as bug fixes.
*Language Engineering Communications and Outreachhttps://www.mediawiki.org/wiki/Language_engineering_communications_and_outreach * The Language Mavens met and discussed areas they can contribute to for Language tools. As part of GSoC 2013, all developers on the team are set to mentor 4 Language Engineering projects supporting jQuery.IME, Language Coverage Matrix, RTL support for Visual Editor and Translate mobile app. Monthly office hours and bug triage were held. The team continued to report and blog on its activities. Hackathon organization and participation at Amsterdam and Tel Aviv were very successful. Mobile https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering
*Wikipedia Zero https://www.mediawiki.org/wiki/Wikipedia_Zero* This month, the team launched Wikipedia Zero with Mobilink in Pakistan, refactored legacy codebase, migrated configuration from monolithic wiki articles to per-carrier JSON configuration blobs, generated utility scripts, patched legacy hyperlink redirect and content rendering bugs, and supported partner on-boarding.
*Mobile Web Photo Uploadhttps://www.mediawiki.org/wiki/Mobile_design/Uploads * This month, we've pushed the 'Nearby' view to the stable version of the mobile site. Now, with a location-aware device, you can easily identify articles close to your current location. We've also made ongoing improvements to photo uploads and the photo upload experience, including improved messaging around image quality and copyright requirements for new uploaders in the beta version of the mobile site. The upload features in general have been a great success, as we've sustained over 1000 unique uploaders http://mobile-reportcard.wmflabs.org/graphs/unique-uploadersover the last two months. We are continuing to experiment in the beta version of the site with improvements to article editing, an improved reorganization of site navigation, Echo notifications, talk pages, and simplifying discovery of article actions (eg editing and watching). We plan to launch these beta features over the course of the next month. Platform Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering MediaWiki Core
*MediaWiki 1.21 https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap* MediaWiki 1.21.0 https://www.mediawiki.org/wiki/MediaWiki_1.21 was released, followed by a MediaWiki 1.21.1 maintenance release.
*MediaWiki 1.22 https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap* MediaWiki 1.22wmf3 https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf3, wmf4 https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf4, and wmf5https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf5were deployed to Wikimedia sites during the month of May. This included one breaking change, gerrit change 49364https://gerrit.wikimedia.org/r/#q,49364,n,z .
*Multimedia https://www.mediawiki.org/wiki/Multimedia* After deploying Score https://www.mediawiki.org/wiki/Extension:Score in April, various performance improvements and fixes were merged in May. To improve scaling of large images, the VipsScaler extensionhttps://www.mediawiki.org/wiki/Extension:VipsScalerwas prepared on the cluster and is ready for deployment in the next weeks. TimedMediaHandlerhttps://www.mediawiki.org/wiki/Extension:TimedMediaHandlerhad various bug fixes and the libraries used to encode WebM videos were updated to improve quality and address encoding issues.
*Wikidata deployment https://www.mediawiki.org/wiki/Wikidata_deployment* Wikidata continued to roll out updates to the client (what lives on the various project wikis) and repository (what powers wikidata.org) software; see the wikidata newsletter for more information: May 3rdhttps://meta.wikimedia.org/wiki/Wikidata/Status_updates/2013_05_03and May 24th https://meta.wikimedia.org/wiki/Wikidata/Status_updates/2013_05_24. Additionally, there was Operations-related work to create helpful URL redirects for entity matching along with fixes to the logging of autopatrolled edits (which was causing massive database usage and issues when doing archival dumps). These URL redirects and autopatrol fixes should go out in June.
*Site performance and architecturehttps://www.mediawiki.org/wiki/Site_performance_and_architecture * Many MediaWiki developers met in Amsterdamhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069569.htmlto discuss architectural principles https://www.mediawiki.org/wiki/Architecture_guidelines and the RFC process.
*Admin tools developmenthttps://www.mediawiki.org/wiki/Admin_tools_development * This month the team worked on Single User Login finalisationhttps://www.mediawiki.org/wiki/Admin_tools_development/SUL_Audit, which will mean that all user accounts will be global across all of Wikimedia's public wikis, and so allowing for cross-wiki notifications and better tools for editors. This will require all user accounts to be uniquely named and not conflict with other accounts. Some more work was done on the global account renaming toolhttps://www.mediawiki.org/wiki/Admin_tools_development/Global_Rename. The team attended the Wikimedia Hackathon 2013 in Amsterdamhttps://www.mediawiki.org/wiki/Amsterdam_Hackathon_2013where we discussed issues with admin tools and plans for the future with a number of volunteer developers.
*Security auditing and responsehttps://www.mediawiki.org/wiki/Security_auditing_and_response * We released MediaWiki 1.20.6/1.19.7 and provided security training for developers at the Amsterdam Hackathon.
*HipHop deployment https://www.mediawiki.org/wiki/HipHop_deployment* Several engineers at the Wikimedia Foundation met with Facebook engineers to discuss potential deployment of HHVM in 2013 (summaryhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069264.html). We formed the HipHop mailing listhttps://lists.wikimedia.org/mailman/listinfo/hiphopto discuss moving forward with this work. Quality assurance
*Quality Assurance https://www.mediawiki.org/wiki/Quality_Assurance* In May, QA worked with a number of parties both in and outside the Wikimedia Foundation to test Echo, VisualEditor, Universal Language Selector, and other projects. We began an official QA mailing listhttps://lists.wikimedia.org/mailman/listinfo/qa .
*Beta cluster https://www.mediawiki.org/wiki/Beta_cluster* In May, Ariel Glenn helped out setting up missing bits of infrastructure to the beta cluster. He added a Redis instance (that holds the job information) and HTTPS support https://en.wikipedia.beta.wmflabs.org. HTTPS will let us write scenarios related to logging in on the wiki or via a mobile device; it will also let us test out OAuth.
udp2log archiving is now working reliably. Max Semenik has set up an EventLogging infrastructure on beta, and Niklas Laxström enabled Universal Language Selector. The Job processing was malfunctioning but was restored.
Since April 30, the MediaWiki instances are using NFS, which is much faster than the previous GlusterFS share; pages serving time went from 560 ms to 260 ms.
Roan Kattouw has deployed Parsoid and VisualEditor on beta. Just like in production, users can enable it in your their preferences under 'Editing'.
*Continuous integrationhttps://www.mediawiki.org/wiki/Continuous_integration * In the beginning of May, Jenkins/Zuul faced overload for a few days; this was resolved by upgrading Zuul and tweaking some time-expensive part of the code. Zuul now lets us define which jobs it triggers by using a predefined template which makes it easier to add new projects. Zuul is now faster to report changes back to Gerrit, which was a complaint during rush hours.
The Wikibase client and repo components are now tested in Jenkins. All puppet repositories are now verifying the puppet manifests and erb templates for syntax validity. The Qunit tests being run for MediaWiki core and VisualEditor seem to be in good shape. PHP CodeSniffer has been upgraded as well as the standard for MediaWiki code. We have yet to enforce it though, since the current code base does not pass the standard.
*Browser testinghttps://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing * In May, we added beta labs as a target for automated browser tests, which allowed us to create tests for the Universal Language Selector at the Hackathon in Amsterdam. We shored up a lot of Jenkins builds at the hackathon as well. We created our first test for VisualEditor, and we are looking forward to working with Rachel Thomas (as part of the Outreach Program for Women) to create more. Analytics
*Analytics infrastructurehttps://www.mediawiki.org/wiki/Analytics/Infrastructure * We continued our efforts of increasing our monitor coverage of the different webrequest dataflows. On the udp2log side, we added monitoring per DC/server rolehttp://ganglia.wikimedia.org/latest/?c=Miscellaneous%20pmtpa&h=emery.wikimedia.org&m=cpu_report&r=hour&s=descending&hc=4&mc=2. Every month, we work on improving the robustness and security of the analytics-related servers that we run: we moved the multicast relay from Oxygen to Gadolinium, we upgraded Oxygen to Ubuntu Precise, and we moved all the Limn-based dashboards from the Kripke labs instance to the Limn0 labs instance. Continous integration for webstatscollector, wikistats and udp-filters now works. The puppet module for Hadoophttps://gerrit.wikimedia.org/r/#/c/61710/has been merged in the Operations reposotiry; this is a big step forward in moving Kraken from beta to production status. Magnus Edenhill demonstrated varnishkafka based on Kafka 0.8; on a local machine varniskafka was able to process 140k msgs/s and we are planning to do production testing mid June. Last, we separated the Kraken machines from the other production servers by installing network ACLs.
*Analytics Visualization, Reporting & Applicationshttps://www.mediawiki.org/wiki/Analytics/Visualization,_Reporting_%26_Applications * For the mobile team, we started collecting pageview countshttp://stats.wikimedia.org/kraken-public/webrequest/mobile/platform/for both official and non-official Wikipedia apps. We changed our Kafka import configuration so that the raw webrequest folders are directly queryable using Hive. The decision was made to re-platform the UMAPI codebase; we have spent quite some time specifying user storieshttps://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=764&view=Features+User+Metrics+APIand had productive discussions about the architecture during the Amsterdam Hackathon. On the development side, the 'page count' metric was introduced. We adapted Ori Livneh's Mediawiki Vagrant VM to also support UMAPI in combination with test data. This will make it much easier to debug issues and open development up to community members. We also fixed numerous stability bugs. Engineering community team
*Bug management https://www.mediawiki.org/wiki/Bug_management* A Wikimedia Labs instance http://boogs.wmflabs.org/ to test Bugzilla software changes and a patch to puppetize Bugzillahttps://gerrit.wikimedia.org/r/#/c/62404/are now available (thanks to Ori Livneh https://en.wikipedia.org/wiki/en:User:Ori.livneh). This will make updating Bugzilla to newer versions and reapplying Wikimedia's custom patches much easier. A MediaWiki Installer bugdayhttps://www.mediawiki.org/wiki/Bug_management/Triage/20130503took place in preparation of the MediaWiki 1.21 release, as well as an IRC office hourhttps://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013-05-13. Andre Klapper worked on a proposal for a Bugzilla admin policyhttps://www.mediawiki.org/wiki/User:AKlapper_%28WMF%29/BugzillaAdminPolicy. In Bugzilla's configuration, the number of Bugzilla administratorshttps://meta.wikimedia.org/wiki/System_administrators/tablehas been decreased in order to improve coordination. As a side-effect of his investigation, Andre documentedhttp://blogs.gnome.org/aklapper/2013/05/28/understanding-bugzilla-groups-and-admin-rights/the meaning of Bugzilla admin rights.
*Mentorship programs https://www.mediawiki.org/wiki/Mentorship_programs* We selected 20 Google Summer of Codehttps://www.mediawiki.org/wiki/Summer_of_Code_2013and 2 Outreach Program for Womenhttps://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_6projects that will be mentored by a total of 32 volunteers. This represents more than double the amount of projects we had last year. We received 69 applications from 60 students for Google Summer of Code 2013, from which 9 were also applying to OPW, and 4 OPW-only individual applications. Google allocated the 21 slots we requested, but we decided to give one back in order to keep a standard on project feasibility.
*Technical communicationshttps://www.mediawiki.org/wiki/Technical_communications * In May, Guillaume Paumier https://www.mediawiki.org/wiki/User:Guillom's major focus was on supporting Tech ambassadorshttps://meta.wikimedia.org/wiki/Tech/Ambassadorsand setting up Tech news https://meta.wikimedia.org/wiki/Tech/News, an initiative aiming to collaboratively monitor recent software changes likely to impact Wikimedians, and distribute a weekly summary, free of technical jargon, to subscribers on their talk page. Two issues of this weekly summary were published this month; starting with the second issue, the content is now distributed in several languages if translations are available. Guillaume also continued to review technical blog posts, and executed the move of the Mobile documentationhttps://www.mediawiki.org/wiki/Technical_communications/Mobile_documentation_consolidationfrom Meta-Wiki to mediawiki.org.
*Volunteer coordination and outreachhttps://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach * Quim Gil https://www.mediawiki.org/wiki/User:Qgil has been preparing a proposal to get automated community metricshttps://www.mediawiki.org/wiki/Community_metricsbased on vizGrimoire http://vizgrimoire.bitergia.org/ and provided by their maintainers, Bitergia http://bitergia.com/. It is currently being discussed with Sumana Harihareswarahttps://www.mediawiki.org/wiki/User:Sharihareswara_%28WMF%29and Rob Lanphier https://www.mediawiki.org/wiki/User:RobLa-WMF for budget approval. Quim also worked on a user-friendly template for the landing page of the wikitech-announcehttps://lists.wikimedia.org/mailman/listinfo/wikitech-announcemailing list that can be used for other Wikimedia lists (source code https://www.mediawiki.org/wiki/User:Qgil/wikitech-announce). He also created a landing page for organizationshttps://www.mediawiki.org/wiki/Mentorship_programs/Organizationswilling to collaborate with Wikimedia to co-organize technical activities. Kiwix http://www.kiwix.org
*The Kiwix project is funded and executed by Wikimedia CHhttps://meta.wikimedia.org/wiki/Wikimedia_CH .* With openZIM http://www.openzim.org, we have finally released the first version of its standard implementation code: the zimlibhttp://www.openzim.org/wiki/Zimlib. Kiwix was introduced in Debian testinghttp://packages.debian.org/search?suite=all&arch=any&searchon=names&keywords=kiwix. A new release of Kiwix for Androidhttps://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile, with a few bug fixes and improvements, was released. Our first GSoC project (ZIM incremental updates for Kiwixhttps://www.mediawiki.org/wiki/User:Kiran_mathew_1993/ZIM_incremental_updates_for_Kiwix) was prepared and accepted; work has already started with Kiranhttps://www.mediawiki.org/wiki/User:Kiran_mathew_1993, the Indian student responsible for this project. Wikidatahttps://meta.wikimedia.org/wiki/Wikidata
*The Wikidata project is funded and executed by Wikimedia Deutschlandhttps://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en .* The Wikidata team worked on 2 major topics in May: the ability to access data from Wikidata in a Wikipedia article by its label and not just its ID, and the ability to enter points in time into Wikidatahttps://www.wikidata.org/wiki/Wikidata:Project_chat#Time_datatype_available_and_short_overview_of_next_steps, which for example now makes it possible to enter the date of birth of a person. Magnus Manske blogged about the tool ecosystemhttp://blog.wikimedia.de/2013/05/06/the-wikidata-tool-ecosystem/that is building around Wikidata. During the next 3 months, the team will be workinghttp://lists.wikimedia.org/pipermail/wikidata-l/2013-May/002293.htmlwith 3 Google Summer of Code students, and 2 other students will be working with other organizations on Wikidata-related projects. The codebase has been reviewedhttp://lists.wikimedia.org/pipermail/wikidata-tech/2013-May/000006.htmlby Qafoo. Wikidata-tech https://lists.wikimedia.org/mailman/listinfo/wikidata-tech, a new mailing list for discussions related to the development around Wikidata, was created. Additionally, the team attended the hackathon in Amsterdam to give tutorials and meet other developers. A prototype of a multilingual map http://4thmain.github.io/d3hacks/wiki-atlas.html was built there, among other things. Future The engineering management team continues to update the *Deploymentshttps://wikitech.wikimedia.org/wiki/Deployments * page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the *engineering roadmaphttps://www.mediawiki.org/wiki/Roadmap *, listing ongoing and future Wikimedia engineering efforts. -- Guillaume Paumier Technical Communications Manager — Wikimedia Foundation https://donate.wikimedia.org