Hi,

The report covering Wikimedia engineering activities in November 2013 is now available.

Wiki version: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/December Blog version: https://blog.wikimedia.org/2014/01/21/engineering-report-december-2013/

We're also proposing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/December/su...

Below is the HTML text of the report.

As always, feedback is appreciated on the usefulness of the report and its summary, and on how to improve them.

------------------------------------------------------------------

Major news in December include:

- a retrospective on Language Engineering eventshttps://blog.wikimedia.org/2013/12/10/language-engineering-events-language-summit-fall-2013/, including the language summit in Pune, India; - the launch of a draft featurehttps://blog.wikimedia.org/2013/12/20/new-draft-feature/on the English Wikipedia, to provide a gentler start for Wikipedia articles.

*Note: We’re also providing a shorter, simpler and translatable version of this report https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/December/summary that does not assume specialized technical knowledge.*

Personnel Work with us https://wikimediafoundation.org/wiki/Work_with_us

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

- VP of Engineering http://hire.jobvite.com/Jobvite/Job.aspx?j=ods8Xfwu - Software Engineer – Growthhttp://hire.jobvite.com/Jobvite/Job.aspx?j=o8NJXfwl - Software Engineer – VisualEditor (Features)http://hire.jobvite.com/Jobvite/Job.aspx?j=oqo6XfwB - Software Engineer – Language Engineeringhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH - Software Engineer http://hire.jobvite.com/Jobvite/Job.aspx?j=o09WXfwM - QA Automation Engineerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oe09Yfw5 - Full Stack Developer – Analyticshttp://hire.jobvite.com/Jobvite/Job.aspx?j=okwbYfwJ - Analytics – Product Managerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=opkhYfwI - Operations Engineerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oX6gYfw1 - Sr. Operations Engineerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=o66gYfwa - Operations Security Engineerhttp://hire.jobvite.com/Jobvite/Job.aspx?j=oT6cYfwT - Graphic Design Interns – Paidhttp://hire.jobvite.com/Jobvite/Job.aspx?j=ohN3XfwO

Announcements

- Sherah Smith joined the Wikimedia Foundation as Fundraising Engineer in Features Engineering (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-December/073624.html ). - Kunal Mehta https://www.mediawiki.org/wiki/User:Legoktm joined the Wikimedia Foundation as contractor in Features Engineering (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-December/073623.html ). - Andrew Green https://en.wikipedia.org/wiki/User:AndyRussG joined the Wikimedia Foundation as a contractor in Features Engineering, working on the Education Program (announcementhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-December/073593.html ).

Technical Operations

*Datacenter RFP https://wikimediafoundation.org/wiki/RFP/2013_Datacenter* As part of our ongoing work in selecting a new location for our next Datacenter, members of the team traveled to several candidate locations throughout the US to tour facilities, meet facility staff, and otherwise continue the selection process. Following this process, we have been able to shortlist our bid proposals, and have begun the final selection process. We hope to complete bid selection and legal review in January.Work continues on migrating our remaining services to our Ashburn datacenter. Consolidation and migration of databases, fundraising infrastructure, Labs, as well as progress on updating the configuration (puppetization) of several miscellaneous services was accomplished in December Additionaly, “triage” of the hardware within the facility was performed, with an eye towards what will be delivered to Ashburn, what will end up in our new facility, and what will be decommissioned.

*Wikimedia Labs https://www.mediawiki.org/wiki/Wikimedia_Labs* Andrew Bogott purged of empty projects and stale instances, resulting in more accurate usage statistics for Labs:

- Number of projects: 140 - Number of instances: 403 - Amount of RAM in use (in MBs): 1,592,832 - Amount of allocated storage (in GBs): 21,525 - Number of virtual CPUs in use: 797 - Number of users: 2,425

Tool Labs saw a bump in usage as the winter holidays provided an opportunity for volunteers to migrate tools from the Toolserver and work on new projects; there are now 531 tools managed by 435 users, ranging from simple database queries to elaborate editing adjuncts using the new OAuth infrastructure.Work for the impending migration of Labs to the Ashburn data center is well on its way: hardware is set up, the new storage servers are configured, and a lot of fresh OpenStack puppet manifests are in progress. Features Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Features_engineering Editor retention: Editing tools

*VisualEditor https://www.mediawiki.org/wiki/VisualEditor* In December, the VisualEditor team worked to continue the improvements to the stability and performance of the system, and to add new features. The deployed version of the code was updated three times (1.23-wmf6https://www.mediawiki.org/wiki/MediaWiki_1.23/wmf6#VisualEditor, 1.23-wmf7 https://www.mediawiki.org/wiki/MediaWiki_1.23/wmf7#VisualEditorand 1.23-wmf8 https://www.mediawiki.org/wiki/MediaWiki_1.23/wmf8#VisualEditor). Most of the team’s focus was on major new features and fixing bugs. There is now basic support for rich copy-and-paste from external sources into VisualEditor, and a basic tool to insert characters not available on users’ keyboards. Work also continued on a dialog for quickly adding citation templated references, and on some major infrastructure changes, splitting out the core of VisualEditor from the MediaWiki-specific items like the transclusion editor.

*Parsoid https://www.mediawiki.org/wiki/Parsoid*

In December, the relentless Parsoid team continued squashing bugs and incompatibilities; see our deployments pagehttps://www.mediawiki.org/wiki/Parsoid/Deploymentsfor details. During the node 0.10 migration, we ran into some issues caused by changed garbage collector behavior, and rolled back to 0.8. We spent some time investigating and fixing this; initial testing on our round-trip testing setup indicates that this is now fixed.

Our testing infrastructure is now exercising the entire stack including the web server, which will help to make sure that we also catch issues in HTTP libraries before deployment.

We wrote several RFCs about embracing a service architecturehttps://www.mediawiki.org/wiki/Requests_for_comment/Services_and_narrow_interfaces, PHP bindings for serviceshttps://www.mediawiki.org/wiki/Requests_for_comment/PHP_Virtual_REST_Service, a general-purpose storage servicehttps://www.mediawiki.org/wiki/Requests_for_comment/Storage_servicebased on our Rashomon revision store, and a public content API based on thishttps://www.mediawiki.org/wiki/Requests_for_comment/Content_API .

Part of the team worked on a new PDF rendering infrastructure using Parsoid HTML, node and PhantomJS. Part of the team has also been mentoring two Outreach Program for Women (OPW) interns. Core Features

*Flow https://www.mediawiki.org/wiki/Flow/Project_information* In December, we deployed Flow to a few selected pages in production ( Talk:Flow https://www.mediawiki.org/wiki/Talk:Flow and Talk:Sandboxhttps://www.mediawiki.org/wiki/Talk:Sandboxon mediawiki.org) and collected feedback about the features and design to date from the community. The results of the feedback period are summarized at Flow/Research#Experienced Users https://www.mediawiki.org/wiki/Flow/Research#Experienced_Users. Throughout the feedback period, we worked on implementing design changes – such as a more compact view of the board and different affordances for topic and post actions, as well as different visualizations of history information – based on the comments of users testing the software. We also began a straw poll about launching Flow as a beta trial in the discussion spaces of WikiProject Breakfast, WikiProject Hampshire, and WikiProject Video Games on English Wikipedia. Based on the outcome of these polls, we hope to deploy Flow to those pages in January. Growth

*Growth https://www.mediawiki.org/wiki/Growth* In December, the Growth team spent time working on product development and research for upcoming Wikipedia article creationhttps://www.mediawiki.org/wiki/Wikipedia_article_creationimprovements. First and foremost, the team fulfilled a request from the English Wikipedia community to launch the new Draft namespacehttps://www.mediawiki.org/wiki/Draft_namespacethere. Pau Giner and others on the team simultaneously began design work on future improvements to drafts functionality (see blog posthttps://blog.wikimedia.org/2013/12/20/new-draft-feature/), including recruiting for usability testing sessions. Support

*Wikipedia Education Program https://www.mediawiki.org/wiki/Wikipedia_Education_Program* In December, we improved and fixed issues with the current Education Program extension, and continued preliminary work towards a new version of the software. We added a message on Special:Contributions about users’ participation in courses, fixed a bug involving course undeletion and tweaked related styling, addressed a breaking change in core, improved i18n (in collaboration with Language Engineering) and began work on notifications for course-related events. We also fleshed out more ideas about the new version and possible synergies with other existing and proposed functionality, and reached out to other teams for input on this. Mobile https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering

*Wikipedia Zero https://www.mediawiki.org/wiki/Wikipedia_Zero* During the last month, the team implemented a global landing page redirector for mobile Wikipedia website access, added support for staged configuration submittals, enhanced interstitials based on input from the field from Wikipedia Zero markets, amended compression proxies support, analyzed USSD/SMS service and partner launch-related access, and added general bugfixes. The team also worked toward a generic JSON configuration extension for use by extension like ZeroRatedMobileAcces, and started on an HTML5 webapp proof of concept as an option for rebooting the Firefox OS app.

*Mobile web projects https://www.mediawiki.org/wiki/Mobile_web_projects* We’ve been working on finishing the redesign of the overlays. Additionally we’ve continued work on mobile on-boarding. The “Keep going” feature has been changed to a workflow that asks users to add blue links and includes a tutorial. This is consistent with what we’ve learned about how guiding users helps accomplish more edits, and it fits into more of micro contributory workflow that we want to experiment with. We’ve also worked on an A/B test displaying an edit guider for users signing up from the left nav menu. This is mirroring the edit guider that displays for users signing up through the edit call to action. It also is consistent with the behavior that the desktop site will be displaying to users as a result of the OB6 A/B testhttps://meta.wikimedia.org/wiki/Research:Onboarding_new_Wikipedians/OB6 .

*Wikimedia Apps https://www.mediawiki.org/wiki/Wikimedia_Apps* The team added saved pages, article navigation, and language support to the mobile Wikipedia app. During the quarterly planning meeting, it was decided to postpone photo uploads from our market release plan in favor of text editing. Language Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Language_engineering Development of the TwnMainPage https://www.mediawiki.org/wiki/Extension:TwnMainPageextension was completed; Translatewiki.net https://translatewiki.net/wiki/ now has a new user registration process and a new dashboard for translators that provides insight in a user’s activity compared to that of other users.Plural rules for MediaWiki have been updated to comply with CLDR version 24http://cldr.unicode.org/index/downloads/cldr-24. There were consequences for existing translations in Russian, languages that fall back to Russian, Serbian, Belarus and Ukrainian. These have been updated semi-automatically, and past contributors have been informed and askedhttps://translatewiki.net/wiki/Thread:Portal_talk:Ru/Plural_changes_in_many_languagesto help in reviewing the updates. MediaWiki Language Extension Bundle 2013.12 https://www.mediawiki.org/wiki/MLEB was released. It is compatible with MediaWiki 1.21 and MediaWiki 1.22. The MediaWiki language extension bundle provides easy way to bring ultimate language support to your MediaWiki. The bundle is a collection of selected few MediaWiki extensions needed by any wiki which desires to be multilingual.A performance issue https://bugzilla.wikimedia.org/53748 in the Translate extension that prevented use of the status field for translatable pages on Meta-Wiki was resolved. Platform Engineeringhttps://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering MediaWiki Core

*Search https://www.mediawiki.org/wiki/Search* We’ve continued our aggressive rolld-out of Cirrus as a Beta Feature. You can search now 52% of pages including Commons and Wikidata via CirrusSearch. We’ve fallen back somewhat on our goal to make Cirrus the primary search engine. Right now, we only handle about 1.5% of search traffic. While we will be switching more wikis over to Cirrus as the primary search back-end in January, the theme of the month really is adding Cirrus as a Beta Feature to more wikis, including the English Wikipedia. We’re not sure how many wikis we’ll be able to add before we consider ourselves out of hardware space. We’re planning on 50% more servers in February so we’ll likely be able to finish adding wikis then.

*Site performance and architecture https://www.mediawiki.org/wiki/Site_performance_and_architecture*

The team wrapped up the Puppetization of Graphite and its migration to Ashburn, and configured Travis CI to run MediaWiki’s test suite under HHVM on each commit to core. They also added an initial HHVM role for MediaWiki-Vagrant and re-wrote MediaWiki’s profiling data aggregator to be more performant. Prior to the rewrite, it was constantly saturated and would drop data; the rewrite reduced average CPU utilization by more than two thirds.

*Auth systems https://www.mediawiki.org/wiki/Auth_systems* The team implemented performance fixes for CentralAuth to reduce the number of calls by anonymous users.

*Wikimania Scholarships app https://www.mediawiki.org/wiki/Wikimania_Scholarships_app* All critical functionality and several stretch goals were reached for the “final” version, which deployed to production on 2013-12-19. Chad Horohoe stepped in in the final days leading up to launch and helped save the i18n features that were scheduled to be scrapped due to time constraints. Siebrand Mazeland and the wonderful volunteers at translatewiki.net are providing translations at a rapid pace. Bryan Davis also put in some extra hours to clean up the look and feel of the application with a new Bootstrap-based theme. The application period for Wikimania 2014 will open at 2014-01-06T00:00:00Z and continue until 2014-02-17T23:59:59Z. The team will continue to monitor and support the product through the application period, and the subsequent review and approval process of the Scholarship Committee.

*Security auditing and response https://www.mediawiki.org/wiki/Security_auditing_and_response* We continued to respond to reported security issues, and completed security reviews of Flow, the Wikimania Scholarships app, and the GLAM Wiki Toolset.

*Admin tools development https://www.mediawiki.org/wiki/Admin_tools_development* The team made several small improvements, including log entries, the addition of global groups to Special:CentralAuth, and the addition of global edit count to Special:MultiLock.

*Release & QA https://www.mediawiki.org/wiki/Wikimedia_Release_and_QA_Team*

In December, the latest and greatest version of MediaWiki was released, 1.22https://www.mediawiki.org/wiki/Release_notes/1.22. This was lead by Mark Hershberger and Markus Glaser, working as the MediaWiki release team, along with help from the Wikimedia Foundation Release and QA team (specifically Greg Grossmeier and Antoine Musso). Of course, this was only possible because of the great work by all of the MediaWiki developers.

The QA team, along with Multimedia team, is working on API level tests starting with UploadWizard. This is close to being done. Another API level testing activity is Parsoid, with help from VE and CI (Antoine) teams.

You can take a look at the first draft of the updated Development and Deployment process flow charthttps://www.mediawiki.org/wiki/User:Greg_%28WMF%29/dev_deploy_flow . Quality assurance

*Quality Assurance https://www.mediawiki.org/wiki/Quality_Assurance* In December, the Quality Assurance team worked particularly closely with the Mobile team, both supporting automated testing and also helping fix issues with Beta labs and with Jenkins. We continued to work with the teams from Language engineering, VisualEditor, Flow, Multimedia, Wikidata, and Search, as well as participated in the Google Code-In event. We are in the process of creating new support not only for automated browser testing, but also for API testing, test data creation, and monitoring of both test and production environments.

*Beta cluster https://www.mediawiki.org/wiki/Beta_cluster* Parsoid on the Beta cluster is now based on the mediawiki/services/parsoidrepository and is properly self-updating whenever a change is merged in that repository via a Jenkins jobhttps://integration.wikimedia.org/ci/job/beta-parsoid-update/. Beta labs played a key role in finding and fixing some significant errors that, in combination, were causing users to see 503 errors in production, particularly on large pages and for Mobile users. For one thing, some timeouts on the Varnish caches had been set too low. We had increased those for the text Varnish servers but had not done so for Mobile Varnish servers. A tricky bug was also causing parts of large pages to be parsed multiple times. Last, the browser tests that incurred the 503 errors should have been capable of ignoring them. Thanks to Beta labs, the Varnish server timeouts are now correct, the multiple-parsing bug is addressed and the browser tests for MobileFrontend are running correctly.

*Browser testing https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing* Besides ongoing regression testing of Wikipedia features in cross-browser tests, in December we made the first steps for new abilities like testing geolocation for Mobile tests, testing and monitoring upload ability in production, adding the ability to create test data via the API, running tests in PhantomJS on the WMF Jenkins server, and monitoring the Beta labs test environment for fatal errors. Engineering Community Teamhttps://www.mediawiki.org/wiki/Engineering_Community_Team

*Bug management https://www.mediawiki.org/wiki/Bug_management* Quim Gil and Andre Klapper continued to run and coordinate Google Code-Inhttps://www.mediawiki.org/wiki/Google_Code-Infor Wikimedia. Andre’s draft for a Bugzilla etiquette https://www.mediawiki.org/wiki/Bug_management/Bugzilla_etiquettereceived lively feedback and discussion. On the technical side, Daniel Zahn prepared the migration of bugzilla.wikimedia.org to WMF’s new data centerhttps://wikitech.wikimedia.org/wiki/Tampa_clusterby turning the existing rudimentary Bugzilla puppet code into a puppet modulehttps://gerrit.wikimedia.org/r/#/q/project:operations/puppet+topic:bugzilla+owner:dzahn+status:merged,n,zand automatically generating documentation on doc.wikimedia.orghttps://doc.wikimedia.org/puppet/classes/bugzilla/bugzilla.html. As part of this preparation, Daniel and Andre also eliminated nearly all Perl CPAN modules (in Bugzilla’s /lib subfolder) on the new server by using default distribution packages instead. Furthermore, Andre worked on a preliminary patch https://bugzilla.wikimedia.org/show_bug.cgi?id=22170#c27 to display some common queries on the Bugzilla front page.

*Project management tools review https://www.mediawiki.org/wiki/Project_management_tools/Review* Andre Klapper and Guillaume Paumier kicked off an evaluation of Wikimedia’s project management toolshttps://www.mediawiki.org/wiki/Project_management_tools/Review. Guillaume prepared a consultation pagehttps://www.mediawiki.org/wiki/Talk:Project_management_tools/Reviewwith topics for stakeholders and improved it together with Andre. It will initially be sent to the teampracticeshttps://lists.wikimedia.org/mailman/listinfo/teampracticesmailing list and individual stakeholders. To facilitate getting input, talking to individual stakeholders via Hangouts and holding an IRC discussion https://meta.wikimedia.org/wiki/IRC_office_hours are also considered.

*Mentorship programs https://www.mediawiki.org/wiki/Mentorship_programs*

Wikimedia’s first participation in the Google Code-Inhttps://www.mediawiki.org/wiki/Google_Code-Inprogram required a lot of dedication from the ECT members, and about a dozen of mentors and other contributors helping creating and reviewing tasks. Students completed about 200 tasks. The GCI inertia and the lessons learned will help us organize a better gateway for new contributors, which was a main reason for us to join this program. We also believe that the experience acquired will help us make future editions as successful with less work.

Round 7 of the FOSS Outreach Program for Womenhttps://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_7started and all projects and on track so far:

- Compacting interlanguage linkshttps://www.mediawiki.org/wiki/User:Niharika/Project_Progress_Report#December_2013 . - MediaWiki Homepage Redesignhttps://www.mediawiki.org/wiki/User:Monteirobrena/MediaWiki_Homepage_Redesign/Monthly_Reports#December . - Complete mediawiki API development course on codecademyhttps://www.mediawiki.org/wiki/User:Diwanshipandey/OPW_Internship_Report#December_2013 . - Clean up Parsoid round-trip testing UIhttps://www.mediawiki.org/wiki/User:5xbe/OPW_Monthly_Progress_Reports#December_10th_-_31st_.28three_weeks.29 . - Clean up tracing/debugging/logging inside Parsoidhttps://www.mediawiki.org/wiki/User:Mariapacana/OPW_Progress_Report#December_2013 . - UploadWizard :OSM Embeddinghttps://www.mediawiki.org/wiki/User:Inchikutty/OPW_Internship_Report#December

We joined Facebook Open Academyhttps://www.mediawiki.org/wiki/Facebook_Open_Academyalmost at the last minute thanks to a reminder from developer Tyler Romeo. Six projects were accepted, which will be developed by teams of university students during the first half of 2014:

- Distributed cron replacementhttps://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects#Distributed_cron_replacement— Coren https://www.mediawiki.org/wiki/User:Coren - Cassandra backend for distributed round-trip test serverhttps://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects#Cassandra_backend_for_distributed_round-trip_test_server— Gabriel Wicke https://www.mediawiki.org/wiki/User:GWicke, Marc Ordinas i Llopishttps://www.mediawiki.org/wiki/User:Marcoil - Flow Right-To-Left language supporthttps://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects#Flow_Right-To-Left_language_support— S Page (WMF) https://www.mediawiki.org/wiki/User:S_Page_%28WMF%29 and Werdna https://www.mediawiki.org/wiki/User:Werdna - Flow Edit Filter integrationhttps://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects#Flow_Edit_Filter_integration— S Page (WMF) https://www.mediawiki.org/wiki/User:S_Page_%28WMF%29 and Werdna https://www.mediawiki.org/wiki/User:Werdna - OpenBadges https://www.mediawiki.org/wiki/OpenBadges and Personahttps://www.mediawiki.org/wiki/Extension:Personasupport for MediaWiki — Parent5446 https://www.mediawiki.org/wiki/User:Parent5446 and Qgilhttps://www.mediawiki.org/wiki/User:Qgil

*Technical communications https://www.mediawiki.org/wiki/Technical_communications* In December, Guillaume Paumier https://www.mediawiki.org/wiki/User:Guillom‘s primary focus was on creating and assigning tasks for the Google Code-inhttps://www.mediawiki.org/wiki/Google_Code-inprogram, mentoring students and reviewing their work. They worked on writing discovery reportshttps://www.mediawiki.org/wiki/Category:Discovery_reports, adding TemplateData https://www.mediawiki.org/wiki/Help:TemplateData to widely-used Wikipedia templates, and converting manually-translated pages on mediawiki.org to pages using the Translate extension. Guillaume also continued to provide ongoing communications supporthttps://www.mediawiki.org/wiki/Technical_communications/Tech_blog_activityfor the engineering staff, and assemble and publish the weekly technical newsletter https://meta.wikimedia.org/wiki/Tech/News, which is now delivered across wikis using MassMessagehttps://meta.wikimedia.org/wiki/MassMessage. Last, he compiled readability metricshttps://meta.wikimedia.org/wiki/Tech/News/Readabilityfor all past issues of the newsletter, as well as translation and subscribers https://meta.wikimedia.org/wiki/Tech/News/Subscribers metrics.

*Volunteer coordination and outreach https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach* We reached all our goals for submissions at FOSDEMhttps://www.mediawiki.org/wiki/Events/FOSDEMin Brussels (February 1−2): a fully scheduled Wikis devroom https://fosdem.org/2014/schedule/track/wikis/, a main track session (The Wikipedia Stackhttps://fosdem.org/2014/schedule/event/the_wikipedia_stack/) by Erik Moeller, and the Wikimedia standhttps://www.mediawiki.org/wiki/Events/FOSDEM#Wikimedia_standcoordinated by Dimitar Dimitrov https://meta.wikimedia.org/wiki/User:Dimi_z. Our hiring process for a technical writer contractorhttp://lists.wikimedia.org/pipermail/wikitech-l/2013-November/073077.htmlwas unsuccessful. After screening dozens of candidates and interviewing several of them, our three final candidates declined for various reasons. Without time to hire a writer before the Architecture Summit 2014https://www.mediawiki.org/wiki/Architecture_Summit_2014, we decided to hold the search for now. Multimedia

*Multimedia https://www.mediawiki.org/wiki/Multimedia*

In December, Mark Holmquist and Gergő Tisza updated the beta version of the Media Viewer https://www.mediawiki.org/wiki/Multimedia/About_Media_Viewer, based on new designs by Pau Giner. This new versionhttps://www.mediawiki.org/wiki/Multimedia/Media_Viewer#Next_Versionnow features next and previous arrows, as well as faster image load and an enhanced metadata panel, as shown on this demo pagehttps://www.mediawiki.org/wiki/Lightbox_demo .

Fabrice Florin managed product development, spearheaded the team’s Multimedia Quarterly Review meetinghttps://commons.wikimedia.org/wiki/File:Multimedia_Quarterly_Review_12-03-2013.pdf, hosted more roundtable discussionshttps://meta.wikimedia.org/wiki/Roundtables/Roundtable_5and presented a Multimedia Vision 2016https://commons.wikimedia.org/wiki/File:Multimedia_vision_2016.webmto get more community feedback about our goals, with help from volunteer Aaron Arcos.

Bryan Davis, Aaron Schulz and other team members helped Dan Entous and David Haskiya release a first version of the GLAM Toolsethttps://commons.wikimedia.org/wiki/Commons:GLAMToolset_project#Goal_1:_GLAM_Upload_Systemfor batch uploads by museum curators. We also started work on fixing bugs for the Upload Wizard, which we’ll aim to improve as our primary focus this quarter.

Last but not least, we are delighted to welcome Gilles Dubuc, who is joining our multimedia team https://www.mediawiki.org/wiki/Multimedia as senior software engineer. To discuss these features and keep up with our work, we invite you to join the multimedia mailing listhttps://lists.wikimedia.org/mailman/listinfo/multimedia. . Analytics https://www.mediawiki.org/wiki/Analytics

*Kraken https://www.mediawiki.org/wiki/Analytics/Kraken* In late December, the Analytics team partnered with Operations to enable log delivery over Kafka (distributed message bus). All logs from the edge caches serving mobile traffic are now delivered via Kafka into a data warehouse on our Hadoop infrastructure. We’re seeing 3−4K messages per second, with a maximum of 8K/sec over Christmas. This is a significant step towards our goals of building an infrastructure that can be used for analysis of all of our page views.

*Wikimetrics https://www.mediawiki.org/wiki/Analytics/Wikimetrics* The team added a small but important feature to Wikimetrics in December: the ability to authenticate against MediaWiki OAuth. This allows users to sign up for Wikimetrics without relying on a third party for authentication and is an early adoption of MediaWiki OAuth.

*Data Quality https://www.mediawiki.org/wiki/Analytics/Data_Quality* The team continues to spend a large amount of time on data quality. The primary effort in December was in isolating and fixing an error in WikiStats that inflated page views from July to December by a significant amount. The error was patched in early December and the statistics were recalculated. There were also issues with Wikipedia Zero traffic and an outage caused by a single point of failure in the legacy infrastructure.

*Research and Data https://www.mediawiki.org/wiki/Analytics/Research_and_Data*

This month, we kicked off a series of monthly research showcases as an opportunity for the team to share what we’re learning about Wikimedia editors and projects, and new features and programs the Foundation is rolling out. Aaron Halfaker presented research on anonymous editorshttps://commons.wikimedia.org/wiki/File:Anonymous_editors_-_WMF_R%26D_showcase_%28Dec._2013%29.pdf. The first showcase was targeted at an internal audience but we’re considering making future showcases open to anyone via a public stream.

We analyzedhttps://docs.google.com/a/wikimedia.org/document/d/1kpJrfataS5KAxGXFoygQVhMlzFftjsvX9HktSAAKfrQthe cause and impact of major over-reporting on page views in the last months of 2013. We filtered bogus traffic from the data, and published updated reports http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm.

We also continued work on metrics standardizationhttps://www.mediawiki.org/wiki/Analytics/Epics/Editor_Engagement_Vital_Signsand presentedhttps://commons.wikimedia.org/wiki/File:Metrics_Standardization_10_Dec_2013.pdfthe rationale for this project and the results of the initial round of analysis we conducted.

This month also saw the completion of the third volume of the research newsletter https://meta.wikimedia.org/wiki/R:Newsletter, which this year covered a total of 196 publications reviewed by volunteer contributors. A retrospective of research covered in the newsletter in 2013 will be published later in January. Offline

*PDF rendering https://www.mediawiki.org/wiki/PDF_rendering* Work started on this project aiming to replace the back-end renderer for the Collection extension (mwlib). This is the renderer that creates the PDFs for the ‘Download to PDF’ sidebar link and creates books (downloadable in multipe formats and printable via PediaPress), using Special:Book. One of the goals is to take advantage of Parsoid to do the parsing from wikitext. The team worked on the new parser, the ‘Collection Offline Content Generator’. The team will continue to work on this project over the coming weeks. Read morehttp://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/74051in the mailing list thread.

*Kiwix http://www.kiwix.org* *The Kiwix project is funded and executed by Wikimedia CH https://meta.wikimedia.org/wiki/Wikimedia_CH.* We have released two new versions of Kiwix for Androidhttps://play.google.com/store/apps/details?id=org.kiwix.kiwixmobilethis month (1.7 & 1.8), providing many new features; most of them were developed by young new developers as part of the Google Code-in programhttps://www.mediawiki.org/wiki/Google_Code-in. Work continues around tools based on Parsoid output, especially as we need to rewrite the ZIM-related code for the MediaWiki offline toolchainhttps://www.mediawiki.org/wiki/PDF_rendering, currently under heavy re-engineering. We have compiled download stats for 2013, and for the first time we have reached 700,000 downloads of the Kiwix app a year. Work to digitally sign the OSX and Windows binaries is ongoing and is the last step before releasing 0.9rc3. We have started experimenting with porting Kiwix-plug http://www.kiwix.org/wiki/Kiwix-plug to RaspberryPi http://www.raspberrypi.org/, and it looks good. Lots of new ZIM files were generated; we now generate a ZIM file of Wiktionaryhttp://download.kiwix.org/zim/wiktionary/, as well as ZIM files without pictures. Wikidata https://meta.wikimedia.org/wiki/Wikidata

*The Wikidata project is funded and executed by Wikimedia Deutschland https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en.* The Wikidata development team continued to work on quantity values, including localization, support for scientific notation and the user interface. They also worked on performance by improving caching and database handling. DataValues Serialization 0.1https://github.com/DataValues/Serialization/was released, as well as Ask Serialization 1.0 https://github.com/wmde/AskSerialization and Wikibase DataModel 0.6http://www.bn2vs.com/blog/2013/12/23/wikibase-datamodel-released/. A new DataModel serializationhttps://github.com/wmde/WikibaseDataModelSerializationcomponent was started, which will allow authors and people analyzing dumps to have the deserialization task solved for them. Future The engineering management team continues to update the *Deployments https://wikitech.wikimedia.org/wiki/Deployments* page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the *annual goals https://www.mediawiki.org/wiki/Wikimedia_Engineering/2013-14_Goals*, listing ongoing and future Wikimedia engineering efforts. ------------------------------

*This article was written collaboratively by Wikimedia engineers and managers. See revision history https://www.mediawiki.org/w/index.php?title=Wikimedia_engineering_report/2013/December&action=history and associated status pages. A wiki version https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/December is also available.*