Wikitech-l February 2013

wikitech-l@lists.wikimedia.org

161 participants
170 discussions

Wikimedia engineering January 2013 report

by Guillaume Paumier

Hi, The report covering Wikimedia engineering activities in January 2013 is now available. Wiki version: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/January Blog version: https://blog.wikimedia.org/2013/02/07/engineering-january-2013-report/ We're also proposing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge: https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/January/su… Below is the full HTML text of the report, as previously requested. As always, feedback is appreciated about the usefulness of the report and its summary, and on how to improve them. ------------------------------------------------------------------ Major news in January include: - the successful migration of our main services<https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-…>to our data center in Ashburn, Virginia; - new features<https://blog.wikimedia.org/2013/01/11/mobile-beta-a-sandbox-for-new-experim…>available in our mobile beta; - progress on input methods<https://blog.wikimedia.org/2013/01/25/language-engineering-progress-with-in…>and our upcoming translation interface<https://blog.wikimedia.org/2013/01/11/a-more-efficient-translation-interfac…> ; - the announcement of GeoData<https://blog.wikimedia.org/2013/01/31/geodata-a-new-age-of-geotagging-on-wi…>, a feature to attach geo-coordinates to Wikipedia and Wikivoyage articles; - a testing event to assess how VisualEditor handles non-Latin characters<https://blog.wikimedia.org/2013/01/28/help-us-test-and-investigate-visualed…> . *Note: We're also proposing a shorter, simpler and translatable version of this report<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/January/su…>that does not assume specialized technical knowledge. * Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us> Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles. - Software Engineer - Editor Engagement<http://hire.jobvite.com/Jobvite/Job.aspx?j=ovvXWfwD> - Technical Writer - (Contract)<http://hire.jobvite.com/Jobvite/Job.aspx?j=oGH5Wfw8> - Software Developer - Fundraising<http://hire.jobvite.com/Jobvite/Job.aspx?j=oF2EWfw1> - Software Engineer (Partners)<http://hire.jobvite.com/Jobvite/Job.aspx?j=oX2hWfwW> - Software Engineer (Apps)<http://hire.jobvite.com/Jobvite/Job.aspx?j=oqU0Wfw0> - Software Developer General (Mobile)<http://hire.jobvite.com/Jobvite/Job.aspx?j=o4cKWfwG> - Software Engineer - Multimedia<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3> - Software Engineer (Search)<http://hire.jobvite.com/Jobvite/Job.aspx?j=ogk1Wfwh> - Product Manager (Mobile)<http://hire.jobvite.com/Jobvite/Job.aspx?j=oGWJWfw1> - Director of User Experience<http://hire.jobvite.com/Jobvite/Job.aspx?j=otv0WfwE> - Visual Designer <http://hire.jobvite.com/Jobvite/Job.aspx?j=oomJWfw9> - Operations Engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf> - Operations Engineer/Database Administrator<http://hire.jobvite.com/Jobvite/Job.aspx?j=obMOWfwr> - Site Reliability Engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=o7k2Wfw9> - Tools Lab Operations Engineer (Contractor)<http://hire.jobvite.com/Jobvite/Job.aspx?j=o7y3Wfwo> Announcements - Yuvaraj Pandian re-joined the Mobile engineering team<https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>as Software developer ( announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065636.html>). He joined the newly created Mobile App team with Brion Vibber and Shankar Narayan. - Munagala Ramanath (Ram) joined the MediaWiki core team of the Platform engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering>group as Senior Software Engineer ( announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065698.html> ). - Runa Bhattacharjee joined the Language Engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineering>team as Outreach and QA coordinator ( announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/066030.html> ). Technical Operations *Production Site Switchover<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning> * The Wikimedia Foundation switched over its primary data center<http://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-c…>from Tampa, Florida to Ashburn, Virginia on January 22. Given the scale and complexity<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning/Steps>of the migration, we scheduled three 8-hour windows to perform the migration, but we were able to complete it<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065892.html>on the first attempt. Because the switchover involved, among other things, moving over the master databases from Tampa to Ashburn, the site was set to 'read-only' mode for about 32 minutes. During that period, the site was available but no new contents were created, edited or uploaded. As expected, there was some minor fallout of the migration, mostly due to configuration changes, but they were quickly contained by the Engineering and Operation teams.With this migration, Tampa data center will now be our fail-over site and we plan to perform site fail-over tests every few months. There are remaining small non-core applications still using Tampa as the primary site, such as RT, etherpad and Bugzilla. They too will be migrated in the coming months. *Site infrastructure* One of the main concerns of the migration was serving traffic from the new data center using empty memcached servers: the spike in load on the Apache and database servers could have been disastrous to the site. To address it, Tim Starling improved on the single instance implementation of 'Parser Cache' persistent store in Tampa (to 3 sharded instances), and Asher Feldman built and replicated the databases across the 2 data centers.Another improvement, done by Asher and Peter Youngmeister, was the implementation of MHA <http://code.google.com/p/mysql-master-ha/> (Master High Availability) on our MySQL clusters. Its primary objective is to automate the promotion of a slave database in a master database fail-over scenario and to to reduce downtime, without suffering from replication integrity problems, without prolong database latency, and without changing existing deployments.Faidon Liambotis and Mark Bergsma continued to work on the Ceph file object store. With Domas Mituzas' help, they identified a performance issue with the RAID card which caused severe read/write latency on the Ceph cluster. Faidon has confirmed with the vendor that it is a known problem and no fix is available yet. We have ordered and substituted those RAID cards, and test results seem to indicate that the performance issue is solved. *Fundraising* Fundraising bastion hosts were deployed in the Ashburn and Tampa data centers. We also tweaked and tuned central logging and monitoring, and converted the remaining fundraising MuISAM tables to InnoDB, which should fix dump-induced replication lag. *Data Dumps <https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps>* This month, we had a look at the process of using the XML dumps to create a local copy of a Wikimedia site: it turned out to be painful and cumbersome at best, and unfathomable for the end-user in the worst case. As part of an attempt to improve this situation, there is now a new experimental tool<https://gerrit.wikimedia.org/r/gitweb?p=operations/dumps.git;a=tree;f=xmlfi…>available for *nix platforms, for generating MySQL tables from the XML stub and page content files. It is intended to read input files from various versions of MediaWiki and generate output for the version the user wants. Testing and feedback is encouraged. *Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>* In January, we had a number of performance and usability improvements. Three compute nodes were added into the pmtpa zone. Alex Monk added Echo notification support to labsconsole, passwordless sudo is now the default for projects, and shell requests are created automatically on account creation. The sysadmin and netadmin roles have been combined into a single projectadmin role. Glusterfs was upgraded to handle a memory leak, but unfortunately a new bug has been introduced that caused some instability in project storage. Work is ongoing to improve the project storage situation. Features Engineering <https://www.mediawiki.org/wiki/Wikimedia_Features_engineering> Editor retention: Editing tools *VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>* In January, the team worked primarily on reviewing and cleaning-up the code deployed<https://blog.wikimedia.org/2012/12/12/try-out-the-alpha-version-of-the-visu…>in December. They spent time with their colleagues in the Parsoid team planning the next phase of development, which is aimed at making the VisualEditor the default editor for all Wikipedias from July 2013. The alpha version of the VisualEditor on mediawiki.org and the English Wikipedia was updated twice (1.21-wmf7<https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf7#VisualEditor>and -wmf8 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf8#VisualEditor>), fixing a number of bugs reported by the community and making some adjustments to the link inspector's functionality based on feedback. *Parsoid <https://www.mediawiki.org/wiki/Parsoid>* In January, the Parsoid <https://www.mediawiki.org/wiki/Parsoid> team did some Spring cleaning and bug fixing. The serialization subsystem was overhauled: it now features simpler and more robust separator handling. Selective serialization was rewritten to deal with content deletions. It also features DOM diff-based change detection that does not rely on client-side change marking. Support for non-English wikis and local configurations was also improved a lot, and will likely stabilize in the next weeks. The team also discussed and documented the longer-term Parsoid / MediaWiki strategy in the Parsoid roadmap<https://www.mediawiki.org/wiki/Parsoid/Roadmap>. The performance-oriented C++ port was deprioritized in favor of DOM-based performance improvements and HTML storage. The basic idea behind storing (close to) fully processed HTML is to speed things up by doing no significant parsing on page view at all. In the longer term, VisualEditor-only wikis can avoid a dependency on Parsoid by switching to HTML storage exclusively. Overall, the plan is to leverage the Parsoid-generated HTML/RDFa DOM format inside MediaWiki core to enable better performance and editing capabilities in the future. Editor engagement features *Notifications <https://www.mediawiki.org/wiki/Echo_%28Notifications%29>* This month, we stepped up development on the Notifications project *Echo*and updated our first experimental release on mediawiki.org. Ryan Kaldari and Benny Situ improved the user experience for core features such as the badge, fly-out, all-notifications page and email notifications, and started developing new features such as bundling<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Bundling>, dismiss <https://www.mediawiki.org/wiki/Echo/Feature_requirements#Dismiss>and web preferences<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Web_Notifications>. Luke Welling completed work on HTML email and started development of a more robust job queue<https://www.mediawiki.org/wiki/Echo/Feature_requirements#JobQueue>. Fabrice Florin led discussions about the Echo product plan, and new features<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Features_under_con…>and notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Sample_Notificatio…>under consideration, while Vibha Bamba designed new components of the user experience. We plan to develop some of these features and notifications in coming weeks, and are aiming for a first release on the English Wikipedia by the end of March; in the meantime, you can try the current version<https://www.mediawiki.org/wiki/Echo/Testing>on mediawiki.org. We are also recruiting for a software engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=ovvXWfwD&c=qSa9VfwQ>to join our team and work with us on this and other editor engagement projects. *Flow <https://www.mediawiki.org/wiki/Flow>* Flow entered the product design phase in early January. OPW intern<https://www.mediawiki.org/wiki/Outreach_Program_for_Women>Kim Schoonover began user research <https://www.mediawiki.org/wiki/Flow/User_tests> regarding how user-to-user talk pages are handled, and collected data about the difficulties that new (and existing) users have when using them. Engineering discussions started about potential back-end and scaling difficulties, the possible use of Wikidata's ContentHandler<https://www.mediawiki.org/wiki/ContentHandler>, and the evaluation of Wikia's MessageWall. A plan for community engagement was proposed and accepted, with a consultation about the problems faced planned for early February, with experienced and newer users alike. *Article feedback <https://www.mediawiki.org/wiki/Article_feedback>* This month, our team updated Article Feedback v5<https://www.mediawiki.org/wiki/Article_feedback/Version_5>and discussed its release with communities in the English, French and German Wikipedias. Developer Matthias Mullie completed a major code refactoring, which is now being reviewed. He also developed a final set of new features<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requireme…>, such as simpler moderation tools and better filters, to be tested next month. Dario Taraborelli and Aaron Halfaker posted a feedback evaluation report<https://meta.wikimedia.org/wiki/Research:Article_feedback/Final_quality_ass…>, which suggests that about 39% of the feedback collected in their study can be used to improve articles (see also their other study results<https://commons.wikimedia.org/wiki/File:AFT5_2012-Q4_report.pdf>). Oliver Keyes responded to community questions in a request for comments<https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment/Article_feedba…>about future deployments on the English Wikipedia, with a final decision expected next month. Fabrice Florin led product planning and discussed a possible deployment on the French Wikipedia<https://en.wikipedia.org/wiki/fr:Wikip%C3%A9dia:Prise_de_d%C3%A9cision/Mise…>and with the German Wikipedia <https://en.wikipedia.org/wiki/de:Wikipedia:Artikel-Feedback>, currently evaluating the tool in an ongoing pilot<https://en.wikipedia.org/wiki/de:Spezial:Artikelr%C3%BCckmeldungen_v5>with a vote expected in May. Once our development is complete and communities reach their decisions for each project, we expect to release Article Feedback v5 on a range of Wikimedia sites in coming months. Editor engagement experiments *Editor engagement experiments<https://www.mediawiki.org/wiki/Editor_engagement_experiments> * In January, the Editor Engagement Experiments team ("E3") planned its goals for the quarter<https://www.mediawiki.org/wiki/Editor_engagement_experiments/Quarterly_Plan…>, which ends in March. We also made progress on the following projects which are included in that plan. First up, we launched<http://blog.wikimedia.org/2013/02/01/guided-tour-launch/> guided tours <https://www.mediawiki.org/wiki/Guided_tours> on the English Wikipedia, including a test tour to demonstrate the capabilities of the extension, and a tour associated with the "onboarding new Wikipedians<https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians>" (aka GettingStarted) project. In addition to tours created by the team, the extension <https://www.mediawiki.org/wiki/Extension:GuidedTour> supports community-created tours. Note that unlike many other projects by the E3 team, guided tours are planned as a permanent addition to Wikipedia, with each tour implementation considered to be experimental. (For example: the "getting started" tour will be delivered via a split A/B test.) While building guided tours, the team also A/B tested the Getting Started<https://www.mediawiki.org/wiki/Extension:GettingStarted>landing page and task list, measuring the effect it had on driving new contributions. Several rounds of analysis were completed and published on Meta (round 1<https://meta.wikimedia.org/wiki/Research:Onboarding_new_Wikipedians/OB1>, round 2<https://meta.wikimedia.org/wiki/Research:Onboarding_new_Wikipedians/OB2>), with the conclusion that the onboarding experience is leading to small but statistically significant increases in new English Wikipedians attempting to edit, as well as saving their first edit. In addition to measuring the effects of the guided tour associated with this project, immediate plans are to redesign the landing page and add additional task types, to entice more new contributors. Work also continued on refining the reliability and precision of the data collected from EventLogging<https://www.mediawiki.org/wiki/Extension:EventLogging>. In particular, we migrated EventLogging to a dedicated database, and began collecting server-side events in addition to client-side, to support work such as measuring account creations on desktop and mobile. January also saw the heavy use of the new User Metrics API<https://meta.wikimedia.org/wiki/Research:Metrics>, in order to complete cohort analysis of onboarding users and for metrics reported at the Board presentation<https://meta.wikimedia.org/wiki/Agenda_February_1-2_2013_Board_Meeting>on the Foundation's year-to-date progress. Development of the API continues, and a public announcement is expected for early March. Last but not least, a call was put out for a part-time Technical Writer<http://hire.jobvite.com/Jobvite/Job.aspx?j=oGH5Wfw8&c=qSa9VfwQ>to work on documenting both of these pieces of infrastructure. Support *2012 Wikimedia fundraiser<https://www.mediawiki.org/wiki/2012_Wikimedia_fundraiser> * January marks the official end of the 2012 fundraiser. The team spent the entirety of the month cleaning up and recovering from the very successful months of November and December, auditing the donations, and writing tools that will help the team run continuous auditing in the future. Mobile <https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering> Web *GeoData Storage & API<https://www.mediawiki.org/wiki/GeoData_Storage_%26_API> * After its soft launch in December, GeoData was officially announced<http://blog.wikimedia.org/2013/01/31/geodata-a-new-age-of-geotagging-on-wik…>this month. Work on improvements and bug fixing continues. The Special:Nearby page, which has been deployed to an experimental version of the site, represents the first major use of this feature on mobile projects. We hope to use it to help contributors identify articles in need of photos. *Mobile QA <https://www.mediawiki.org/wiki/Mobile_QA>* The push to get MobileFrontend<https://www.mediawiki.org/wiki/Extension:MobileFrontend>up and running on Beta Labs is well underway. We've also added test cases for Wikipedia Zero<https://www.mediawiki.org/wiki/Wikipedia_Zero/Test_cases>and we are planning a community test event for Mobile Upload and Commons in February. *Mobile Web Photo Upload<https://www.mediawiki.org/wiki/Mobile_design/Uploads> * This month, the mobile web team finished up work on the watchlist feature and kicked off a 3-month sprint on photo uploads. The focus in January was on developing basic uploading infrastructure: uploading images to Commons under a single Creative Commons license. We also built out the UX/UI design for a call to action on articles lacking images in the lead section. Through this workflow, users can upload an image to Commons and add a thumbnail of the image to the appropriate article on their local Wikipedia or sister project, in one simple step. We also developed a mobile uploads page where contributors can see their recent uploads and potentially donate more images from their mobile device to Commons. These features are currently live on the Beta mobile site and are set to be released to the full mobile site in February. Apps *Commons App <https://www.mediawiki.org/wiki/Apps/Commons>* January marked the first month of the Apps team's existence. Yuvaraj Pandian has started work with Brion Vibber on iOS and Android-based apps to upload photos to Commons. Both platforms are being developed concurrently and will have feature parity. Shankar Narayan joined us and and will be supporting the team for all design needs. While the first iteration of the Commons App isn't scheduled to finish until February 8th, the team has already created two skeleton apps that can upload, share and show the user's contributions. The team will be spending their next iteration tweaking workflows and styling the app. We also released new versions of the Wikipedia app on iOS and Android in order to bring it into compliance for legal privacy/disclaimer issues. Partners *Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>* During January, Wikimedia was awarded a grant<https://blog.wikimedia.org/2013/01/17/wikimedia-foundation-winner-of-knight…>in the Knight News Challenge for our work in expanding Wikimedia mobile projects. Part of this grant will be used for Wikipedia Zero and the SMS/USSD <https://www.mediawiki.org/wiki/Wikipedia_over_SMS_%26_USSD>projects to improve access to knowledge in the developing world. In addition, we've partnered with VimpelCom<https://blog.wikimedia.org/2013/01/24/vimpelcom-partnership-grows-wikipedia…>to provide Wikipedia Zero to at least 100 million additional customers this year. *J2ME App <https://www.mediawiki.org/wiki/Wikipedia_S40_J2ME_app>* During January, we've begun to explore ways to reduce the memory and processor requirements of our J2ME app, to increase the number of phones that can use this application. *Wikipedia over SMS & USSD<https://www.mediawiki.org/wiki/Wikipedia_over_SMS_%26_USSD> * We are finishing work on capturing the metrics from the SMS server to learn usage numbers and determine how many sessions are completed. Platform Engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering> MediaWiki Core *MediaWiki 1.21 <https://www.mediawiki.org/wiki/MediaWiki_1.21/Roadmap>* MediaWiki 1.21wmf7 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf7> and 1.21wmf8 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf8> were deployed in January on a modified schedule, due to holidays and because of the data center migration. Deployments have returned to their usual fortnightly schedule. *Git conversion <https://www.mediawiki.org/wiki/Git/Conversion>* The ExtensionDistributor<https://www.mediawiki.org/wiki/Extension:ExtensionDistributor>was rewritten in early January. While this was primarily done to support the data center migration<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning>, this was the first time ExtensionDistributor had received any signification attention since the migration to Git. The new version now utilizes the Github API to generate extension snapshots. We hope that the new version will be more reliable for users. SVN-based extensions are no longer supported, but this is not expected to impact many users since these extensions are largely unmaintained (all popular and active extensions have long since moved to Gerrit). As always, these extensions will remain in SVN should anyone still want the code. *TimedMediaHandler <https://www.mediawiki.org/wiki/TimedMediaHandler>* Jan Gerber continues bugfixing and refining TimedMediaHandler, mainly focusing on operational improvements to make more efficient use of our server infrastructure. *Wikidata deployment <https://www.mediawiki.org/wiki/Wikidata_deployment>* Sam Reed helped the Wikidata deployment, deploying the Wikibase Client extension to Wikipedia in Hungarian, Hebrew, and Italian. Chris Steipp reviewed the Wikidata team's work to extend AbuseFilter for use with structured data. Aaron Schulz worked with Daniel Kinzler on job queue improvements. *Wikivoyage migration <https://www.mediawiki.org/wiki/Wikivoyage_migration>* Wikivoyage officially launched<https://blog.wikimedia.org/2013/01/15/wikimedia-foundation-launches-wikivoy…>on January 15. Most of the Wikimedia Foundation's involvement was completed in November, but some minor bugfixing was done in support of the official launch. *SwiftMedia <https://www.mediawiki.org/wiki/SwiftMedia>* NFS for uploads/thumbnails has been unmounted from all Apache servers and the NFS back-end configuration was removed from MediaWiki; all files now only use Swift. A workaround has been added for the Swift back-end class when used with Ceph, so that temporary URLs can be used (for making video thumbnails for example). A Python script to copy files into Ceph has been run and is being worked on. Various issues have been reported in Ceph's bug tracker and are being looked at by the developers. *Lua scripting <https://www.mediawiki.org/wiki/Lua_scripting>* Lua development was put on hold through the Ashburn data center migration. We've now resumed work on Lua, with Brad Jorsch and Tim Starling making more functions available in Lua that are currently already available in template parser functions. *Site performance <https://www.mediawiki.org/wiki/Site_performance>* A patch to allow moving the DB job queue to another cluster is under review. An experimental redis-based job queue patch also exists in gerrit. *Incremental architectural improvements<https://www.mediawiki.org/wiki/Incremental_architectural_improvements> * Code was merged to support more complex data structures (lists, sets) in memcached (with atomic updates). *Admin tools development<https://www.mediawiki.org/wiki/Admin_tools_development> * The team mainly focused this month on improving the AbuseFilter extension, which is now working on the Wikidata<https://www.mediawiki.org/wiki/Wikidata>site after support was added for other content types (as defined using ContentHandler <https://www.mediawiki.org/wiki/ContentHandler>). There was some significant work done on blocking abusive proxies and abuse limits, and some additional progress made on global AbuseFilters, user renaming and the interface for Stewards to mass-lock user accounts<https://www.mediawiki.org/wiki/Admin_tools_development/CentralAuth_Locking> . *Security auditing and response<https://www.mediawiki.org/wiki/Security_auditing_and_response> * The team continued to respond to reported vulnerabilities, began a security review of fundraising extensions, and continued reviews of Wikidata features. Quality assurance *Quality Assurance <https://www.mediawiki.org/wiki/QA>* We started to schedule opportunities for community testing events<https://www.mediawiki.org/wiki/QA/Weekly_goals>. Echo, AFTv5, and VisualEditor are all current candidates for testing. A week-long focus on VisualEditor's support for non-Latin characters<https://www.mediawiki.org/wiki/VisualEditor/Testing_Non-Latin_Characters_In…>uncovered at least one major issue causing data loss, and another one with tool-assisted Chinese input. *Beta cluster <https://www.mediawiki.org/wiki/Beta_cluster>* The main use for the Beta Cluster in January was to test git-deploy<http://wikitech.wikimedia.org/view/git-deploy>. Zeljko Filipin continues to run regular tests there. Antoine Musso, Max Semenik, and Andrew Bogott are setting up MobileFrontend to run on Beta for testing purposes. *Continuous integration<https://www.mediawiki.org/wiki/Continuous_integration> * Antoine Musso worked with several MediaWiki extension authors to ensure that the unit tests for those extensions are run by Jenkins and that they work. He hopes to have all extensions that run on the Wikimedia production cluster fully operational by the end of February. Antoine also integrated PHP CodeSniffer<https://www.mediawiki.org/wiki/Continuous_integration/PHP_CodeSniffer>into our automated test runs. *Browser testing <https://www.mediawiki.org/wiki/QA/Browser_testing>* Architecture and configuration for browser testing are now stable, and the focus shifted to increasing test coverage by making existing tests more extensive and covering new features. An example is the Math extension<https://www.mediawiki.org/wiki/Extension:Math>, which was briefly broken after the data center migration. The team has also instituted a weekly pair-programming session every Friday. Analytics *Limn <https://www.mediawiki.org/wiki/Analytics/Limn>* The team made performance improvements and added new visualizations. Bar, line, and geo plots can now be built ad hoc from arbitrary data. Evan Rosen's Grantmaking and Programs dashboard was migrated to this new version of Limn. Current work includes a collaboration with the E3 team to provide visualizations, and development of a MediaWiki extension that will allow creation and editing of graphs. Engineering community team *Bug management <https://www.mediawiki.org/wiki/Bug_management>* This month, a first bugday was held<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065792.html>, targeting bug reports which had not seen any changes for more than one year, resulting in about 30 tickets being updated. In addition, some cleanup work (decreasing the number of unprioritized bug reports and going through open reports in "ASSIGNED" status for more than a year) took place. Andre Klapper worked on small Bugzilla code changes<https://bugzilla.wikimedia.org/buglist.cgi?bug_id=42467,24992>and published initial information on Bugzilla usage per development team<https://www.mediawiki.org/wiki/Bug_management/Development_teams_usage>. Community members were invited to join the MediaWiki Group Bug Squad<https://www.mediawiki.org/wiki/Groups/Proposals/Bug_Squad>. Furthermore, some problems due to data center migration<https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-…>were investigated, and it was discussed how to improve interaction on Bugzilla tickets that need handling by the Operations team (who mostly prefers to use the RT <https://rt.wikimedia.org/> bugtracker instead). *Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>* Six Outreach Program for Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women>interns started on January 3rd and will work full time until April. Mariya <https://www.mediawiki.org/wiki/User:Mitevam> is working on a discussion among third-party MediaWiki users<https://www.mediawiki.org/wiki/Third-party_MediaWiki_users_discussion>. Valerie <https://www.mediawiki.org/wiki/User:Valeriej> has completed the Bug Squad group proposal<https://www.mediawiki.org/wiki/Groups/Proposals/Bug_Squad>and a first Bug Day. Priyanka <https://www.mediawiki.org/wiki/User:Priyanka_Nag> created a script<https://www.mediawiki.org/wiki/User:Priyanka_Nag/common.js>and plans to move to Git <https://www.mediawiki.org/wiki/Git>. Sucheta<https://www.mediawiki.org/wiki/User:Sucheta_Ghoshal>is on schedule following her project plan<https://www.mediawiki.org/wiki/User:Sucheta_Ghoshal/OPW-EtherEditor#Plans_a…>. Kim <https://www.mediawiki.org/wiki/User:Isarra> is learning about Flow<https://www.mediawiki.org/wiki/Flow>and the basics of interactive design as indicated by her mentor. Teresa <https://www.mediawiki.org/wiki/User:Chot> has completed a solid base<https://gerrit.wikimedia.org/r/#/q/owner:tcho708%2540gmail.com,n,z>for her extension and is working on the main functionality. She hit a snag with her work environment this week, but is still on track with her proposed timeline. The Google Summer of Code 2013<https://www.mediawiki.org/wiki/Summer_of_Code_2013>page was created, a pre-planning discussion<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065771.html>started on wikitech-l, and LevelUp <https://www.mediawiki.org/wiki/Mentorship_programs/LevelUp>matchmaking for the first quarter of 2013 is nearly done. *Technical communications<https://www.mediawiki.org/wiki/Technical_communications> * Guillaume Paumier <https://www.mediawiki.org/wiki/User:Guillom> provided communications support<https://www.mediawiki.org/wiki/Technical_communications/Tech_blog_activity>to the engineering team, notably around the data center migration<https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-…>and associated banners<https://meta.wikimedia.org/wiki/CentralNotice/Generic_maintenance_notice>, notices<https://meta.wikimedia.org/w/index.php?title=Global_message_delivery/Spam&o…>& translations<http://lists.wikimedia.org/pipermail/translators-l/2013-January/002199.html>. He started to organize and clean up the MediaWiki version pages (like MediaWiki 1.21/wmf7 <https://www.mediawiki.org/wiki/MediaWiki_1.21/wmf7>) to make them more useful for tech ambassadors<https://meta.wikimedia.org/wiki/Tech/Ambassadors>, by highlighting the most important changes, improving translatability and adding navigation. He also prepared and organized translations for the *How to report a bug <https://www.mediawiki.org/wiki/How_to_report_a_bug>* and *How to contribute <https://www.mediawiki.org/wiki/How_to_contribute>* pages, to facilitate the involvement of volunteers who don't necessarily communicate in English. Last, he created a Project:Calendar<https://www.mediawiki.org/wiki/Project:Calendar>to consolidate and centralize announcements for all events <https://www.mediawiki.org/wiki/Template:Event>, to make opportunities for participation more visible. Events around a particular topic (like QA, testing and bugs<https://www.mediawiki.org/wiki/QA/Weekly_goals>) can still be selectively transcluded, using Labeled Section Transclusion<https://www.mediawiki.org/wiki/Extension:Labeled_Section_Transclusion> . *Volunteer coordination and outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach> * The MediaWiki groups <https://www.mediawiki.org/wiki/Groups> for Promotion<https://www.mediawiki.org/wiki/Groups/Promotion>and San Francisco <https://www.mediawiki.org/wiki/Groups/San_Francisco> were officially approved by the Wikimedia Affiliations Committee<https://meta.wikimedia.org/wiki/Affiliations_Committee>, and are the first Wikimedia User Groups<https://meta.wikimedia.org/wiki/Wikimedia_User_Groups#Approved_user_groups>created. We helped the Editor Engagement <https://www.mediawiki.org/wiki/Editor_Engagement> team organize a sprint to test Echo <https://www.mediawiki.org/wiki/Echo/Testing>, but our plans to collaborate further with the Editor Engagement and Mobile<https://www.mediawiki.org/wiki/Mobile>teams were delayed; Quim Gil proposed a different approach<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065758.html>combining regular, time-based QA <https://www.mediawiki.org/wiki/QA> and bug management<https://www.mediawiki.org/wiki/Bug_management>activities, in the form of QA weekly goals <https://www.mediawiki.org/wiki/QA/Weekly_goals>. Two such events (non-Latin character testing in VisualEditor<https://www.mediawiki.org/wiki/VisualEditor/Testing_Non-Latin_Characters_In…>and a review of old bugs<https://www.mediawiki.org/wiki/Bug_management/Triage#Next_meeting>) happened in January, and more are scheduled. Heavy work was done with Chris McMahon to improve the top QA pages<https://www.mediawiki.org/wiki/Category:QA>, although some problems remain<http://lists.wikimedia.org/pipermail/wikitech-l/2013-January/065696.html>. Template:MediaWiki News<https://www.mediawiki.org/wiki/Template:MediaWiki_News>is now manually synced with social media <https://www.mediawiki.org/wiki/Social_media>, bringing fresh updates to the mediawiki.org homepage <https://www.mediawiki.org/wiki/MediaWiki>and News <https://www.mediawiki.org/wiki/News> page. Quim also took the lead on organizing the Wikipedia Engineering Meetup<http://www.meetup.com/Wikipedia-Engineering-Meetup/events/89239012/>on January 17th. He prepared an intro to MediaWiki & Wikimedia tech contributions<https://www.mediawiki.org/wiki/File:How_to_hack_on_Wikipedia.pdf>, which he tested at FOSDEM <https://www.mediawiki.org/wiki/Events/FOSDEM>, designed to be reused by other presenters. Last, we confirmed that technical projects are eligible to Individual Engagement Grants<https://meta.wikimedia.org/wiki/Grants:IEG> . Language engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineering> *Language tools <https://www.mediawiki.org/wiki/Language_tools>* Development of the new user interface for Translate, as well as the translation editor functionality, continued throughout the month of January. Focus was on back-end work and extending the WebAPI to support the remaining features which are needed to reach feature parity with current editor. The MediaWiki Language Extension Bundle 2013.01 was released. Universal Language Selector was deployed with limited features to a selection of Wikimedia sites projects using the Translate extension. Collaboration projects also continue with Red Hat's language technologies teams, with an upcoming work sprint to complete several projects extending internationalization support for Indic languages. Runa Bhattacharjee kicked off the Language coverage matrix, an attempt to compile a snapshot of our internationalization tools coverage per language for 300 languages. *Milkshake <https://www.mediawiki.org/wiki/Milkshake>* More input methods were added to jQuery.IME, and bugs were fixed in jQuery.ULS. Kiwix <http://www.kiwix.org> *The Kiwix project is funded and executed by Wikimedia CH<https://meta.wikimedia.org/wiki/Wikimedia_CH> .* We have adapted the kiwix-plug <http://www.kiwix.org/index.php/Kiwix-plug>script to Tonidoplug2 <http://www.tonidoplug.com/>, a device cheaper than the Dreamplug<https://www.globalscaletechnologies.com/p-54-dreamplug-devkit.aspx>. Kiwix was elected by Sourceforge users as February's Project of the Month and an interview of Emmanuel Engelhart<http://sourceforge.net/blog/potm-201302/>was published. For the first time, Kiwix has reached 100.000 downloads a month in January. Beside Kiwix, the openZIM website <http://www.openzim.org>was revamped and simplified for better readability. The openZIM bug tracker and source code management were migrated to the Wikimedia infrastructure (Bugzilla and Git). Wikidata <https://meta.wikimedia.org/wiki/Wikidata> *The Wikidata project is funded and executed by Wikimedia Deutschland<https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en> .* January has been an exciting month for Wikidata. The deployment on the first Wikipedia sites (Hungarian<https://blog.wikimedia.de/2013/01/14/first-steps-of-wikidata-in-the-hungari…>, Hebrew and Italian<https://blog.wikimedia.de/2013/01/30/wikidata-coming-to-the-next-two-wikipe…>) was completed. At the same time, work has continued on the user interface and back-end for statements, the core part of Wikidata's second phase. This will enable users to enter information like the children of a given person or a link to their portrait on Wikimedia Commons. These features can already be tested on the demo system <http://wikidata-test.wikimedia.de>. We've also worked on making AbuseFilter work with Wikidata, and wrote a new mechanism to distribute changes to the clients (Wikipedia) so they can show Wikidata changes in their RecentChanges. We made progress on using Solr for search and rewrote the draft for the inclusion syntax<https://meta.wikimedia.org/wiki/Wikidata/Notes/Inclusion_syntax_v0.3>to be much simpler. This is the syntax that editors will use to include data from Wikidata in Wikipedia. A manual for using Pywikipedia on Wikidata<https://www.mediawiki.org/wiki/Manual:Pywikipediabot/Wikidata>was written as well. If you want to code on Wikibase, the software powering Wikidata, have a look at the outstanding bugs and tasks<https://bugzilla.wikimedia.org/buglist.cgi?keywords=need-volunteer%2C%20&ke…> . Future The engineering management team continues to update the * Deployments <http://wikitech.wikimedia.org/view/Deployments>* page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the *engineering roadmap<https://www.mediawiki.org/wiki/Roadmap> *, listing ongoing and future Wikimedia engineering efforts. -- Guillaume Paumier Technical Communications Manager — Wikimedia Foundation https://donate.wikimedia.org

11 years, 2 months

Documentation for Extensions

by Matthew Walker

All, Do we have an existing method for extensions to autogenerate documentation (via a commit hook or something) and then to upload that to a documentation server? (Something akin to what we do with MediaWiki core.) We should have this for Doxygen, and make something similar for JSDuck -- the JS documentation system being pioneered by the VisualEditor team. -- ~Matt Walker

11 years, 2 months

MediaWiki API at Codecademy?

by Quim Gil

Long story short: the MediaWiki API could be included at http://www.codecademy.com/tracks/apis if someone wants to do the work. Codecademy is happy to have us there. I think it is a good idea, in need of someone willing to drive this: - It is a good excuse to improve our API documentation at mediawiki.org. - Codecademy is a good place to reach to more developers. - Maybe it is a good chance for someone to take this as a paid job? The Wikimedia movement wants to spread the 1,3T of content we have, and get more free content from as many channels as possible. Our API plays a big role on this. Therefore, I *personally* believe that a project to update the API documentation at mediawiki.org and have corresponding exercises at Codecademy has a chance to receive a grant if the proposal and the candidate(s) are solid. Interested? Let me help you. -- Quim Gil Technical Contributor Coordinator @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

11 years, 2 months

Technical design review of extension

by Paul Selitskas

Hello. I wrote an extension[1] for Wikinews and Wiktionary to replace the JS-driven custom tabs (Opinions in Wikinews, Citations/Template documentation in Wiktionary). According to the manual, as there were no "pros" heard from Design-l, now the extension must undergo a technical design review, before it is decided whether the extension is able to fly on production sites - deployment review. -- [1] https://www.mediawiki.org/wiki/Extension:NamespaceRelations -- З павагай, Павел Селіцкас/Pavel Selitskas Wizardist @ Wikimedia projects

11 years, 2 months

NullLockManager and the math extension

by Matthew Flaschen

On my local Vagrant MediaWiki install (https://github.com/wikimedia/wmf-vagrant), I'm getting: Running test BUG 1887, part 2: A <math> with a thumbnail- math enabled... No lock manager defined with the name `nullLockManager`. when running the parser tests: php tests/parserTests.php --file=extensions/Math/mathParserTests.txt against master of the Math extension (MediaWiki is at git latest). Can anyone shed some light on this? Is it expected this would work in production, meaning the test is broken? Matt Flaschen

11 years, 2 months

Jenkins tests reporting more errors

by Antoine Musso

Hello, Due to some mistake I made during the Jenkins job overhaul last year, the PHPUnit jobs were no more using $wgDevelopmentWarnings. That is now fixed. I have also enabled $wgShowExceptionDetails per bug 43059. If that cause any serious trouble, shell users could change the setting file on gallium at: /var/lib/jenkins/jobs/_shared/ExtraSettings.php -- Antoine "hashar" Musso

11 years, 2 months

Resign

by Huib Laurens

Dear List(s), Currently I'm inactive for Wikimedia after a lot of things changed. The latest change is that I'm now working for a big datacentre and I do not feel that I have enough time for other things. I already got a few mails that I was collecting hats, so hereby I stop as: LangCom member Administrator for Mediawiki-l Administrator for Wikitech I thank you all. -- Met vriendelijke groet, Huib Laurens

11 years, 2 months

1.21wmf9 deployment blocker: bug 44748 (editing broken ku.wiktionary and sr.wikinews)

by Rob Lanphier

Hi folks, We had to revert 1.21wmf9 on a few wikis due to bug 44748. Bug report: https://bugzilla.wikimedia.org/44748 The problem is that, when 1.21wmf9 is enabled, editing is completely broken on ku.wiktionary and sr.wikinews, among others. Below is the stack trace. Any ideas? Rob 2013-02-07 15:30:22 mw1061 kuwiktionary: [3f063856] /w/index.php?title=d%C3%AAw&action=edit Exception from line 235 of /usr/local/apache/common-local/php-1.21wmf9/languages/classes/LanguageKu.php: Broken variant table: #0 /usr/local/apache/common-local/php-1.21wmf9/languages/LanguageConverter.php(558): KuConverter->translate('d??w', 'ku') #1 /usr/local/apache/common-local/php-1.21wmf9/languages/Language.php(3660): LanguageConverter->convertTitle(Object(Title)) #2 /usr/local/apache/common-local/php-1.21wmf9/includes/parser/Parser.php(439): Language->convertTitle(Object(Title)) #3 /usr/local/apache/common-local/php-1.21wmf9/includes/cache/MessageCache.php(860): Parser->parse('Anti-spam check...', Object(Title), Object(ParserOptions), true) #4 /usr/local/apache/common-local/php-1.21wmf9/includes/Message.php(652): MessageCache->parse('Anti-spam check...', NULL, true, true, Object(LanguageKu)) #5 /usr/local/apache/common-local/php-1.21wmf9/includes/Message.php(455): Message->parseText('Anti-spam check...') #6 /usr/local/apache/common-local/php-1.21wmf9/includes/Message.php(510): Message->toString() #7 /usr/local/apache/common-local/php-1.21wmf9/extensions/SimpleAntiSpam/SimpleAntiSpam.php(40): Message->parse() #8 [internal function]: efSimpleAntiSpamField(Object(EditPage), Object(OutputPage)) #9 /usr/local/apache/common-local/php-1.21wmf9/includes/Hooks.php(256): call_user_func_array('efSimpleAntiSpa...', Array) #10 /usr/local/apache/common-local/php-1.21wmf9/includes/GlobalFunctions.php(3875): Hooks::run('EditPage::showE...', Array) #11 /usr/local/apache/common-local/php-1.21wmf9/includes/EditPage.php(2091): wfRunHooks('EditPage::showE...', Array) #12 /usr/local/apache/common-local/php-1.21wmf9/includes/EditPage.php(421): EditPage->showEditForm() #13 /usr/local/apache/common-local/php-1.21wmf9/includes/actions/EditAction.php(51): EditPage->edit() #14 /usr/local/apache/common-local/php-1.21wmf9/includes/Wiki.php(439): EditAction->show() #15 /usr/local/apache/common-local/php-1.21wmf9/includes/Wiki.php(305): MediaWiki->performAction(Object(Article), Object(Title)) #16 /usr/local/apache/common-local/php-1.21wmf9/includes/Wiki.php(565): MediaWiki->performRequest() #17 /usr/local/apache/common-local/php-1.21wmf9/includes/Wiki.php(458): MediaWiki->main() #18 /usr/local/apache/common-local/php-1.21wmf9/index.php(59): MediaWiki->run() #19 /usr/local/apache/common-local/live-1.5/index.php(3): require('/usr/local/apac...') #20 {main}

11 years, 2 months

Documentation talk, the second

by Petr Bena

Hi, this is a second time I am opening this - last time we decided to merge wikitech and labsconsole to make a documentation base for developers and operation engineers (at least those working at labs - I consider operation of bots and such services operation as well). Since then, not much has happened. Me and Thehelpfulone were working on some preparation for merge, by indexing some uncategorized pages on wikitech, and flagging what can be safely removed (by leaving it on wikitech and not merging at all). Can we move forward in this? What needs to be done to merge these 2 and finally have a central tech documentation for whole wikimedia project

11 years, 2 months

RFC: Standardized thumbnails sizes

by Antoine Musso

Hello, Wikitext editors can have media materials rendered using any arbitrary size (the thumbnails wikitext syntax). Each different size would produce a hit to the server backend which will generate a thumbnail for that size. The following requests for comment is about having a consistent set of thumbnails sizes that are allowed to be rendered and let the client browser to do the up or down scaling. https://www.mediawiki.org/wiki/Requests_for_comment/Standardized_thumbnails… It would be nice to discuss about it on the list and amend the document with any idea you might have :-] -- Antoine "hashar" Musso

11 years, 2 months

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l February 2013