Wikitech-l April 2015

wikitech-l@lists.wikimedia.org

98 participants
78 discussions

Welcome Chromium to our QUnit testing
by Krinkle 08 Apr '15

08 Apr '15

TL;DR: * Our QUnit jobs now uses latest Chromium instead of PhantomJS. * You can run the test suite from the command line locally now. Thanks to Tim Starling, Antoine "hashar" Musso, Kunal (legoktm), Bryan Davis, S Page, and James Forrester; for their help in different areas. https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing Are you sitting comfortably? The past few months can be summarised as a long journey through a forest of technical debt. It was also amazing to see just how many layers of infrastructure were able to block this task (and did). [1] From a local development point of view it couldn't be simpler. In the Gruntfile: Remove grunt-contrib-qunit, add grunt-karma with karma-chrome-launcher. Done? Not quite. For standalone front-end projects it actually was this simple. OOjs and VisualEditor have been enjoying this new stack since July 2014. This helped refine the stack and the underlying technologies over the past months. More about what this new stack provides in a minute. == Journey == === Export from Special:JavaScriptTest === For MediaWiki core and extensions, one must install MediaWiki before running QUnit. [2][3] That in itself isn't too complicated (set up DB and run install; we have standard macros for that). But, the test doesn't just communicate with MediaWiki. The test suite is actually served *by* MediaWiki. Karma enforces the principle that unit tests run in pure javascript (blank page with sockets, load source files, load QUnit, run test suite). Back in MediaWiki 1.19 our test suite abided by these best practices. The test suite was a static HTML file that loaded relevant scripts directly. [4] This file was later migrated to Special:JavaScriptTest, which opened the door for undeclared dependencies. 1. Source code is registered to ResourceLoader. 2. Extensions register tests via hooks in PHP. 3. OutputPage and Skin provide config vars and HTML that tests could depend on. Point 3 was easily resolved by adding the appropriate mw.config or DOM fixture to the few tests that were missing it. Though Point 2 and 1 weren't going anywhere. We needed to, once again, access our tests suite as a pure JavaScript payload, with no HTML or script tags loading relevant resources. Hence, the introduction of Special:JavaScriptTest/qunit/export in https://gerrit.wikimedia.org/r/178551. === Migrate all the things === Objective: Karma [5] on Chromium. Easy to run locally for developers. To be re-usable, the logic must be generic and use composer/npm. In 2013, most jobs ran on Wikimedia production servers. In order to use npm we had to migrate to Wikimedia Labs first. Back then, production had only just begun using Ubuntu Trusty. The relevant modules require Node.js v0.10. Ubuntu Precise has Node.js v0.8. Precise is also stuck with Chromium 37 (EOL). [7] To get auto-updating latest stable Chromium and Node v0.10, we needed to migrate our infrastructure to Ubuntu Trusty first. Much of the process to install MediaWiki on a Jenkins slave also wasn't puppetised. === SQLite === This is its own story. See https://phabricator.wikimedia.org/T89180 for details. The short of it is, we also migrated everything to MySQL. Which we wanted to do anyway. == New stack == 1. Chromium PhantomJS is a great application for many purposes. It was good for us while it lasted. But for JavaScript unit testing, PhantomJS just isn't meant to be. It's too distant from a "real" browser. When we introduced PhantomJS, it was a big step forward toward testing cross-browser. It "uses WebKit" which meant we were kind of covering Chrome and Safari (in 2012). Safari has had several major releases since PhantomJS v1.9 came out. Chrome had even more releases (and since dropped WebKit in favour of Blink). And, in truth, PhantomJS wasn't that much like Safari and mainstream WebKit to begin with. [8] Chromium is an actual browser. A browser we actually support. A browser that represents real users of our software. It's only one browser for now, but it's a start. 2. Karma With Karma as a solid foundation, it's truly simple to add more browsers in the future. OOjs has already added Firefox alongside Chromium in Jenkins. Karma uses Web Driver, a standardised protocol to operate browsers (locally and remotely alike). It has an official plugin hooking it up to SauceLabs, which opens the door to any other browser and platform we want. It's as simple as adding 1 line to the config. [5][6] We haven't done this yet as there are scalability, performance and security considerations. Having said that, OOjs is currently using an experimental pipeline to test in six different browsers from Jenkins via SauceLabs. Including IE 6 and IE 11 on Windows, and Safari 5 on Mac! [9] 3. Code coverage One of the advantages of having a pure stack is the ability to run pre-processors on the code using Karma built-ins. [10] The first one I'm looking at is code coverage. 4. Local development and Jenkins equally It runs the full QUnit test suite in a real browser (multiple even) by simply running "npm test". [11] Aside from Node.js, it requires no pre-installed software and is as easy as "npm install" to set up. — Krinkle [1] https://phabricator.wikimedia.org/T74063 [epic] Adopt Karma with Chromium (tracking). [2] https://phabricator.wikimedia.org/T89433 Make QUnit tests run without installing MediaWiki. [3] https://phabricator.wikimedia.org/T89432 Make PHPUnit tests run without installing MediaWiki. [4] https://github.com/wikimedia/mediawiki/blob/REL1_19/tests/qunit/index.html [5] https://karma-runner.github.io/ [6] https://karma-runner.github.io/0.12/config/browsers.html [7] http://packages.ubuntu.com/precise/chromium-browser [8] http://codepen.io/Krinkle/blog/phantomjs-anno-2014 [9] https://github.com/wikimedia/oojs/blob/v1.1.6/Gruntfile.js#L76-L107 https://integration.wikimedia.org/ci/job/npm/1680/console [11] https://karma-runner.github.io/0.12/config/preprocessors.html https://github.com/karma-runner/karma-coverage [12] https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing

2 1

Requiring PHP >= 5.3.3 for MediaWiki core
by Legoktm 08 Apr '15

08 Apr '15

Hi, As part of the librarization project[1], we are planning on taking the CSSJanus library that is currently in includes/lib/ and bringing it in with composer. However, it requires PHP >=5.3.3 in its composer.json[2]. Krinkle has stated[3] that is due to the fact that it has only been tested on 5.3.3 and higher, and it's also what travis-ci provides. After doing some research[4], it appears that we would be dropping support for Ubuntu 10.04LTS, which has security support until April 2015. MediaWiki 1.25.0 is expected to be released in May 2015. Does anyone have any objections to dropping 5.3.2 support? I've uploaded [5] that actually increments the required version number. Thanks, -- Legoktm [1] https://www.mediawiki.org/wiki/Library_infrastructure_for_MediaWiki [2] https://github.com/cssjanus/php-cssjanus/blob/master/composer.json [3] https://github.com/cssjanus/php-cssjanus/pull/5#issuecomment-60069126 [4] https://phabricator.wikimedia.org/T839#14160 [5] https://gerrit.wikimedia.org/r/168535

11 14

Welcome Stephane Bisson to the Collaboration Team
by Tomasz Finc 07 Apr '15

07 Apr '15

I am pleased to announce that Stephane Bisson joins WMF this week as a Software Engineer on the Collaboration Team! Stephane is an avid learner and traveler. He is passionate about history, cultures, and languages. He runs, cooks, and enjoys wine tasting. And is eager to travel to Napa with anyone who knows it well. Professionally, Stephane spent 5 years writing software for the manufacturing industry and another 5 years as a consultant for ThoughtWorks. Ruby and Javascript have a very special place in his heart. Stephane will join the Collaboration team focusing on front-end development and will be in SF with the team this week. Please welcome Stephane! --tomasz

2 1

Re: [Wikitech-l] [Engineering] [wikitech-l] Welcome Chromium to our QUnit testing
by Greg Grossmeier 07 Apr '15

07 Apr '15

<quote name="Timo Tijhof" date="2015-04-04" time="04:40:33 +0100"> > TL;DR: > * Our QUnit jobs now uses latest Chromium instead of PhantomJS. > * You can run the test suite from the command line locally now. > > Thanks to Tim Starling, Antoine "hashar" Musso, Kunal (legoktm), > Bryan Davis, S Page, and James Forrester; for their help in different areas. > > https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing Thanks a ton for this great work, Timo and company. I really want to call out Timo and others here. Our continuous integration setup is a good one, and is getting better. A big part of why it continues to improve is because of the cross-organizational support we (#releng) receive from people like Timo, Bryan, Kunal, James, etc. It's great seeing the collaboration happen on these projects. </kumbaya_moment> Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

1 0

Moving MWLogger classes to a namespace in MediaWiki core
by Bryan Davis 07 Apr '15

07 Apr '15

The new PSR-3 debug logging system brought namespaced external code (Psr\Log\LoggerInterface) into use in MediaWiki core. The classes I built out to work with this system are using faux namespaces by virtue of class names like "MWLoggerFactory", "MWLoggerLegacyLogger" and "MWLoggerMonologSyslogHandler". Before 1.25 starts rolling out as a tarball release I'd like to change these classes to use actual PHP namespaces rather than this clunky collision avoidance mechanism. [0] There is also a task to backport minimal PSR-3 support to the 1.23 LTS system to simplify backports of code and extensions that adopt direct use of PSR-3 and I'd like to only do that once if possible. The color I have picked for this namespace bikeshed is MediaWiki\Core\Logger. The MediaWiki root namespace is a pretty obvious choice. "Core" is inserted to distinguish this fundamental MediaWiki functionality from any existing or future extensions that might use namespaces. I'm hoping "Logger" is sufficiently distinct from other uses of the term "log" in MediaWiki which generally mean "audit trail" rather than "debugging information". I'd be fine with throwing Debug in between Core and Logger too if consensus found for that instead. I'd also like to start organizing these files in a directory structure that would be compatible with the PSR-4 auto loader standard. PSR-1 required all namespace elements to be in the file path as directories, but PSR-4 allows a common prefix for all classes to be dropped. I was thinking an includes/Core/ directory could be used as the common base for these and future namespaced classes in MediaWiki core. We had some discussion last summer [2] about namespace use in extensions that seemed to end with "cool do it when you want" and "we don't really need any standard conventions". Since I'm suggesting namespace usage in core I figured this was worth another (hopefully short) round of discussion with the larger community than is likely to see my patches when they land in Gerrit. [0]: https://phabricator.wikimedia.org/T93406 [1]: https://phabricator.wikimedia.org/T91653 [2]: http://www.gossamer-threads.com/lists/wiki/wikitech/476296 Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855

3 4

Roadmap and deployment update - week of April 6th
by Greg Grossmeier 04 Apr '15

04 Apr '15

Hello and welcome to the latest edition of the WMF Engineering Roadmap and Deployment update. The full log of planned deployments next week can be found at: <https://wikitech.wikimedia.org/wiki/Deployments#Week_of_April_6th> For a longer term view, see the new Roadmap project in Phabricator: <https://phabricator.wikimedia.org/tag/roadmap/> A quick list of notable items for next week... == All Week == * The convertion of LQT pages on MediaWiki.org to Flow will begin on Monday and progress throughout the week. * Deploying the SandboxLink extension to cluster and enable on wikis where it currently is a gadget will likely happen next week; date TBD. == Monday == * ContentTranslation will be enabled on Vietnamese Wikipedia ** <https://phabricator.wikimedia.org/T91164> == Tuesday == * MediaWiki deploy ** group1 to 1.25wmf24: All non-Wikipedia sites (Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity, and a few other sites) ** <https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf24> == Wednesday == * MediaWiki deploy ** group2 to 1.25wmf24 (all Wikipedias) ** group0 to 1.26wmf1 (test/test2/testwikidata/mediawiki) *** Just calling it out, 1.26 begins this day Thanks and as always, questions and comments welcome, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

1 0

Thumbnail image hinting in articles via metadata tags
by James Pearson 04 Apr '15

04 Apr '15

Hello. I'm a developer at reddit, and this morning I was looking into how we generate thumbnail images from Wikipedia (and I suppose all Mediawiki) articles, due to a user report that they had an unexpected thumbnail on their submission[0]. You can read my post there for more details, but essentially since we can't find an og:image or other similar metadata tag hinting to us what we should use for a thumbnail, we iterate through all the linked images on the page and pull out the largest one (you can view the code online if you're curious[1]). While this works reasonably well as a general heuristic, Wikipedia articles often have some more structure that could give us a better image to use. While writing this, I recalled that the Wikipedia Android app displays thumbnails in its search results. I think that's pulling from OpenSearch with the PageImages extension[2]? but I haven't really delved into that yet. I'm curious how those images get pulled - if it's taking into account infoboxes or such, or just the first image on the page, or what. Would it be feasible to include an og:image tag on pages for which we have a reasonable guess as to the thumbnail? Open Graph[3] is supported by what seems anecdotally to me to be a wide range of services, so good hints there would improve thumbnails for links on not just reddit, but Facebook, Twitter, various chat clients, I think several Wordpress plugins, etc. Thanks, - P [0]: https://www.reddit.com/r/bugs/comments/317n1v/thumbnail_acquisition_from_wi… [1]: https://github.com/reddit/reddit/blob/master/r2/r2/lib/media.py#L485-L542 [2]: https://www.mediawiki.org/wiki/API:Opensearch [3]: http://ogp.me/

10 12

Wikitext style norms for Parsoid serialization?
by Subramanya Sastry 04 Apr '15

04 Apr '15

Greetings! Among other things, Parsoid converts HTML to wikitext. There are two requirements as far as this serialization / conversion goes: * Preserving HTML -> HTML semantics (i.e. a list when serialized to wikitext and parsed back should render as an identical list) * Enforcing certain wikitext norms (i.e. use <nowiki/> to clarify quote parsing instead of <nowiki>'</nowiki>; serialize categories on their own line, etc.) In a large set of scenarios, there is no conflict between these two requirements, i.e. Parsoid's serialization to conform to certain wikitext style norms preserves the HTML -> HTML semantics. But, there are some scenarios where these requirements are in conflict. This conflict arises when Parsoid receives HTML that is malformed [1], or sends HTML that has no representation in wikitext [2], or sends HTML that will serialize to wikitext that editors will complain about [3]. We have adopted a somewhat adhoc strategy so far while favouring the HTML -> HTML preservation strategy. In some scenarios like [1], it is easier to argue that Parsoid should break HTML -> HTML semantics by effectively blaming the breakage in HTML roundripping on the client. But, it is a little less clear in scenarios like [3]. ==<nowiki/>== is clearly valid wikitext and can be parsed back to preserve HTML semantics, there is a good argument to be made to also drop such empty headers in Parsoid. So, even though WYSIWYG model breaks here, that could be seen as a less serious bug that outputting ==<nowiki/>==. That said, we definitely do not want to get into the business of implementing ad hoc heuristics to work around client bugs (while we have and could continue implement temporary workarounds till client bugs are fixed). That is a slippery slope to code complexity. So, two questions to answer here: 1. Are there a set of wikitext norms that are applicable across wikis and should be enforced as a syntactic style standard by Parsoid independent of clients? What are they and can they be documented on a wiki page by the editor community? Dont' emit "==<nowiki/>==" be one of those? 2. Independent of (1), should Parsoid implement an optional HTML normalization pass that it applies on behalf of clients when the right API parameter is passed in? This might be useful in scenarios where it is simpler to fix bad HTML than prevent the generation of bad HTML. In an ideal situation, if we can establish norms for (1), it will eliminate the need for (2) -- (2) is less desirable and is mostly a fallback beyond (1). Thanks, Subbu. [1] T94599: https://phabricator.wikimedia.org/T94599 [2] Example: <a data-my-attribute="foo" href="http://google.com">foo</a> or see T94766: https://phabricator.wikimedia.org/T94766 [3] T94867: https://phabricator.wikimedia.org/T94867

1 1

client-side HTML templating now in core
by Ryan Kaldari 03 Apr '15

03 Apr '15

MediaWiki core now has built-in support for using Mustache HTML templates from your client-side Javascript code. You can also use the same templates on the server-side with the TemplateParser class. Documentation: https://www.mediawiki.org/wiki/Manual:HTML_templates#mw.template_.28client-…

2 1

Moritz Muehlenhoff joins as Ops Security Engineer
by Mark Bergsma 03 Apr '15

03 Apr '15

Hi all, I'm very pleased to announce that as of <s>today</s>yesterday, Moritz Mühlenhoff will be joining the Ops team in the role of Operations Security Engineer. We're excited as for the first time we'll have an engineer on our team able to focus on enhancing the security of our infrastructure. Some of you Debian users may recognize his name; in his spare time he's very active in the Debian Security Team and sends out a large portion of their security advisory mails. ;) Moritz lives in Bremen, North Germany (internationally perhaps best known for being the home of Beck's beer) with his spouse Silvia and their 16 m/o son Tjark. Besides being a Debian Developer, he also very much enjoys Rugby Union and plays tighthead prop in his local club "Union 60 Bremen" in the third divison of Germany. He used to be a frequent visitor of film festivals such as the San Sebastian festival, but with the baby around home theatre has become more prevalent. :-) Moritz is working with us remotely, and can usually be found using his nick "jmm" on Freenode. Please join me in welcoming Moritz to the team! -- Mark Bergsma <mark(a)wikimedia.org> Lead Operations Architect Director of Technical Operations Wikimedia Foundation

4 3

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2015