Wikitech-l July 2015

wikitech-l@lists.wikimedia.org

97 participants
78 discussions

Getting mediawiki core to pass codesniffer standard
by Vivek Ghaisas 26 Sep '15

26 Sep '15

Hello, wikitech-l, tl;dr trying to make mw-core pass mw-codesniffer, expect large patches on Gerrit, and please help A lot of work has been done on MediaWiki codesniffer <https://phabricator.wikimedia.org/tag/mediawiki-codesniffer/> (the PHP_CodeSniffer standard for MediaWiki) over the last few months and this might be a good time to get core to pass our coding conventions <https://www.mediawiki.org/wiki/Manual:Coding_conventions/PHP>. Work on this has already started at T102609 <https://phabricator.wikimedia.org/T102609>. There were two primary reasons to send this email: 1. This is a pretty big task and any help would be very welcome! If we can get phpcs to run against core's master, it would make every patch contributors' work a little easier. 2. Work is being organized as subtasks of T102609, and has been divided on the basis of sniffs. Because of this, every patch is going to change a lot of unrelated files and a lot of reviewers are going to get added on Gerrit. Let's make this happen! Vivek Ghaisas (polybuildr)

5 4

RFC meeting this week
by Tim Starling 19 Sep '15

19 Sep '15

In the next RFC meeting, we will discuss the following RFC: * Improving extension management <https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_man…> The meeting will be on the IRC channel #wikimedia-office on chat.freenode.net at the following time: * UTC: Wednesday 21:00 * US PDT: Wednesday 14:00 * Europe CEST: Wednesday 23:00 * Australia AEST: Thursday 07:00 -- Tim Starling

5 6

I love Parsoid but it doesn't want me
by Ricordisamoa 18 Sep '15

18 Sep '15

Are there any stable APIs for an application to get a parse tree in machine-readable format, manipulate it and send the result back without touching HTML? I'm sorry if this question doesn't make any sense.

8 29

MediaWiki, as seen on TV
by Steven Walling 13 Sep '15

13 Sep '15

There's a pretty hilarious American police procedural TV show in 2015 called "CSI: Cyber", featuring mostly cybercrime. Obviously they have to dredge up snippets of code from places for screenshots on the show. Episode 4 happened to include a tidbit from MediaWiki 1.25/wmf3. Supposedly the code was a hack to make your printer blow up. Original lulz and screenshots via http://moviecode.tumblr.com/post/114815574587/this-is-from-csi-cyber-s01e04…

6 5

MediaWiki user survey and un-reachable users
by Mark A. Hershberger 17 Aug '15

17 Aug '15

We haven't announced this survey on wikitech-l yet, so if you run a wiki outside of those run by the WMF, please take the time to fill out the survey at <http://hexm.de/MWSurvey>. More information about the survey and its purpose can be found at <https://www.mediawiki.org/wiki/2015_MediaWiki_User_Survey>. That said, we received a report (T104010) from the Analytics team today that most downloads of MediaWiki are coming from China. The report indicated there were twice as many downloads from China as the U.S. in June. I don't know of a good way to reach these users -- I wasn't even aware of them till today. Through our efforts at outreach so far we've uncovered a number of private wikis that we wouldn't have been able to discover otherwise, but I'd like to extend our reach even farther. Can we add a link to the survey to the top of https://www.mediawiki.org/wiki/Download until the end of July? -- Mark A. Hershberger NicheWork LLC 717-271-1084

3 4

Bikeshedding a good name for "the api.php API"
by Brad Jorsch (Anomie) 11 Aug '15

11 Aug '15

When api.php was basically the only API in MediaWiki, calling it "the API" worked well. But now we've got a Parsoid API, Gabriel's work on a REST content API, Gabriel's work on an internal storage API, and more on the way. So just saying "the API" is getting confusing. So let's bikeshed a reasonably short name for it that isn't something awful like "the api.php API". I'm horrible at naming. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation

30 38

[recommended reading] do terrible things to your code
by Brian Gerstle 31 Jul '15

31 Jul '15

...or your users will! "Doing Terrible Things To Your Code <http://blog.codinghorror.com/doing-terrible-things-to-your-code/>" is a good read on testing by Jeff Atwood on his blog, Coding Horror <http://blog.codinghorror.com/>. I also found the "falsehood" snippets poignant—maybe we should come up with some for Wikipedia ;-). Here are a couple off the top of my head, at least for "official" Wikipedia instances: 1. Wikipedia sites all have standard ISO/BFC prefixes (see sitematrix <https://www.mediawiki.org/wiki/Special:SiteMatrix>) 2. A site's main page is always titled "Main Page" (also see sitematrix) 3. A page's revision is a reliable snapshot of its content (nope: transclusions [and images?] can change independent of a page revision) 4. API error messages are plain text (nope, can contain HTML <https://phabricator.wikimedia.org/T107082>) Interested in hearing "falsehoods" you've encountered. Cheers, Brian -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

3 2

Measuring performance
by Jon Robson 30 Jul '15

30 Jul '15

The reading team are currently focusing energy on speeding up the site for all our users (https://phabricator.wikimedia.org/T98986 is the tracking bug where this work can be followed) Off the back of https://phabricator.wikimedia.org/T105361 I had a quick chat with Ori to document how the performance team is currently identifying problems with MediaWiki's code. I'm sharing here, so anyone who is interested in helping us improve the time our users can load our content can analyse our data, raise tasks, and submit patches. I'm hoping this will be useful for anyone who wants to get involved in an effort to make our site faster for our users (this is not desktop specific). If you have anything useful to add please do, after some discussion or nods I'd love to share some best practices on mediawiki.org Tool 1) Use http://webpagetest.org (no credentials necessary) * Use https://en.wikipedia.org/wiki/Facebook as an example wiki page * Choose a region of the world and browser * Select first view only since this is what we are currently interested in (repeat view is when they load again - and it should be quicker as it is from cache). * Capture video can be turned off - I personally find the screenshots more useful To shout out some of the advanced settings, the more interesting/useful features include: *Chrome > capture dev tools timeline * Setting speed 3G or 2G * Script can be used to conditionally turn on things which are not yet available to everyone e.g. VisualEditor You can do a lot of this in your Chrome browser locally, but different browsers may have different behaviours and are in a fixed location so this does not get captured in this tool. The visual screenshots also make it easier to see where things get blocked. With the timeline from advanced tools you can match up white screens with blocking scripts/styles Tool 2) Add http://performance.wikimedia.org to your browser bookmarks. Navigation timing section is probably the most interesting right now. It points to https://grafana.wikimedia.org (no credentials needed) which is powered by http://graphite.wikimedia.org (Access graphite with your wikitech credentials). This data is sourced from our users, so is a good representation of how we are doing. If a graph is missing you can create a new one from data in graphite by clicking "add row" or editing an existing graph. Clicking edit on https://grafana.wikimedia.org/#/dashboard/db/navigation-timing?panelId=12&f… you'll be able to understand where the data comes from on graphite e.g. metrics/frontend/navtiming/totalPageLoadTime Note for graphs median data is less sensitive to edge cases so best to use this as a more realistic indicator. Folders in graphite, are populated by scripts that live in: https://github.com/wikimedia/operations-puppet/tree/production/modules/webp… To create a graph, simply go to an existing workboard, save it under a different name (this clones it) - don't worry you can't mess up and delete existing workboards. Tool 3) Speedcurve requires you to setup an account but it gives you an opiniated view about things you care about and is nicely presented so could be a good source of inspiration for your own grafana dashboard. To oversimplify what it does: each day it will access a page, store result, allow you to see historic data. Note the performance team has plans to setup infrastructure to automate this. Tool 4) is one we are not using - http://sitespeed.io. We might want to use it for performance regressions test. In the grand scheme of things it would be great to get to a place where Jenkins complains if you cause a regression in firstPaint time but we are a long way from that but let's work in that direction :-) Let's live up to the Hawaiian word after which we are named! Apologies if this is oversimplified, please take this as an opportunity to share how you/your team/your company test page performance. I see this mailing list as a good place to share these sort of things!

2 1

compressed old_text
by wp mirror 30 Jul '15

30 Jul '15

0) Context In the XML dump files, I get <text ...>plaintext</text>. When I build a mirror using XML dump files, I get: old_text: plaintext old_flags: utf-8 However, when I then create a new page on my mirror, I get: old_text: ciphertext old_flags: utf-8.gzip 1) Objective When I build a mirror, I would like to compress the <text ...>plaintext</text> to get: old_text: ciphertext old_flags: utf-8,gzip I would like this done for every text revision, so as to save both disk space and communication bandwidth between web server and browser. 2) Problem There is little relevant documentation on <https://www.mediawiki.org>. So I have run a few experiments. exp1) I pipe the plaintext through gzip, escape for MySQL, and build the mirror. However, when I browse, I get the message: ``The revision #165770 of the page named "Main Page" does not exist'' When I look in the database, some kind of ciphertext does indeed exist. 3) Variants Many utilities compress plaintext using LZ77 and Huffman encoding, but each differs as to the file header and tail. Some versions of deflate have no header at all. So I try four more experiments: exp2) gzip, but throw away the 10 byte header (to simulate deflate) /bin/gzip | tail -c +11 exp3) perl compress /usr/bin/perl -MCompress::Zlib -e 'undef $/; print compress(<>)' exp4) python compress, then throw away the single-quotes /usr/bin/python -c \"import zlib,sys;print repr(zlib.compress(sys.stdin.read()))\" | /bin/sed 's/^.//; s/.$//' exp5) zlib-flate from the qpdf DEB package /usr/bin/zlib-flate -compress For all experiments, the browser gives the same error message. 4) Reading compressed old_text It should be possible to read the old_text ciphertext using command-line tools. I created a user page which mediawiki stored compressed. It is displayed correctly by the browser. But when I tried to read it directly from the database, there were problems. (shell) mysql --host=localhost --user=root --password simplewiki --skip-column-names --silent --execute 'select old_text from simplewiki.text where old_id=5146705' | zlib-flate -uncompress Enter password: flate: inflate: data: incorrect data check 5) Request Please provide documentation as to how mediawiki handles compressed old_text. a) How is plaintext compressed? b) Is the ciphertext escaped for MySQL after compression? c) How does mediawiki handle old_flags=utf-8,gzip? d) How are the contents of old_text unescaped and decompressed for rendering? e) Where in the mediawiki code should I be looking to understand this better? SIncerely Yours, Kent

2 1

2015-07-29 Scrum of Scrums notes
by Grace Gellerman 30 Jul '15

30 Jul '15

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-07-29

3 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l July 2015