Wikitech-l May 2016

wikitech-l@lists.wikimedia.org

113 participants
123 discussions

MediaWiki-Codesniffer 0.7.1 released
by Legoktm 06 May '16

06 May '16

Hello! MediaWiki-Codesniffer 0.7.1 is now available for use in your MediaWiki extensions and other projects. 0.7.0 was a botched release that had bugs, please don't use it. Here are the notable changes since the last release (0.6.0): * Also check for space after elseif in SpaceAfterControlStructureSniff (Lethexie) * Factor our tokenIsNamespaced method (addshore) * Make IfElseStructureSniff can detect and fix multiple white spaces after else (Lethexie) * Make SpaceyParenthesisSniff can fix multiple white spaces between parentheses (Lethexie) * Make spacey parenthesis sniff work with short array syntax (Kunal Mehta) * Speed up PrefixedGlobalFunctionsSniff (addshore) * Update squizlabs/php_codesniffer to 2.6.0 (Paladox) Thanks to some awesome work by Addshore, MW-CS is now twice as fast for large repos like mediawiki/core! This release also features contributions from GSoC student Lethexie who will be working on improving our static analysis tools this summer! -- Legoktm

1 0

2016-05-04 Scrum of Scrums meeting notes
by Grace Gellerman 06 May '16

06 May '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-05-04 = 2016-05-04 = == Technology == === Release Engineering === For all: * T128190 - Migration of browsertests* Jenkins jobs to selenium* jobs ** The migration of browsertests* to selenium* is almost complete, however, Željko needs people to claim their failing browser tests. See the task for more information. *** The task has a table, but it's not clear what you want people to do (are you just asking about the rows with missing "contact from browsertests.yaml" fields)? For operations: * T126594 - Disable HHVM fcgi server on CI slaves * Need help from ops to review and merge these two patches (we dont need HHVM running as a daemon on CI boxes): ** https://gerrit.wikimedia.org/r/#/c/269946/ ** https://gerrit.wikimedia.org/r/#/c/269947/ * https://phabricator.wikimedia.org/T133911 - Bump quota of Nodepool instances (contintcloud tenant) ** More instances needed. Clarified with Chase last week: pending Andrew. No urgency. * Two related tasks, each have patches that are needed to streamline the scap3 migration: ** T133211 - Automate the generation deployment keys (keyholder-managed ssh keys) *** https://gerrit.wikimedia.org/r/#/c/284418/ ** T132747 - scap::target shouldn't allow users to redefine the user's key *** https://gerrit.wikimedia.org/r/#/c/285519/ === Security === * Reviews: ** json-schema done, AuthManger done (no more comments, a few minor things before all patches are +1'ed) ** Starting on T129584 this week * Starting work on AuthService next week (heads up to Services, we'll probably be scheduling a few meetings with you) (Marko: ack && yay!) * Ops: ping again on T128819 === Services === (Marko cannot attend today, sorry) * RESTBase ** working on storing all auth checks locally (now we are calling the MW API every time) * EventBus / Change propagation ** started using it in production for the summary endpoint today ** more dependency updates to follow soon * Cassandra move to 2.2.6 soon ** first up: maps cluster * use Scap3 - https://lists.wikimedia.org/pipermail/wikitech-l/2016-April/085299.html === Technical Operations * '''Blocking''': ** none * '''Blocked''': ** none * '''Updates''': ** May 15 (Chrome SPDY removal deadline). Getting HTTP/2 working fully deployed till then ** started using letsencrypt for various small services == Product == === Reading === * Most of Reading engineering is at an offsite today, I believe. ==== Reading Infrastructure ==== * AuthManager core patches are just waiting for security +1s. Work is ongoing on extensions; CentralAuth, LdapAuthentication, ConfirmEdit could use reviews if anyone is interested. === Editing === ==== Collaboration ==== * '''Blocking''': ** External store work. External Store deployed everywhere on Beta with no complications. Work on this continues. We now need to set up a second External Store on Beta for Flow, to test the migration. * '''Blocked''': ** Work on Flow dumps continuing on the ops side. https://phabricator.wikimedia.org/T119511 and https://phabricator.wikimedia.org/T89398 . * '''Updates''': ** Continuing notification work on: *** Cross-wiki notifications coming by default on May 12th! *** Echo email formatter *** Work continues on the new Echo MVC architecture ==== Parsing ==== * We got our first visual diff test run comparing Tidy with HTML5depurate. Results @ http://mw-expt-tests.wmflabs.org/ ... Making notes @ https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy ... We will use this as the basis for figuring out what things might break if we replace Tidy today and what needs fixing and where. * Scott has been working with Ops to get OCG kinks ironed out. ==== Language ==== * '''Blocking''': ** Apertium->Jessie. Yet to finalize plan and proceed. * '''Blocked''': * '''Updates''': ** cxserver service will be migrated to scap3 deployment soon. ** Translate (twn, meta,..) now using Apertium MT from cxserver. == Discovery == * '''Blocking''': none * '''Blocked''': none * Preparing for ElasticSearch 2.0 migration * Results for A/B test on portal language link location published: https://commons.wikimedia.org/wiki/File:Wikipedia_Portal_Test_of_Language_D… * TextCat A/B test launching soon * Portal A/B test adding descriptions to project links to start this week * WDQS redeployed, some performance issues * Graphs have ability to use WDQS directly now * Team offsite in 2 weeks (17-21) ==Analytics == *Scaling of pageview API , more work than anticipated *Working new domain analytics.wikimedia.org, wikistats 2.0 migration, meeting with research to map early states of project *Still trouble with jenkins * == Fundraising Tech == * coding new PayPal integration * pulling in lots of CiviCRM upstream changes * making CentralNotice fail gracefully in odd cache edge cases * Casey hanging out with reading offsite * more work towards replacing ActiveMQ

5 6

garbage characters show up when fetching wikimedia api
by Trung Dinh 06 May '16

06 May '16

Hi all, I have an issue why trying to parse data fetched from wikipedia api. This is the piece of code that I am using: api_url = 'http://en.wikipedia.org/w/api.php<http://en.wikipedia.org/w/api.php?action=query&list=recentchanges&rclimit=5…>' api_params = 'action=query&list=recentchanges&rclimit=5000&rctype=edit&rcnamespace=0&rcdir=newer&format=json&rcstart=20160504022715<http://en.wikipedia.org/w/api.php?action=query&list=recentchanges&rclimit=5…>' f = urllib2.Request(api_url, api_params) print ('requesting ' + api_url + '?' + api_params) source = urllib2.urlopen(f, None, 300).read() source = json.loads(source) json.loads(source) raised the following exception " Expecting , delimiter: line 1 column 817105 (char 817104" I tried to use source.encode('utf-8') and some other encodings but they all didn't help. Do we have any workaround for that issue ? Thanks :)

3 2

Fall 2015 Tool Labs user survey data published
by Bryan Davis 05 May '16

05 May '16

[0]: https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855

2 4

load.php will now enforce the "no sessions" constraint
by Brad Jorsch (Anomie) 05 May '16

05 May '16

It has long been the case that the load.php entry point, to support more aggressive caching, should not depend on details of the current session. For example, it should not access the session user, and should use its own 'lang' parameter for i18n messages rather than the session user's language. After a fair bit of work[1] by Gergő Tisza, Timo Tijhof, Bryan Davis, and myself, we have now flipped the switch to make attempted session access fail with an exception when serving load.php. This will go out to Wikimedia wikis with 1.28.0-wmf.1 next week.[2] We've had this in logging mode for a while and haven't seen any log messages in production recently, so we anticipate that this change won't cause any issues for Wikimedia wikis. Third-party wikis may have to update their extensions. If there are problems, they'll manifest as missing CSS or JavaScript in the page, and when examining the load.php output there will be a comment at the top reporting that a BadMethodCallException was caught. The logged exception's message will be "Sessions are disabled for this entry point". [1]: See https://phabricator.wikimedia.org/T127233 and its subtasks. [2]: See https://www.mediawiki.org/wiki/MediaWiki_1.28/Roadmap for the schedule. -- Brad Jorsch (Anomie) Senior Software Engineer Wikimedia Foundation

1 0

"Troubleshooting Git/Gerrit/git-review" docs: Help welcome.
by Andre Klapper 05 May '16

05 May '16

Hej, I went ahead & centralized the 5 git/Gerrit/git-review troubleshooting sections that I've found so far on random mediawiki.org pages into one single page (and eliminated duplicates with different solutions): https://www.mediawiki.org/wiki/Gerrit/Troubleshooting 1) More help is welcome to clean up and structure that page (plain git vs. git-review vs. Gerrit UI). 2) As a user, I prefer section headings to describe problems instead of solutions (as I don't know which solution is related to my problem). On https://www.mediawiki.org/wiki/Gerrit/Tutorial there is one section called "Pushing via HTTPS when SSH is not functional" which I'd also like to move to [[mw:Gerrit/Troubleshooting]]. Anyone knows how users would actually realize that SSH is not functional? (Specific output after a specific command?) How would Gerrit users realize they are behind a proxy server? Thanks for your help. andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/

5 6

Analytics Kafka Upgrade next week
by Andrew Otto 05 May '16

05 May '16

Hiya, We’ll be upgrading the Analytics Kafka cluster from 0.8.2 to 0.9.0.1 next week. This is scheduled to start at Wednesday May 11th at 13:00 UTC (9:00 EST, 6:00 PST). If all goes well*, this should be a rolling upgrade with no downtime. Just a heads up, thanks! -Andrew & Luca *everybody knock on wood now

1 0

Fwd: [Wikimedia-l] Invitation to upcoming office hours with interim ED
by Pine W 05 May '16

05 May '16

Forwarding. Pine ---------- Forwarded message ---------- From: "Katherine Maher" <kmaher(a)wikimedia.org> Date: May 4, 2016 17:47 Subject: [Wikimedia-l] Invitation to upcoming office hours with interim ED To: <wikimedia-l(a)lists.wikimedia.org>, <wmfall(a)lists.wikimedia.org>, < wikimediaannounce-l(a)lists.wikimedia.org> Cc: Hi everyone, **Summary: I am delighted to invite you to join me for two upcoming office hours, where I’ll answer community questions and share updates on the Foundation’s work.** It’s been a busy few weeks around the Wikimedia Foundation offices. We shared our 2016-2017 annual plan, finished our quarterly reviews, and attended Wikimedia Conference 2016 in Berlin with the Wikimedia affiliates. [1] In Berlin, I had the chance to do one of my favorite things: sit with Wikimedians, listen, debate, and plan for the future. Of course, Berlin is just one gathering, and there are thousands of other perspectives out there. I want to hear more of these perspectives, and so I’m looking forward to hosting two office hours over the coming weeks. We plan to hold a traditional office hours on IRC, and will also experiment with a video Q&A. We hope these different formats will make it easier for more people to participate using their preferred communications channels. We’ve chosen two different time zones, with the goal of reaching as many people as possible. They are as follows: *Video session* *This session will be recorded, and the video will be posted on Commons/Meta. Due to video conferencing limitations, we encourage advance questions.* Wednesday, 11 May 2016 00:00-1:00 UTC | 17:00-18:00 PDT [2] *IRC session* *This session follows the May monthly metrics meeting.[4] Like other office hours, it will be held in #Wikimedia-office on Freenode.* Thursday, 26 May 2016 19:00-20:00 UTC | 12:00-13:00 PDT [3] We’re also collecting questions in advance for those who can’t make either of those sessions. We’ve created a page on Meta where you can leave questions or comments, check the details on the location of each session: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Executive_Director/May… Please share this invitation with others you think may be interested! I look forward to speaking soon, Katherine Translation notice - This message is available for translation on Meta-Wiki: https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Executive_Director/May… [1] https://meta.wikimedia.org/wiki/Wikimedia_Conference_2016 [2] Time converter link: http://www.timeanddate.com/worldclock/fixedtime.html?hour=0&min=00&sec=0&da… [3] Time converter link: http://www.timeanddate.com/worldclock/fixedtime.html?hour=19&min=00&sec=0&d… [4] https://meta.wikimedia.org/wiki/WMF_Metrics_and_activities_meetings -- Katherine Maher Wikimedia Foundation 149 New Montgomery Street San Francisco, CA 94105 +1 (415) 839-6885 ext. 6635 +1 (415) 712 4873 kmaher(a)wikimedia.org _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>

3 2

ArchCom-RFC status update: 2016-05-04
by Rob Lanphier 05 May '16

05 May '16

Hi everyone, I've included the ArchCom-RFC status update in this mail below, which now has a dedicated wiki page: <https://www.mediawiki.org/wiki/Architecture_committee/Status> Assuming we build the habit, this should be updated weekly. Rob ---- Recent RFC office hours - 2016-05-04: Phab:E169 - T130528: RFC: PSR-6 Cache interface in Mediawiki core - Declined. A lot of the discussion was about these three questions: 1. what problem was this RFC trying to solve? 2. if BagOStuff is an obstacle, then which is true? - a) need a better caching abstraction than BagOStuff? or - b) does BagOStuff just need incremental improvement - c) should BagOStuff be deprecated, and not replaced with anything? 3. if we decide to rethink our caching model, do we need tagged caching? Upcoming RFC office hours - 2016-05-11: Phab:E66/34(E171) - T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap - The ArchCom-RfCs board has a "Ready for RFC meeting" column which should contain an ordered queue of RFCs planned for IRC office hour Entering Final Comment Period - Phab:T120164 - this is a last call for a process RFC about last calls. See RobLa's email to wikitech-l on 2016-05-04 about this. RFC inbox - ArchCom RFC board: - empty on 2016-05-04 Shepherd status - Brion - (?) - Daniel - Software Quality working group? - Working on Multi Content Rev Spec with Brion - Gabriel - T39902 RFC: Implement rendering of redlinks as post-processor (Gabriel): Solutions for highlighting links to non-existing pages in Parsoid HTML. Plan in place / agreed between Parsing and Services. Implementation in change propagation service ready, preparing for deploy possibly next week. - T122942 RFC: Support language variants in the REST API (Gabriel): Waiting for progress on more general question of language selection granularity / strategy & T114662. - Roan - I need to start the second part of T108655, but the recent comments have me confused. I'll need to talk to Timo and figure out what the subject of part two should be. - RobLa - Created mw:RFCstatus - still need to schedule an RFC triage meeting outside of ArchCom-RFC time - Tim - No update - Timo - No update No activity in the last two weeks - T123753 Establish retrospective reports for Security and Performance incidents (RobLa) - T122825 Service ownership and minimum maintenance requirements (Gabriel) - T124504 Transition WikiDev '16 working areas into working groups (RobLa) - T66214 Use content hash based image / thumb URLs & define an official thumb API (Brion) - T113034 RFC: Overhaul Interwiki map, unify with Sites and WikiMap (Daniel) - T128351 RFC: Notifications in core (Brion) - T122825 Service ownership and minimum maintenance requirements (Gabriel) - T54807: Identify and remove legacy preferences from MediaWiki core (no shepherd) - T88596 Improving extension management (Daniel) - T114444 RFC: Introduce notion of DOM scopes in wikitext (Tim)

1 0

Phabricator upgrade and UI changes
by Federico Leva (Nemo) 05 May '16

05 May '16

AFAICT, in the last few days there were two unannounced upgrades of Phabricator, to an unspecified version of the software, the second of which applied rather radical UI changes, especially to Maniphest. Initially I wasn't sure most/which UI changes were bad/good or why, but now I've observed myself for a few days and I took the time to take screenshots and describe my findings. Of course, most are about * the consistent enlargement of some items to the expense of others, * the introduction of a sidebar to host an ever-growing amount of buttons, * the move of most task information under the description or under the sidebar. Please comment on the reports, add your findings and file new reports for separate issues (I've not tested areas outside Maniphest, for instance): * https://phabricator.wikimedia.org/T134393 * https://phabricator.wikimedia.org/T134394 * https://phabricator.wikimedia.org/T134398 It would also be nice to have some precise and authoritative information on when Phabricator was upgrade to what version, what the release notes and gotchas are, etc. Did I miss such an announcement and if yes where is it? Nemo

5 4

Last call: on the idea of "last call"
by Rob Lanphier 05 May '16

05 May '16

Hi folks, One thing we've implicitly adopted is "last calls" for ArchCom-RFCs. I filed it as an RFC to get it on our workboard: https://phabricator.wikimedia.org/T120164 However, what that means is that we need to have a last call on the "last call" RFC. Obligatory xkcd reference: https://xkcd.com/244/ Barring objection in Phab, we'll consider T120164 approved then. Rob

1 0

How to click custom html option button via api?
by Shrinivasan T 04 May '16

04 May '16

I am trying to work on https://ta.wikisource.org with python API. This tamil wikisource has custom html option buttons to mention the edit status. See a screenshot here. https://snag.gy/f3aTBn.jpg How to enable these custom option buttons via API? Thanks. -- Regards, T.Shrinivasan My Life with GNU/Linux : http://goinggnu.wordpress.com Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com Get Free Tamil Ebooks for Android, iOS, Kindle, Computer : http://FreeTamilEbooks.com

3 5

Automatic image colorization
by Ori Livneh 04 May '16

04 May '16

Colorization <https://en.wikipedia.org/wiki/Film_colorization#Digital_colorization> refers to the process of adding color to black-and-white photographs. This work was historically done by hand. These days, colorization is usually done digitally, with the support of specialized tooling. But it is still quite labor-intensive. A forthcoming paper <http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en/> from researchers at Waseda University of Japan have developed a method for automatic image colorization using deep learning neural network. The results are both impressive and easy to reproduce, as the authors have published their code <https://github.com/satoshiiizuka/siggraph2016_colorization> to GitHub with a permissive license. Someone has already taken this code and packaged it as a simple webapp, available at http://colorizr.io/ (NSFW). The webapp lets you upload black-and-white pictures and colorizes them for you. (The site is currently not safe for work because it displays a gallery of recent uploads. We know how that goes.) In this thread, let's discuss how this technology could be integrated with the projects. Should we have a bot that can perform colorization on demand, the way Rotatebot <https://commons.wikimedia.org/wiki/User:Rotatebot> can rotate images?

4 6

ImageMagick reported security vulnerability
by Brion Vibber 04 May '16

04 May '16

There's a reported ImageMagick security vulnerability making the rounds: http://arstechnica.com/security/2016/05/easily-exploited-bug-exposes-huge-n… Many MediaWiki sites are configured to use ImageMagick's 'convert' command to perform image rescaling/thumbnailing, so it's worth double-checking that everything is secure... MediaWiki already performs file type validation checks on uploads, which *should* prevent exploitation of the vulnerability for the standard image types (JPEG, PNG, GIF, SVG etc) -- this is one of the recommended mitigation strategies. But I am not confident enough to pronounce us immune without actually checking! Some folks are also recommending tweaking the ImageMagick policy.xml to disable the most dangerous code modules; see the article linked above for further links. -- brion

1 0

upgrade of irc.wikimedia.org
by Daniel Zahn 03 May '16

03 May '16

Hello, today the server that runs irc.wikimedia.org has been upgraded. Or actually, we have installed a new, second server running on Debian jessie and ported the puppet manifests and the IRC bot to work on that, while the old server still exists. MW appservers are sending RC data to both of them. If you are making a new connection to irc.wikimedia.org you should now be served by the new host, kraz.wikimedia.org but old connections have not been broken. The old server is still running unchanged and is reachable as argon.wikimedia.org (as it was before too). We have also made the change "URLs in the recent changes IRC feed will no longer be rewritten to unencrypted HTTP." which was announced and scheduled for today, May 2nd. No action is required if your bot automatically reconnects, but bot owners should ensure no IP addresses are hardcoded (see T123729 for details.) DNS caches slowly roll over and new connections will use kraz. Existing sessions on argon and clients that hardcoded the argon IP won't be affected yet, but after a grace period we are going to shut down the old server argon. refs: https://phabricator.wikimedia.org/T123729#2216681 https://meta.wikimedia.org/wiki/Tech/News/2016/02 Best regards, Daniel -- Daniel Zahn <dzahn(a)wikimedia.org> Operations Engineer

1 0

Readership metrics for the timespan until April 24, 2016
by Tilman Bayer 03 May '16

03 May '16

Hi all, here is the usual look <https://www.mediawiki.org/wiki/Reading/Readership_metrics_reports> at our most important readership metrics. This time, highlights include the impact of the iOS app’s big release of a revamped version, a Google bug that brought the Android app a windfall of about half a million new users, and a first examination of the new unique devices dataset. As laid out earlier <https://lists.wikimedia.org/pipermail/mobile-l/2015-September/009773.html>, the main purpose is to raise awareness about how these are developing, call out the impact of any unusual events, and facilitate thinking about core metrics in general. As always, feedback and discussion are welcome. Week-over-week and month-over-month changes are now being recorded on the Product page <https://www.mediawiki.org/wiki/Wikimedia_Product#Reading> at MediaWiki.org. This edition of the report covers a timespan of eleven weeks. Some other recent items of interest, in case they didn’t already catch your attention: - At the Foundation’s February metrics meeting, I gave an update on longer-term traffic trends since 2013. (TL;DR: Overall pageviews have been flat to slightly declining, mobile has been steadily rising but recently slowed down, desktop is declining.) See the slide deck here or the chart below (updated with March data): [image: Wikimedia monthly pageviews (worldwide, mobile vs desktop), 2013-2016.png] <https://commons.wikimedia.org/wiki/File:Wikimedia_monthly_pageviews_(worldw…> - The WMF Reading team published its quarterly review presentation <https://commons.wikimedia.org/wiki/File:Reading_Quarterly_Review_Q3_2015-16…> for Q3 2015-16 (January-March), with lots of traffic and usage data. - I recently gave a Tech Talk about “new readership data” <https://commons.wikimedia.org/wiki/File:New_readership_data_(Wikimedia_Foun…> - i.e. some data that, separately from the core metrics discussed here, gives insights on how Wikipedia is read. Now to the usual data. (All numbers below are averages for February 8-April 24, 2016 unless otherwise noted.) Our family of core metrics who are populating this report mourns a deplorable casualty (iOS daily active users are no longer available due to the app’s switch to opt-in data collection, T130432 <https://phabricator.wikimedia.org/T130432>), rejoices about the speedy convalescence of an accident victim thanks to expert therapy administered by the Analytics and iOS teams (iOS pageviews, T131824 <https://phabricator.wikimedia.org/T131824>), sends well-wishes for recovery to two other patients (Android app DAUs and pageviews, T132965 <https://phabricator.wikimedia.org/T132965>), and lastly welcomes a new member: unique devices. Pageviews Total: 538 million/day (+1.5% from the previous report) (caveat: likely undercounting by 1-2 million/day since April due to a bug <https://phabricator.wikimedia.org/T132965> related to the Android app’s gradual switch to RESTBase) Context (April 2015-April 2016): [image: Wikimedia daily pageviews, all vs. mobile (April 2015-2016-04-24).png] Overall, pageviews were up compared to the previous report (which had still covered the slump around Christmas). There was a drop in early/mid March though, which probably can’t be fully explained by a change in the pageview definition <https://meta.wikimedia.org/wiki/Research:Page_view#Change_log> on March 9 that improved the exclusion of bot pageviews, nor does it look seasonal compared to March 2014 and March 2015. See also the Vital Signs dashboard <https://vital-signs.wmflabs.org/#projects=all/metrics=Pageviews>. Desktop: 55.2% (previous report: 54.3%) Mobile web: 43.6% (previous report: 44.4%) Apps: 1.2% (previous report: 1.3%) (caveat: app percentage likely too low due to the aforementioned Android-related bug) Context (April 2015-April 2016): [image: Wikimedia daily pageviews, mobile percentage (April 2015..2016-04-24).png] Mobile pageviews now regularly reach parity on weekends. Global North ratio: 76.5% of total pageviews (previous report: 78.3%) Context (April 2015-April 2016): [image: Percentage of Wikimedia pageviews from the Global North (April 2015..2016-04-24).png] <https://commons.wikimedia.org/wiki/File:Percentage_of_Wikimedia_pageviews_f…> In the previous report, we could already witness the ratio of Global North pageviews falling (or conversely, the Global South ratio rising) from a peak at the beginning of January. Afterwards, it saw a somewhat conspicuous drop in mid-February, went back up a bit but is now back to the levels from a year ago, before the HTTPS-only rollout in June. Unique devices Recently, the Analytics team made a new metric available: Daily and monthly unique devices (see their announcement blog post <https://blog.wikimedia.org/2016/03/30/unique-devices-dataset/> for background and details). These estimated numbers are provided for all Wikimedia projects (separately for the desktop and mobile web version), but because of the instrumentation method, there is no global metric for all projects. For now, we’ll pick the daily numbers of the English Wikipedia for inclusion in this report. As for other parts of this reports, the selection and presentation are always being optimized; feedback is welcome. Daily unique devices estimate for English Wikipedia: - Mobile web: 33.2 million - Desktop: 25.7 million - Total: 58.9 million - Average mobile web ratio: 56.5% Like on many of our projects, mobile(-site using) devices outnumber desktop devices, but there are fewer pageviews per device on mobile. (For comparison, the English Wikipedia’s pageviews were at 55.9% desktop, 42.9% mobile web and 1.2% apps; i.e. slightly less mobile than our sites overall.) Context (January-April 2016): [image: Daily unique devices for English Wikipedia (estimate), mobile web vs. desktop, 2016-01-01..2016-04-24.png] The early/mid March drop observed above for pageviews is visible here, too. New app installations Android: 31.6k/day (-17.8% from the previous report) Daily installs per device, from Google Play Context (last three months, log scale): <https://commons.wikimedia.org/wiki/File:Wikipedia_Android_app_-_Daily_insta…> Around March 11/12, Google accidentally redirected Wikipedia search results directly to the app’s listing in Google Play, instead of offering the user to read the article on the mobile web version of Wikipedia. (This was a general bug affecting other apps/websites too.) While resulting spike of ca. 600k additional installs was partially negated by rising uninstalls, a month later the app’s install base still remained around half a million - ca. 3% - larger than before the incident. Overall though, due to a general decrease, downloads per day are down from the previous report <https://commons.wikimedia.org/w/index.php?title=File%3AReadership_metrics_f…>, which had included the “Best Apps of 2015” promotion that brought ca. 350k additional installs. (See also the summary <https://commons.wikimedia.org/w/index.php?title=File%3AReading_Quarterly_Re…> in our quarterly review and the discussion of retention rates below) iOS: 5.06k/day (+2.5% from the previous report) Download numbers from App Annie Context (last three months): <https://commons.wikimedia.org/wiki/File:Wikipedia_iOS_app_daily_downloads,_…> In March, the launch of the revamped 5.0 version of the app, with accompanying media attention, brought roughly 50k additional installs. There also was a smaller spike in mid-February whose reason we don’t know. App user retention Android: 15.4% (previous report: 16.6%) (Ratio of app installs opened again 7 days after installation, among all installed one week before a date that falls within this report. 1:100 sample.) Context (last three months): [image: Day-7 retention of Wikipedia Android app users (install dates 2015-10-01..2016-04-17).png] <https://commons.wikimedia.org/wiki/File:Day-7_retention_of_Wikipedia_Androi…> As remarked in earlier reports, this data is a bit too noisy for drawing conclusions about whether retention changed significantly between different releases. But we can discern an effect of the aforementioned Google bug that inadvertently sent many surfers from the search results page directly to the app’s store page. While retention dropped a lot on the first day of the incident, it actually rose far above average later before the spike ended. Overall, the retention of users who came in during those three days (March 11-13) was 14.1%, not far below the average of 15.4%. iOS: N/A A new instrumentation has been built to measure retention in the same way as for the Android app, but unfortunately it turned out to be a bit buggy, yielding rosy retention rates above 100% ;) We’ll resume reporting after this is sorted out (T126693 <https://phabricator.wikimedia.org/T126693>). Since improving retention was a central goal of the new 5.0 version (and the new instrumentation would not have been available for comparison in older versions of the app, anyway), the iOS team resorted to Apple’s metric once again in order to assess the effect of the rollout. Disregarding the imperfections that had made us stop relying on it earlier, it indicated that day-7 retention had risen from about 10% to 15% in the new version (see quarterly review <https://commons.wikimedia.org/w/index.php?title=File%3AReading_Quarterly_Re…> ). Unique app users Android: N/A Unfortunately, there is currently no valid data for this metric because of issues having to do with the app’s recent switch to the RESTBase-based Content Service, see T132965 <https://phabricator.wikimedia.org/T132965>. (I might still have included the usual 3-month chart showing the development up to that point, but as I’m writing this, Google Sheets doesn’t let you <https://productforums.google.com/forum/#!msg/docs/yB_xNk6mn8U/QjRDR4WtNAAJ> create timeline charts that include the month of February. True story; this report covers not one but two severe bugs in Google products ;) The linked bug contains a chart with a different timespan though - keep in mind that the decline visible from the beginning April is artificial.) iOS: N/A With the new 5.0 version that launched in March, only a small minority of devices continue to send the data that this metric is based on (T130432 <https://phabricator.wikimedia.org/T130432>), so it is no longer valid as a measure of the apps overall usage volume. Data sources For reference, the queries and source links used are listed below (access is needed for each). Unless otherwise noted, all content of this report is © Wikimedia Foundation and released under the CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/> license. Most of the above charts are available on Commons, too <https://commons.wikimedia.org/w/index.php?title=Special:ListFiles&offset=20…> . hive (wmf)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, sum(IF(access_method <> 'desktop', view_count, null)) AS mobileviews, SUM(view_count) AS allviews FROM wmf.projectview_hourly WHERE year>0 AND agent_type = 'user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; hive (wmf)> SELECT access_method, SUM(view_count)/77 FROM wmf.projectview_hourly WHERE agent_type = 'user' AND CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) BETWEEN "2016-02-08" AND "2016-04-24" GROUP BY access_method; hive (wmf)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")), SUM(view_count) AS all, SUM(IF (FIND_IN_SET(country_code, 'AD,AL,AT,AX,BA,BE,BG,CH,CY,CZ,DE,DK,EE,ES,FI,FO,FR,FX,GB,GG,GI,GL,GR,HR,HU,IE,IL,IM,IS,IT,JE,LI,LU,LV,MC,MD,ME,MK,MT,NL,NO,PL,PT,RO,RS,RU,SE,SI,SJ,SK,SM,TR,VA,AU,CA,HK,MO,NZ,JP,SG,KR,TW,US') > 0, view_count, 0)) AS Global_North_views FROM wmf.projectview_hourly WHERE year > 0 AND agent_type='user' GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000; https://console.developers.google.com/storage/browser/pubsite_prod_rev_0281… (“overview”) https://www.appannie.com/dashboard/252257/item/324715238/downloads/ (select “Total”) SELECT LEFT(timestamp, 8) AS date, SUM(IF(event_appInstallAgeDays = 0, 1, 0)) AS day0_active, SUM(IF(event_appInstallAgeDays = 7, 1, 0)) AS day7_active FROM log.MobileWikiAppDailyStats_12637385 WHERE timestamp LIKE '2016%' AND userAgent LIKE '%-r-%' AND userAgent NOT LIKE '%Googlebot%' GROUP BY date ORDER BY DATE; (with the retention rate calculated as day7_active divided by day0_active from seven days earlier, of course) https://analytics.itunes.apple.com/#/retention?app=324715238 hive (wmf)> SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS Android_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'Android'; hive (wmf)> SELECT CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, unique_count AS iOS_DAU FROM wmf.mobile_apps_uniques_daily WHERE platform = 'iOS'; -- Tilman Bayer Senior Analyst Wikimedia Foundation IRC (Freenode): HaeB

1 0

How might interactive maps be used on Wikipedia?
by Chris Koerner 02 May '16

02 May '16

Hello, The Maps team at the Wikimedia Foundation is getting closer to make it possible to add interactive maps <https://www.mediawiki.org/wiki/Maps> to Wikipedia. If you've ever used services like Google Maps or Mapquest you may be familiar with interactive maps. We’d like to invite editors to have a conversation on how these maps might be used within articles. We've put together information on how these maps and their style works from a technical perspective <https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use> – where the data comes from, how maps are styled, how to add an interactive map, and a few example use cases. In particular we would like to focus the discussion around three key questions (open discussion outside these questions is welcome too). * What types of articles would use interactive maps? * How do these articles differ in their requirements? * Are there any classes of articles whose map styling requirement is fundamentally in conflict with other article classes, thus requiring multiple styles? If you are interested, please visit https://www.mediawiki.org/wiki/Maps/Conversation_about_interactive_map_use to learn more and get involved. -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Migration of browsertests* Jenkins jobs to selenium* jobs
by Željko Filipin 02 May '16

02 May '16

Hi, We have a set of Jenkins jobs that run daily and execute Ruby+Selenium tests. Recently, the way we run those jobs reached a point where we had to do some serious refactoring[0]. The old jobs were named browsertests*[1], the new are named selenium*[2]. Changes: #1 The creation and deletion of jobs have been made simpler. Each repository now has only a single job defined in Jenkins. It is a multi configuration job that spawns one or more child job based on a configuration in each repository: `tests/browser/ci.yml`[3]. The main job will then spawn child jobs based on its content. #2 Jobs execute `selenium` Rake target (`bundle exec rake selenium`). It is defined in the Rakefile of each repository and load the Rake task from mediawiki_selenium Ruby gem version 1.7.0[4]. What does it mean for you? At this point, no action in needed. All required changes have already been made. When the selenium* job passed for a repository, I have already deleted the browsertests* legacy one. There is still a few repositories (Flow, MobileFrontend, MultimediaViewer, Wikidata) that need to be moved, but we are working on that. If you have any questions, let me know. Željko -- 0: https://phabricator.wikimedia.org/T128190 1: https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ 2: https://integration.wikimedia.org/ci/view/Selenium/ 3: https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#ci.yml 4: https://www.mediawiki.org/wiki/Continuous_integration/Entry_points#Rake

1 0

Weekly update #2
by Amir Ladsgroup 02 May '16

02 May '16

The second weekly update on the Revision Scoring project. *New developments* - ORES has graphite dashboard now [1,2] - Deploying new campaigns and testing Wikilabels got easier [3,4] - Revscoring feature extraction got about 13% faster [5] - We deployed new versions of ORES and Wikilabels [6,7] - Wikidata ScoredRevision gadget had a serious issue, it got fixed [8] *Progress in supporting new languages* - Wikidata damaging and goodfaith models are built and deployed [9,10,11] - Dutch damaging and goodfaith models are built and deployed [12] - We are working on langauge utilities of Tamil [13] *Active Labeling campaigns* - Edit quality (damaging and good faith) - Wikipedias: Arabic, Azerbaijani, German, French, Hebrew, Hungarian, Indonesian, Italian, Japanese, Norweigian, Persian (v2), Polish, Spanish, Ukranian, Urdu, Vietnamese - Edit type - English Wikipedia 1. 1. https://grafana.wikimedia.org/dashboard/db/ores 1. 2. https://phabricator.wikimedia.org/T127594 1. 3. https://phabricator.wikimedia.org/T133557 1. 4. https://phabricator.wikimedia.org/T102336 1. 5. https://github.com/wiki-ai/revscoring/pull/268 1. 6. https://phabricator.wikimedia.org/T134032 1. 7. https://phabricator.wikimedia.org/T134174 2. 8. https://phabricator.wikimedia.org/T133903 1. 9. https://phabricator.wikimedia.org/T130274 1. 10. https://phabricator.wikimedia.org/T130301 1. 11. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008641.html 1. 12. https://phabricator.wikimedia.org/T133563 13. https://phabricator.wikimedia.org/T134105 Sincerely, The Revision Scoring team.

1 0

[GSoC 2016] Community Bonding
by Alangi Derick 02 May '16

02 May '16

Hi everyone I am Alangi Derick and also d3r1ck on the IRC. I was selected in the GSoC 2016 program and was opportuned to work on the project titled "Integration of IFTTT support for Wikidata" and I am very happy to be the first African GSoCer in Wikimedia Foundation. This is indeed a priviledge and I wish to thank everyone that guided me in this movement to attain this level. I specially want to thank my mentors (Stephen Laporte, Marius Hoch, Lydia Pintscher, Benedikt Seidl and Sam Tarling) for helping me before and within this program. Also I won't forget to thank some very key org admins for their great help when I just joined the movement (Quim Gil, Brian Wolf, Andre Klapper), I really thank you all for your help. I will love to keep working with you all so that I can better shape my future in Wikimedia and also hope to see you all in person when the time comes :) and also to all Wikimedia Developers, I shall keep in touch and we all shall make the movement better in the future. Cheers!!!! Regards Alangi Derick Ndimnain

2 1

Re: [Wikitech-l] [Wikitech-ambassadors] Tech News (2016, week 12)
by Alex Monk 01 May '16

01 May '16

Reminder: The IRC breaking change is happening tomorrow! On 21 March 2016 at 16:07, Johan Jönsson <jjonsson(a)wikimedia.org> wrote: > Hi, > > The latest technical newsletter is now available at < > https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News/2016/12>. > Below is the English version. > > You can help write the next newsletter: whenever you see information about > Wikimedia technology that you think should be distributed more broadly, you > can add it to the next newsletter at < > https://meta.wikimedia.org/wiki/Tech/News/Next>. > > More information on how to contribute is available at < > https://meta.wikimedia.org/wiki/Tech/News#contribute>. You can also > contact me directly. > > As always, feedback (on- or off-list) is appreciated and encouraged. > > ————————————————————————————— > > Other languages: > čeština <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/cs> • ‎ > Deutsch <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/de> • ‎ > Ελληνικά <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/el> • ‎ > English • ‎español > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/es> • ‎français > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/fr> • ‎עברית > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/he> • ‎italiano > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/it> • ‎日本語 > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/ja> • ‎Ripoarisch > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/ksh> • ‎norsk bokmål > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/nb> • ‎português > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/pt> • ‎русский > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/ru> • ‎svenska > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/sv> • ‎українська > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/uk> • ‎Tiếng Việt > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/vi> • ‎中文 > <https://meta.m.wikimedia.org/wiki/Tech/News/2016/12/zh> > > Latest *tech news > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News>* from the > Wikimedia technical community. Please tell other users about these changes. > Not all changes will affect you. Translations > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News/2016/12> are > available. > > *Problems* > > - Wikimedia has a new backup data center in Dallas. The Wikimedia > Technology department planned to test it this week. This will take place > the week of April 18 instead. You can read more about the server work > <https://meta.wikimedia.org/wiki/Tech/Server_switch_2016>. There will > be more notifications and updates about this. > - In Firefox, the visual editor would sometimes just jump down to the > bottom of the page when you double-clicked on a table cell. This has now > been fixed. [1] <https://phabricator.wikimedia.org/T128282> > > *Changes this week* > > - It will now be easier to list IP addresses that don't have to solve > CAPTCHAs <https://en.wikipedia.org/wiki/en:CAPTCHA>. [2] > <https://phabricator.wikimedia.org/T103122> > - You will now be able to mark Echo notifications as unread. [3] > <https://phabricator.wikimedia.org/T73564> > - [image: Octicons-sync.svg] The new version > <https://www.mediawiki.org/wiki/MediaWiki_1.27/wmf.18> of MediaWiki > will be on test wikis and MediaWiki.org from 21 March. It will be on > non-Wikipedia wikis and some Wikipedias from 22 March. It will be on all > wikis from 23 March (calendar > <https://www.mediawiki.org/wiki/MediaWiki_1.27/Roadmap>). > > *Meetings* > > - [image: Octicons-sync.svg] You can join the next meeting with the > VisualEditor team. During the meeting, you can tell developers which bugs > you think are the most important. The meeting will be on 22 March at > 19:00 (UTC) > <http://www.timeanddate.com/worldclock/fixedtime.html?hour=19&min=00&sec=0&d…>. > See how to join > <https://www.mediawiki.org/wiki/VisualEditor/Weekly_triage_meetings>. > > *Future changes* > > - [image: Octicons-tools.svg] URLs in the recent changes IRC feed will > no longer be rewritten to unencrypted HTTP. This could be a breaking change > for bots dependent on the IRC feed. This will happen on May 2. [4] > <https://phabricator.wikimedia.org/T122933> > > *Tech news > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News> prepared > by tech ambassadors > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/Ambassadors> and > posted by bot > <https://meta.wikimedia.org/wiki/Special:MyLanguage/User:MediaWiki_message_d…> • Contribute > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News#contribute> • Translate > <https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/News/2016/12> • Get > help <https://meta.wikimedia.org/wiki/Tech> • Give feedback > <https://meta.wikimedia.org/wiki/Talk:Tech/News> • Subscribe or unsubscribe > <https://meta.wikimedia.org/wiki/Global_message_delivery/Targets/Tech_ambass…>.* > > > //Johan Jönsson > > > > _______________________________________________ > Wikitech-ambassadors mailing list > Wikitech-ambassadors(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-ambassadors > >

1 0

Fwd: Developer metrics workshop next to OSCON
by Quim Gil 01 May '16

01 May '16

FYI ---------- Forwarded message ---------- From: Quim Gil <qgil(a)wikimedia.org> Date: Thu, Apr 28, 2016 at 9:37 AM Subject: Developer metrics workshop next to OSCON To: "A mailing list for the Analytics Team at WMF and everybody who has an interest in Wikipedia and analytics." <analytics(a)lists.wikimedia.org>, Manrique Lopez <jsmanrique(a)bitergia.com> Hi, is there anybody in this list planning to attend OSCON (Austin, May 16-19)? http://conferences.oreilly.com/oscon/open-source-us Next to that event there will be a workshop about Software Development Analytics and the new Grimoire toolkit platform, on May 16th (10am - 1pm CDT ): https://www.eventbrite.com/e/software-development-analytics-workshop-ticket… The registration is not free, but the organizers (Bitergia, the developers of http://korma.wmflabs.org/ ) are offering us a couple of invitations. Andre and I are not attending OSCON (we have participated in the FOSDEM edition of this workshop). If you or someone you know is interested, contact me. -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

2 1

Phabricator monthly statistics - 2016-04
by communitymetrics＠wikimedia.org 01 May '16

01 May '16

Hi Community Metrics team, This is your automatic monthly Phabricator statistics mail. Accounts created in (2016-04): 227 Active users (any activity) in (2016-04): 863 Task authors in (2016-04): 493 Users who have closed tasks in (2016-04): 274 Projects which had at least one task moved from one column to another on their workboard in (2016-04): 0 Tasks created in (2016-04): 2663 Tasks closed in (2016-04): 2420 Open and stalled tasks in total: 29610 Median age in days of open tasks by priority: Unbreak now: 20 Needs Triage: 158 High: 262 Normal: 415 Low: 733 Lowest: 544 (How long tasks have been open, not how long they have had that priority) TODO: Numbers which refer to closed tasks might not be correct, as described in https://phabricator.wikimedia.org/T1003 . Yours sincerely, Fab Rick Aytor (via community_metrics.sh on iridium at Sun May 1 00:00:10 UTC 2016)

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l May 2016