Wikitech-l May 2016

wikitech-l@lists.wikimedia.org

113 participants
123 discussions

Discovery Weekly Update for the week starting 2016-05-09
by Chris Koerner 14 May '16

14 May '16

Hello, Here is this week's update from the Discovery department. * Closed out the latest A/B test for adding descriptive text [1] on the Portal on May 10 * Started a short banner survey [2] on the wikipedia.org portal page on May 11 * Stalled on updating wikipedia.org portal [3] stats but hope to be resolved soon (with DBA assistance) * Created a testing dev workflow page [4] for wikipedia.org portal site A/B testing * Released Discernatron! [5]. You can help make search better by grading search results. * Presented geo boosted full text search [6], geo queries in WDQS [7], and updates to mapping for wikivoyage [8] in the monthly CREDIT showcase [9]. [1] https://phabricator.wikimedia.org/T131238 [2] https://phabricator.wikimedia.org/T134512 [3] https://phabricator.wikimedia.org/T128546 [4] https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing/dev_process [5] https://discernatron.wmflabs.org/ [6] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=18m45s [7] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=21m16s [8] https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=27m31s [9] https://www.mediawiki.org/wiki/File:CREDIT_-_May_2016.webm ---- Feedback and suggestions on this weekly update are welcome! The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Insecure (non-HTTPS) API Requests to become unsupported starting 2016-06-12
by Brandon Black 14 May '16

14 May '16

TL;DR: ---- * All access to Wikimedia production sites/APIs should use https:// URLs, not http:// -- your bot/tool will break in the near future if it does not! * 2016-06-12 - insecure access is unsupported; starting on this date we plan to break (deny with 403) 10% of all insecure requests randomly as a wake-up call. * 2016-07-12 - we plan to break all insecure requests. ---- Hi all, As you may remember, all production Wikimedia wikis switched to HTTPS-only for all canonical domainnames nearly a year ago: https://blog.wikimedia.org/2015/06/12/securing-wikimedia-sites-with-https/ Since way back then, we've been forcing insecure HTTP requests to our canonical domains over to HTTPS by using redirects and Strict-Transport-Security, which is effective for the vast majority of access from humans using browsers and apps. In the time since, we've been chasing down various corner-case issues where loopholes may arise in our HTTPS standards and enforcement. One of the most-difficult loopholes to close has been the "Insecure POST" loophole, which is discussed in our ticket system here: https://phabricator.wikimedia.org/T105794 . To briefly recap the "Insecure POST" issue: * Most of our humans using browser UAs are not affected by it. They start out doing GET traffic to our sites, their GETs get redirected to HTTPS if necessary, and then any POSTs issued by their browser use protocol-relative URIs which are also HTTPS. * However, many automated/code UAs (bots, tools, etc) access the APIs using initial POST requests to hardcoded service URLs using HTTP (rather than HTTPS). * For all of the code/library UAs out there in the world, there is no universally-compatible way to redirect them to HTTPS. There are different ways that work for some UAs, but many UAs used for APIs don't handle redirects at all. * Regardless of the above, even if we could reliably redirect POST requests, that doesn't fix the security problem like it does with GET. The private data has already been leaked in the initial insecure request before we have a chance to redirect it. If we did some kind of redirect first, we'd still just be putting off the inevitable future date where we have to go through a breaking transition to secure the data. Basically, we're left with no good way to upgrade these insecure requests without breaking them. The only way it gets fixed is if all of our API clients in the world use explicit https:// URLs for Wikimedia sites in all of their code and configuration, and the only way we can really force them to do so is to break insecure POST requests by returning a 403 error to tools that don't. Back in July 2015, I began making some efforts to statistically sample the User-Agent fields of clients doing "Insecure POST" and tracking down the most-prominent offenders. We were able to find and fix many clients along the way since. A few months ago Bryan Davis got us further when he committed a MediaWiki core change to let our sites directly warn offending clients. I believe that went live on Jan 29th of this year ( https://gerrit.wikimedia.org/r/#/c/266958 ). It allows insecure POSTs to still succeed, but sends the clients a standard warning that says "HTTP used when HTTPS was expected". This actually broke some older clients that weren't prepared to handle warnings at all, and caused several clients to upgrade. We've been logging offending UAs and accounts which trigger the warning via EventLogging since then, but after the initial impact the rate flattened out again; clients and/or users that didn't notice the warning fairly quickly likely never will. Many of the remaining UAs we see in logs are simply un-updated. For example, https://github.com/mwclient/mwclient switched to HTTPS-by-default in 0.8.0, released in early January, but we're still getting lots of insecure POST from older mwclient versions installed out there in the world. Even in cases where the code is up to date and supports HTTPS properly, bot/tool configurations may still have hardcoded http:// site config URLs. We're basically out of "soft" ways to finish up this part of the HTTPS transition, and we've stalled long enough on this. ** 2016-06-12 is the selected support cutoff date ** After this date, insecure HTTP POST requests to our sites are officially unsupported. This date is: * A year to day after the public announcement that our sites are HTTPS only * ~ 11 months after we began manually tracking down top offenders and getting them fixed * ~ 4 months after we began sending warning messages in the response to all insecure POST requests to the MW APIs * ~ 1 month after this email itself On the support cutoff date, we’ll begin emitting a “403 Insecure POST Forbidden - use HTTPS” failure for 10% of all insecure POST traffic (randomly-selected). Some clients will retry around this, and hopefully the intermittent errors will raise awareness more-strongly than the API warning message and this email did. A month later (two months out from this email) on 2016-07-12 we plan to break insecure access completely (all insecure requests get the 403 response). In the meantime, we'll be trying to track down offending bots/tools from our logs and trying to contact owners who haven't seen these announcements. Our Community team will be helping us communicate this message more-directly to affected Bot accounts as well. Thank you all for your help during this transition! -- Brandon Black Sr Operations Engineer Wikimedia Foundation

1 0

Mobile Content Service: naming of new endpoints for feeds
by Bernd Sitzmann 14 May '16

14 May '16

We plan to add more RESTBase endpoints to support the new "Explore feeds" feature in the apps. The currently proposed names are listed in [1]. It introduces a new top-level hierarchy, called "project". If you have issues with the current proposal or ideas to improve them please comment on the Phab ticket by Thursday, May 19, 2016. [1] https://phabricator.wikimedia.org/T132597 Thank you, Bernd Sitzmann Android app & Mobile Content Service

1 0

Re: [Wikitech-l] [Engineering] 1.28.0-wmf.0 on hold for now
by Chad Horohoe 13 May '16

13 May '16

Thanks Roan & Brad! We'll get back on track with wmf.1 deployments today :D -Chad On Wed, May 11, 2016 at 11:08 PM, Roan Kattouw <rkattouw(a)wikimedia.org> wrote: > TLDR: the bug is fixed and the errors have stopped. > > I started working around this train hold by backporting the entire Echo > extension from wmf1 to wmf23, assuming that the bug would be in MW core and > updating Echo wouldn't affect it. Right after I deployed that, these errors > started being thrown by wmf1 too. > > It turned out that one of the Echo changes I backported stores the integer > -1 in redis under some circumstances. RedisBagOStuff treats integers > specially, in order to make incr() work: it stores them as plain numbers > instead of PHP-serialized data. But when retrieving this value, the code > didn't recognize -1 as a plain number because it didn't consist solely of > digits ('-' is not a digit), so it thought it was PHP-serialized data and > passed it to unserialize(), which caused the error. Apparently no one had > ever tried to store a negative integer in redis (!) until my Echo change > exposed the bug. > > Brad did all the hard work, diagnosing this and writing up a fix on > Phabricator. I turned that into a patch and deployed it about an hour ago. > There haven't been any more errors since then. > > On Wed, May 11, 2016 at 1:59 PM, Chad Horohoe <chorohoe(a)wikimedia.org> > wrote: > >> Hi, >> >> When we deployed the first 1.28 release to the cluster yesterday, we got >> a new error[0] relating to >> unserialization of redis data. It's pretty spammy already, so I'm >> paranoid about deploying wider until >> we figure out why. Deploying some debugging work soon so we can figure >> out what's going on. >> >> If you've got any information you think would help, please chime in on >> the bug. >> >> -Chad >> >> [0] https://phabricator.wikimedia.org/T134923 >> >> _______________________________________________ >> Engineering mailing list >> Engineering(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/engineering >> >> >

3 3

Today's CREDIT showcase - video
by Adam Baso 13 May '16

13 May '16

Here's video from today's 12-May-2016 CREDIT showcase <https://www.mediawiki.org/wiki/CREDIT_showcase>. https://www.youtube.com/watch?v=GwTDDWjxoek&feature=youtu.be&t=7 https://commons.wikimedia.org/wiki/File:CREDIT_-_May_2016.webm == Contents == * Derk-Jan Hartman: Video.js progress * Dmitry Brant: Wikidata infoboxes in Android app * Joaquin Hernandez: Vicky chat bot * Baha: mobile printing for offline reading * Monte: "smart random" content service endpoint * Erik: Geo boosting search queries * Julien: Maps for Wikivoyage Thanks! -Adam

2 1

PyWikibot reviewers needed!
by Jon Robson 13 May '16

13 May '16

I'm keen for us give attention to people who have open patchsets that they are looking for review. We had the first code review office hours today and I hope in the long term during this hour we can work on reviewing patchsets from this search: https://gerrit.wikimedia.org/r/#/q/(+label:Code-Review%253D0+OR+NOT+(label:… Pywikibot has 145 open patches on Gerrit. You can see them all here: https://gerrit.wikimedia.org/r/#/q/status:open+project:pywikibot/core+(+lab… Can someone who is familiar with this codebase commit to helping getting all these reviewed?

2 1

Phabricator Differential lackings
by Krinkle 13 May '16

13 May '16

I apologise if some of the below does in fact exist, I'd gladly find out! There are three major parts of my workflow as project maintainer and contributor that I find lacking in Differential right now and are strong reason for me to discourage anyone from migrating to it. In addition, projects that have already migrated are essentially non-existent and blind to my workflow as a result. I've tried but even if I accept loss in productivity, I can't even find a workaround for these. 1. Casual notifications about new patches and merged commits (IRC notifications). - https://phabricator.wikimedia.org/T116330 2. Discover pending patches. While Diffusion is a fine repo browser, it appears to lack any link back to Differential. Even when manually going to Differential, it doesn't appear to be any concept of "repo". It only has a global concept of "diff" and "reviewer" (e.g. you can list diffs for review, and your own open diffs). Unlike Maniphest (which has a concept of projects that have their own pages with useful outgoing links). There is no list of projects with a link to see a project's patches. This unlike Gerrit or GitHub where projects are linked to searches for open patches or open pull-requests. This makes it very non-transparent for potential consumers of our code, and casual contributors to find out about open patches. This is an important indicator for developers to determine the health of a project. It's also an easy way-in to for new reviewers, and an important interface for project maintainers to easily list open patches from time to time. Without this, I expect our code review habits to become even worse than they already are. https://phabricator.wikimedia.org/differential/ (no projects listed) -> https://phabricator.wikimedia.org/differential/query/all/ (no repos named alongside the diffs) -> https://phabricator.wikimedia.org/D229 (links to diffusion) -> https://phabricator.wikimedia.org/diffusion/GOJR/ (no link to query to differential). If I see this correctly, there is simply no way to naturally get to a list of open patches of a project. 3. Notification about new patches and merged commits. There is appears to be no way to subscribe to a repo (e.g. as a maintainer of that repo) so that I may be notified of new diffs and/or landed commits. This is worse due to #2, which would've been a workaround (albeit a costly one, due to pull:N vs push:1). In addition to these workflow concerns, there is also Continuous integration of course. But that's a separate issue. I'm bringing up these concerns now because contrary to what I expected, Differential is being adopted by quite a few repos now despite it being premature. -- Krinkle

3 3

12-May-2016 CREDIT, going back to Hangouts on Air/YouTube
by Adam Baso 12 May '16

12 May '16

Hi all, The next CREDIT showcase will be Thursday, 12-May-2016 at 1800 UTC (1100 SF). https://www.mediawiki.org/wiki/CREDIT_showcase For this one we'll use Hangouts on Air for presenters, and the customary YouTube stream for viewers. See you next month! -Adam

2 2

Issue with composer-merge-plugin, composer install, MW 1.25.5
by James Montalvo 12 May '16

12 May '16

I'm getting the following error when running `composer install` after cloning MW core and checking out 1.25.5. I've gotten the error on two separate systems: 1. Windows with PHP 5.4 and Composer version 1.2-dev 2. CentOS 7 with PHP 5.6 and Composer version 1.1.0 I did NOT get the error on the same Windows machine prior to running `composer self-update`. It had Composer version 1.0-dev (dbdd4978a7dd4cd29d0d7dfc912aab404245b736) prior to update. *** Error message *** Loading composer repositories with package information Updating dependencies (including require-dev) Your requirements could not be resolved to an installable set of packages. Problem 1 - Installation request for wikimedia/composer-merge-plugin 1.0.0 -> satisfiable by wikimedia/composer-merge-plugin[v1.0.0]. - wikimedia/composer-merge-plugin v1.0.0 requires composer-plugin-api 1.0.0 -> no matching package found. Potential causes: - A typo in the package name - The package is not available in a stable-enough version according to your minimum-stability setting see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details. Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems. *** End error message *** Thanks, James

2 1

REL1_27 branches up
by Chad 12 May '16

12 May '16

Hi, Sorry for the delay, I've had a few too many things to get done this week so I haven't communicated as clearly as I should've. The release branches (REL1_27) have been created for MW core, vendor, all extensions and all skins. MediaWiki core is now on 1.28.0-alpha. It's time to start testing out the release branch and make sure everything is good and polished. You can do this on your vagrant by navigating to the mediawiki/ directory and checking out the branches. If you find a problem in the release, please file a task in Phabricator or submit a patch. Minor reminder: please don't start landing large breaking changes on master now that we've swapped...it makes porting patches between branches way more difficult than they need to be :) Thanks and have a great week! -Chad

6 8

← Newer
1
...
5
6
7
8
9
10
11
12
13
Older →

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l May 2016