Wikitech-l August 2016

wikitech-l@lists.wikimedia.org

78 participants
100 discussions

29 Aug '16

Can we get a repo for each major release that contains core, skins, and all extensions in a single checkout? Right now I am updating and with all of the sub modules, I am running into issues where I cant set a branch on the extensions/ repo for REL 1_27 and get all extensions as of that branching, Instead I am forced to go thru one by one and manually set it, if and only if a given extension has said branch set, otherwise I am out of luck. Getting everything together would cause a larger checkout, but keeps things together for those who want to pick and choose.

4 5

New models coming to ORES & notes
by Aaron Halfaker 29 Aug '16

29 Aug '16

Hey folks, We've been working on generating some updated models for ORES. These models will behave slightly differently from the models that we currently have deployed. This is a natural artifact of retraining the models on the *exact same data* again because of some random properties of the learning algorithms. So, for the most part, this should be a non-issue for any tools that use ORES. However, I wanted to take this opportunity to highlight some of the facilities ORES provides to help automatically detect and adjust for these types of changes. *== Versions ==* ORES provides information about all of the models. This information includes a model version number. If you are caching ORES scores locally, we recommend invalidating old scores whenever this model number changes. For example, https://ores.wikimedia.org/v2/scores/enwiki/damaging/12345678 currently returns { "scores": { "enwiki": { "damaging": { "scores": { "12345678": { "prediction": false, "probability": { "false": 0.7141333465390294, "true": 0.28586665346097057 } } }, "version": "0.1.1" } } } } This score was generated with the "0.1.1" version of the model. But once we deploy the new models, the same request will return: { "scores": { "enwiki": { "damaging": { "scores": { "12345678": { "prediction": false, "probability": { "false": 0.8204647324045306, "true": 0.17953526759546945 } } }, "version": "0.1.2" } } } } Note that the version number changes to "0.1.2" and the probabilities change slightly. In this case, we're essentially re-training the same model in a similar way, so we increment the "patch" number. However, we're switching modeling strategies for the article quality models (enwiki-wp10, frwiki-wp10 & ruwiki-wp10), so those versions increment the minor version from "0.3.2" to "0.4.0". You may see more substantial changes in prediction probabilities with those models, but a quick spot-checking suggests that the changes are not substantial. *== Test statistics and threshholding ==* So, many tools that use our edit quality models (reverted, damaging and goodfaith) will set threshholds for flagging edits for review. In order to support these tools, we produce test statistics that suggest useful thresholds. https://ores.wmflabs.org/v2/scores/enwiki/damaging/?model_info=test_stats produces: ... "filter_rate_at_recall(min_recall=0.75)": { "filter_rate": 0.869, "recall": 0.752, "threshold": 0.492 }, "filter_rate_at_recall(min_recall=0.9)": { "filter_rate": 0.753, "recall": 0.902, "threshold": 0.173 }, ... These two statistics show useful thresholds for detecting damaging edits. E.g. if you want to be sure that you catch nearly all vandalism (and are OK with a higher false-positive rate), set the threshold at 0.173, but if you'd like to catch most vandalism with almost no false-positives, set the threshold at 0.492. These fields can be read automatically by tools so that they do not need to be manually updated every time that we deploy a new model. Let me know if you have any questions and happy hacking! -Aaron

2 2

Re: [Wikitech-l] [Wlm-global-org] JetBrains IDE licenses for MW developers - php, js/node, python, C/C++, puppet, ruby, ...
by Ilya Korniyko 28 Aug '16

28 Aug '16

Well I could use If I ever get the subscription. So far I sent several emais but got no response. On Fri, Aug 5, 2016 at 1:34 PM, Florian Schmidt < florian.schmidt.welzow(a)t-online.de> wrote: > We've an all products Plan, so all IDEA Desktop products are included, so > I'm pretty sure, you can use our existing subscriptions. > > Best, > Florian > > _______________________________________________ > Wlm-global-org mailing list > Wlm-global-org(a)lists.wmnederland.nl > https://lists.wmnederland.nl/mailman/listinfo/wlm-global-org > >

2 1

Discovery Weekly Update for the week starting 2016-08-22
by Chris Koerner 26 Aug '16

26 Aug '16

Happy Friday, Here is this week's update from the Discovery department. It's a big one, so grab your favorite drink and settle in. == Events and News == * Interactive team plans to deploy <maplink> support to all Wikipedia next week. T144062 [1] == Other Noteworthy Stuff == * Updated the Wikipedia.org Portal page with new information on quarterly goals and recent production releases [2] [3] * Updated the Help:CirrusSearch documentation to reflect the great work Cspiral did in organizing the information in a more approachable and logical way. Kept existing translation tags in place and have asked for review. This is in preparation for this small fix to the "Help" link on every * * * * * Wikimedia Special:Search page. [4] [5] * Analysis: completed a retro of how the wikipedia.org portal page has evolved [6] * Analysis: completed a review of why portal pageviews increased in June/July 2016 (phab ticket) [7] [8] * Analysis: updated pageview counting on the portal dashboard (dashboard page showing new documented changes) [9] [10] * Analysis: added an easy to use page to show metrics year over year and month over month (dashboard page) [11] [12] * Analysis: added a portal dashboard page to show clicks/users by language (dashboard page) [13] [14] * Analysis: updated dashboard to take into account a new schema for mobile app events (dashboard page) [15] [16] * Analysis: added DuckDuckGo to the search engine referral dashboard page (dashboard page) [17] [18] * Portal: added several new small visual enhancements and optimizations for wikipedia.org along with an article stats and translations updates for code review [19] * Portal: Clean up project portals (community involvement) [20] * Search: deployed ascii folding / stemming re-ordering changes [21] * Search: disable an old search schema since a new one has already been deployed [22] * Search: implement an updated full text search query [23] * Search: Refactor GeoData elasticsearch mapping using the new search field definitions from core [24] * Search: update wildcard queries for BM25 [25] * Search: Centralize document id and namespace generation [26] * Search: Enable BM25 by default in cirrus and evaluate impact [27] * Search: Add a "reverse" suggestion field [28] * Interactive: Large number of small maps usability fixes T144008, T143883, T137357, T143179, T144001 [29] [30] [31] [32] [33] * Interactive: Added a box frame and captioning to <mapframe> (similar to framed images) [34] == Did you know? == * Friday, August 26th is National Dog Day - go pet a pooch! [35] [1] https://phabricator.wikimedia.org/T144062 [2] https://www.mediawiki.org/wiki/Wikipedia.org_Portal#Current_Quarter_Goals [3] https://www.mediawiki.org/wiki/Wikipedia.org_Portal#Portal_Updates:_April_-… [4] https://www.mediawiki.org/wiki/Help:CirrusSearch [5] https://phabricator.wikimedia.org/T141003 [6] https://commons.wikimedia.org/wiki/File:Impact_of_Wikipedia.org_Portal_Chan… [7] https://github.com/wikimedia-research/Discovery-Research-Portal/blob/master… [8] https://phabricator.wikimedia.org/T143045 [9] https://phabricator.wikimedia.org/T143064 [10] http://discovery-beta.wmflabs.org/portal/#pageview_tab [11] https://phabricator.wikimedia.org/T124098 [12] http://discovery.wmflabs.org/metrics/#monthly_metrics [13] https://phabricator.wikimedia.org/T140816 [14] http://discovery.wmflabs.org/portal/#languages_visited [15] https://phabricator.wikimedia.org/T143447 [16] http://discovery.wmflabs.org/metrics/#app_events [17] https://phabricator.wikimedia.org/T143287 [18] http://discovery.wmflabs.org/external/#traffic_by_engine [19] https://gerrit.wikimedia.org/r/306241 [20] https://phabricator.wikimedia.org/T135929 [21] https://phabricator.wikimedia.org/T142721 [22] https://phabricator.wikimedia.org/T130321 [23] https://phabricator.wikimedia.org/T128073 [24] https://phabricator.wikimedia.org/T139021 [25] https://phabricator.wikimedia.org/T143537 [26] https://phabricator.wikimedia.org/T139496 [27] https://phabricator.wikimedia.org/T139576 [28] https://phabricator.wikimedia.org/T107006 [29] https://phabricator.wikimedia.org/T144008 [30] https://phabricator.wikimedia.org/T143883 [31] https://phabricator.wikimedia.org/T137357 [32] https://phabricator.wikimedia.org/T143179 [33] https://phabricator.wikimedia.org/T144001 [34] https://phabricator.wikimedia.org/T128913 [35] https://www.google.com/#hl=en&q=national+dog+day+images ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Re: [Wikitech-l] [WikimediaMobile] Mobile site is now lazy loading images
by Brion Vibber 26 Aug '16

26 Aug '16

Awesome! This will also make it easier to experiment with alternate delivery formats: WebP, direct SVG usage, etc that depend on knowing whether there's client-side support. I'll fire up some more research... :D -- brion On Tue, Aug 23, 2016 at 2:20 PM, Jon Robson <jrobson(a)wikimedia.org> wrote: > FYI after much experimentation, research and testing the mobile site has > been lazy loading images [1] since Thursday 18th August. This means if you > do not see an image you will not download it. We have taken care to ensure > users without JavaScript can still view images and that most users will > barely notice the difference. > > We are currently crunching the data this change has made and we plan to > write a blog post to reporting the results. > > In our experiments on Japanese Wikipedia we saw a drop in image bytes per > page view by 54% On the Japanese Japan article bytes shipped to users > dropped from 1.443 MB to 142 kB. > > This is pretty huge since bytes equate to money [3] and we expect this to > be significant on wikis where mobile data is more expensive. In a nutshell > Wikipedia mobile is cheaper. > > As I said blog post to follow once we have more information, but please > report any bugs you are seeing with the implementation (we have already > found a few thanks to our community of editors). > > ~Jon > > [1] https://www.mediawiki.org/wiki/Reading/Web/Projects/ > Performance/Lazy_loading_images > [2] https://www.mediawiki.org/wiki/Reading/Web/Lazy_loading_ > of_images_on_Japanese_Wikipedia > [3] https://whatdoesmysitecost.com/ > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >

1 0

Re: [Wikitech-l] [WikimediaMobile] Mobile site is now lazy loading images
by Pine W 26 Aug '16

26 Aug '16

Good to hear. Does this also mean that pages load faster? Pine On Tue, Aug 23, 2016 at 2:20 PM, Jon Robson <jrobson(a)wikimedia.org> wrote: > FYI after much experimentation, research and testing the mobile site has > been lazy loading images [1] since Thursday 18th August. This means if you > do not see an image you will not download it. We have taken care to ensure > users without JavaScript can still view images and that most users will > barely notice the difference. > > We are currently crunching the data this change has made and we plan to > write a blog post to reporting the results. > > In our experiments on Japanese Wikipedia we saw a drop in image bytes per > page view by 54% On the Japanese Japan article bytes shipped to users > dropped from 1.443 MB to 142 kB. > > This is pretty huge since bytes equate to money [3] and we expect this to > be significant on wikis where mobile data is more expensive. In a nutshell > Wikipedia mobile is cheaper. > > As I said blog post to follow once we have more information, but please > report any bugs you are seeing with the implementation (we have already > found a few thanks to our community of editors). > > ~Jon > > [1] https://www.mediawiki.org/wiki/Reading/Web/Projects/ > Performance/Lazy_loading_images > [2] https://www.mediawiki.org/wiki/Reading/Web/Lazy_loading_ > of_images_on_Japanese_Wikipedia > [3] https://whatdoesmysitecost.com/ > > > > _______________________________________________ > Mobile-l mailing list > Mobile-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > >

2 2

MediaWiki Language Extension Bundle 2016.08 release
by Kartik Mistry 25 Aug '16

25 Aug '16

Hello all, I would like to announce the release of MediaWiki Language Extension Bundle 2016.08. This bundle is The bundle is compatible with MediaWiki 1.26 and 1.27 or above and requires PHP 5.5.9 or above. Next MLEB is expected to be released in 3 months. If there are major changes or important bug fixes, we will do intermediate release. Please give us your feedback at [[Talk:MLEB|https://www.mediawiki.org/wiki/Talk:MLEB]]. * Download: https://translatewiki.net/mleb/MediaWikiLanguageExtensionBundle-2016.08.tar… * sha256sum: aba09ead5f2c646e5960ce2aecc36866ee092079f51598d0aab461c7655410da Quick links: * Installation instructions are at: https://www.mediawiki.org/wiki/MLEB * Announcements of new releases will be posted to a mailing list: https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n * Report bugs to: https://phabricator.wikimedia.org/ * Talk with us at: #mediawiki-i18n @ Freenode Release notes for each extension are below. -- Kartik Mistry == Highlights and upgrade notes == * Translate extension now requires Elastica extension for ElasticSearch support. Wikimedia search plugin needs to be installed for ElasticSearch 2.x series. * Special:Translate now has a summary field. * Special:Translate is much more robust thanks to improved error handling and to many small bugs having been fixed. * Page translation is much more robust. * The sandbox feature of Translate extension requires MediaWiki 1.27 or newer. * Translate's export functionality is now on separate special page called Special:ExportTranslations * The style of ULS's dialogs have been updated and polished to be more streamlined. * The default geoip service for Universal Language Selector is changed because the old service is discontinued. * Major update on webfont formats: woff2 format added, eot and svg formats dropped. == Babel == * Ricordisamoa changed language categories to be formatted consistently following the BCP47 standard. == CLDR == * Thiemo Mättig changed "Multiple Languages" and "Unknown Language" to "multiple languages" and "unknown languages" in English. == CleanChanges == * Niklas Laxström fixed a JavaScript error in Special:WatchList. T138702 * Brian Wolff improved the way we invoke JavaScript for click handlers. == LocalisationUpdate == * Maintenance updates only. == Translate == * Kunal Grover's patch to update translation pages upon deletion of translation units was merged with the help of Glaisher. * David Causse made translation memory and translation search work with new ElasticSearch version. * Pratik Lahoti's improvements to the user interface of Special:PageMigration was merged with help of Federico Leva. * Glaisher fulfilled a long requested feature to have summary field on Special:Translate. * Glaisher fixed bugs and improved stability of the page translation feature. * Glaisher added a log subtype selector for the page translation log on Special:Log. * Glaisher fixed false warnings about bad translation in Special:Translate when the source text contained discouraged html or had smileys. * Glaisher moved export functionality from Special:Translate to Special:ExportTranslations. * Niklas Laxström improved message group selector on Special:Translate. * Niklas Laxström improved display of Special:Translate for small screens. * Niklas Laxström fixed flash of unstyled content on many of Translate's special pages. * Niklas Laxström changed JSON file format handler to always output a metadata section. T118205 * Niklas Laxström fixed multiple issues related to statsbar updates on Special:Translate. T62949 * Niklas Laxström improved the refresh-translatable-pages.php maintenance script so that it creates missing source language pages. * Niklas Laxström updated Translate's sandbox feature to use AuthManager. T111486 * Federico Leva made characterEditStats.php to able to scan history further than the recent changes table. T64833 == UniversalLanguageSelector == * Compact language links feature is much improved, and can now be enabled without being a beta feature. * Santhosh Thottingal added support for browser provided language preferences for compact language links. * Lots of styling updates. * Niklas Laxström did cleanups to language name search which might affect search results. * Niklas Laxström fixed style issues for input method listing. * Niklas Laxström updated the default geoip provider. * Fonts: ** Santhosh Thottingal added support for WOFF2 format for web fonts. Niklas Laxström dropped support for EOT and SVG formats. ** Kartik Mistry updated Skiers II font for Gothic language. T124785 * Input Methods: ** Added Khmer NiDA standard Unicode keyboard. ** Added Akan language keyboard. ** Added Enhanced Ol Chiki InScript and Sarjom Baha phonetic keyboards for Santali language. ** Various fixes for Punjabi, Urdu keyboards. -- Kartik Mistry/કાર્તિક મિસ્ત્રી | IRC: kart_ {kartikm, 0x1f1f}.wordpress.com

1 0

Last call on ArchCom-RFC T69223 (Schema change for page content language)
by Rob Lanphier 25 Aug '16

25 Aug '16

Hi everyone Per <https://phabricator.wikimedia.org/T69223#2581292> During E263, Jaime (@jcrespo) put the following choice to us. Should he: a. Apply this change to all wikis b. Apply this change to a subset of wikis, on request c. Not apply this change We seemed to get consensus around "a" (applying this change to all wikis), so that's the decision we're putting in "final comment". ArchCom plans to move this task to "Approved" during E265 (barring any unaddressed objections in the comments on this task). See also: <https://www.mediawiki.org/wiki/ArchComStatus> Rob

1 0

2016-08-24 Scrum of Scrums meeting notes
by Grace Gellerman 24 Aug '16

24 Aug '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-08-24 = 2016-08-24 = == Product == === Reading === ==== Android native app ==== * Current sprint: https://phabricator.wikimedia.org/project/view/2142/ * Next sprint: https://phabricator.wikimedia.org/project/view/2178/ * Summary: Navigation UX overhaul work continues ==== Mobile Content Service (MCS) * no update ==== iOS native app ==== * Current release board: https://phabricator.wikimedia.org/project/board/1736/ ** 5.1 is in regrssion / beta. Due to go to the app store next week * Next board: https://phabricator.wikimedia.org/project/view/2150/ ** 5.2 is in developmentwith expected deployment alongside iOS 10 release in late September *** Adding iOS 10 support (with widgets) *** Dropping iOS 8 support ==== Reading Web ==== * Current sprint: * Shipped lazy loaded images to all wikipedias * Diagnosing hovercards EL * Improving search exp in mobile web * Improvements to related pages and the footer in mobile web * Next sprint: * Related pages improvements * Shipping wikidata descriptions in mobile web to some wikis ==== Reading Infrastructure ==== === Community Tech === * AbuseFilter/AntiSpoof fix (https://phabricator.wikimedia.org/T29987) - deployed and most filters + titleblacklists updated * Numeric sorting ** Deployed numeric sorting to Swedish Wikipedia for further testing, seems to be working ** Will deploy to English Wikipedia next week barring any bug discoveries * Cross-wiki watchlists - making schema changes to CentralAuth * No blockers === Editing === ==== Language ==== * Blocked: none * Blocking: none * Updates: ** Apertium packages for Jessie ready (few waiting for dependencies to be uploaded to WMF repo). ** CX getting updates on template handling soon. ==== Parsing ==== * Still waiting on a security review of the ParserMigration extension. * Thinking about going ahead with https://phabricator.wikimedia.org/T111604 in the coming months (spliting up parser tests into multiple files). If you have thoughts / concerns, please comment on the ticket. * Nothing else significant to report. ==== Collaboration ==== * '''Blocking''': ** Stephane is making progress on the Flow caching rewrite, which is blocking multi-DC. * '''Blocked''': * '''Updates''': ** We have enabled Flow on all user talk pages on kabwiki. ** New feature to allow unwatching Flow boards from an Echo notification. ** Rasterizing Echo icons for email to work around Gmail bug. ** Fixed dot-dot-dot menu on Special:Notifications on mobile ** MediaWiki-Vagrant change to use psysh for Zend, since phpsh is broken == Technology == === Analytics === * AQS (pageview api) cluster is having some performance issues, Luca is back tomorrow and will take a closer look * Getting ready to build the Wikistats 2.0 front-end, evaluating js frameworks, do weigh in if interested * same ongoing work as last week, loading the new cluster, deploying the new event bus schemas, vetting the reconstruction of edit history === Security === * Security reviews this week: ** *ACL configuration for url-downloader.wikimedia.org <http://url-downloader.wikimedia.org> allowing upload.wikimedia.org <http://upload.wikimedia.org>* ** Catching up on reviews from last week * Incident handling * Darian working on hiring tasks with Recruiting === Services === Blocking: none Blocked: * Parsoid move to scap3 blocked by RelEng: *https://phabricator.wikimedia.org/T142990* <https://phabricator.wikimedia.org/T142990> Updates: * Deployed Change Prop with new kafka driver === Discovery === * No blockers * Working on BM25 implementation * Working on multi-wiki indexes * Working on integrating Polestar (http://vega.github.io/polestar/) and WDQS GUI to allow create graphs visually from WDQS results * Deployed portal updates & blog post: https://blog.wikimedia.org/2016/08/16/wikipedia-portal-update/ * Retro of portal evolution: https://commons.wikimedia.org/wiki/File:Impact_of_Wikipedia.org_Portal_Chan… * Reordering of stemming/asciifolding deployed, but no changes pending reindex (https://phabricator.wikimedia.org/T142721 ) * Analysis of asciifolding in French: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Adding_Ascii-Folding… ** TLDR: French more complex than English ;) But positive results. * Asked for community help to update "showing results from" translation: https://phabricator.wikimedia.org/T143355 * Analysis of portal pageview increase: https://github.com/wikimedia-research/Discovery-Research-Portal/blob/master… === RelEng === * Blocking ** Parsoid scap3 deployment: https://phabricator.wikimedia.org/T142990 * Blocked ** (Ops) Contint networking: https://phabricator.wikimedia.org/T140257#2553490 ** (Ops/puppet) https://gerrit.wikimedia.org/r/#/c/300092/ ("contint: tidy Nodepool slaves config history") ** (Ops) Help requested: Upgrade base MW-Vagrant image to Jessie - https://phabricator.wikimedia.org/T136429 *** Outline from bd808: https://phabricator.wikimedia.org/T136429#2572195 *** Ori suggesting Ops support: https://phabricator.wikimedia.org/T136429#2572433 * Updates ** Bugfix release of scap ** European SWAT started Monday! === Fundraising Tech === * More queue work * More dedupe work * Applied security release to payments * Reacting to GeoIP lookup domain removal (3rd party impact) === ArchCom === * https://www.mediawiki.org/wiki/Architecture_committee/Status * Last week: 2016W33: 2016-08-17 (Wednesday) ** ArchCom Planning meeting: [https://phabricator.wikimedia.org/E260 Phab:E260] *** Notes: [https://www.mediawiki.org/wiki/Architecture_committee/2016-08-17 Architecture committee/2016-08-17] ** ArchCom-RFC office hour: [https://phabricator.wikimedia.org/E261 Phab:E261] *** Agenda: [https://phabricator.wikimedia.org/T105652 Revisiting T105652 (content model storage)] * This week: 2016W34: 2016-08-24 (Wednesday) ** ArchCom Planning meeting: [https://phabricator.wikimedia.org/E262 Phab:E262] *** Notes: [https://www.mediawiki.org/wiki/Architecture_committee/2016-08-24 Architecture committee/2016-08-24] ** ArchCom-RFC office hour: [https://phabricator.wikimedia.org/E263 Phab:E263] *** Agenda: [https://phabricator.wikimedia.org/T69223 Schema change for page content language] == Wikidata == * Blockers: none. * Progess on the "structured data for Commons" prototype … ( https://phabricator.wikimedia.org/T140760) * … as well as concept work for automated list generation ( https://twitter.com/simulo/status/766717877047193600). * We wrote a policy about guarantees we give about our public interfaces for data access ( https://www.wikidata.org/wiki/Wikidata:Stable_Interface_Policy). * Experimenting with sorting statements ( https://phabricator.wikimedia.org/T129244).

1 0

ORES review tool default sensitivity changed now
by Amir Ladsgroup 24 Aug '16

24 Aug '16

Hey all, We just deployed a change that changed default sensitivity of ORES review tool from "hard" to "soft" (meaning recall would drop from 0.9 to 0.75 but percentage of false positives drops too). You are still able to change it back in your preferences (Recent changes tab). Please come to us for any issues or questions. Best

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2016