Wikitech-l February 2016

wikitech-l@lists.wikimedia.org

103 participants
107 discussions

Windows Single Sign-On Extension
by François St-Arnaud 14 Feb '16

14 Feb '16

Hello, To enable Single Sign-On to a MediaWiki hosted on IIS in a Windows Domain, the best MediaWiki extension I could find was NTLMActiveDirectory. https://www.mediawiki.org/wiki/Extension:NTLMActiveDirectory However, I had two peeves with this extension: 1) Its name; I'm not doing NTLM, but Negotiate and Kerberos; and 2) Its use of LDAP; feels too much like a wart on Windows! See, I'm sitting on an IIS box on a Windows domain with Integrated Windows Authentication enabled. By the time the MW extension gets hit, IIS has already authenticated the user, so why not just leverage that instead? I therefore used NTLMActiveDirectory as a starting point, but threw out all the LDAP stuff and replaced it with a simple Web call to an IIS-hosted handler to get the AD group membership for the already authenticated user. Of NTLMActiveDirectory, I kept the AD / MW group mapping configuration required for authorization. Personally, I find this solution much simpler and intuitive for AD integration when hosting MW on a Windows/IIS box. Does this make sense to others in the community? Do others feel there was a need for a better AD integration extension? Would others in the community benefit from such an extension? If so, I would be happy to share my work, following instructions found here: https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment Regards, François

3 8

Wikitech maintenance and CI downtime tomorrow, 2015-02-12 16:00 UTC
by Andrew Bogott 12 Feb '16

12 Feb '16

Executive summary: CI tests won't run for a while tomorrow. Also you won't be able to log into Wikitech. This will begin at 16:00 UTC (8:00 AM in California) and may take a couple of hours. Full story: As part of a long-overdue tech-debt payment[0], I'll be migrating all of the Labs project, membership and role data from ldap into mysql[1]. Keystone (the Labs Openstack authentication layer) will be shut off briefly, and subsequently will have incomplete data while things are gradually copied over from Ldap. At the same time, Alex will be rolling out a bunch of OpenStackManager patches[2] to cope with the new reality. I've tested these quite a lot, but no doubt there will be unexpected issues with such a big refactor. Existing, running labs and tools sessions should not be affected. New instance creation will be disabled for part of the window, which will prevent CI from starting new tests. Wikitech login will be disabled, and users will have to login afresh after logins are restored. There is, unfortunately, no user-facing improvement associated with this update. If all goes perfectly, Wikitech will be restored to its original state but be slightly slower. If, after the update, anyone encounters new Wikitech issues (and I emphasize the NEW), please open a phab ticket and inform me immediately. As always, in the worst case scenario you can refer to our doc site of last resort, https://wikitech-static.wikimedia.org -Andrew [0] https://phabricator.wikimedia.org/T115029 [1] https://wikitech.wikimedia.org/wiki/Labs_keystone_roles [2] The patchset begins with https://gerrit.wikimedia.org/r/#/c/252615/

2 3

Looking for information about LUA loading/parsing
by Strainu 12 Feb '16

12 Feb '16

Hi, I plan to try and use a big (~67K entries) table as a "database" in a LUA script, with a basic lookup function. One call of the module would be well within the time and memory limits, but I'm worried for pages that might invoke the module several hundred times, so I did an experiment with only a single entry in the table and the results are encouraging: 1 call: Lua time usage 0.006/10.000 seconds Lua memory usage 513 KB/50 MB 10 calls: Lua time usage 0.010/10.000 seconds Lua memory usage 873 KB/50 MB 100 calls: Lua time usage 0.074/10.000 seconds Lua memory usage 1,35 MB/50 MB There are obviously some optimizations there and I would be curious what they are? Do you use some CoW or other trick to prevent loading that table 100 times? Do you have the loading/parsing process documented somewhere? Thanks, Strainu

2 1

Developer Relations Weekly Summary
by Quim Gil 12 Feb '16

12 Feb '16

Developer Relations focus this week: * 2017 Wikimedia Hackathon: Looking for Proposals.[0] * Review Wikimedia Hackathon 2016 [1] travel sponsorship requests. * Wikimedia applications for Google Summer of Code 2016 [2] and Outreachy Round 12.[3] * Now Developer Relations is a subteam of Technical Collaboration,[4] see our quarterly goals,[5] annual plan [6] (WIP), strategy [7] (WIP), and planning process.[8] There is plenty more at https://www.mediawiki.org/wiki/Developer_Relations/Weekly_summary#2016-02-12 [0] http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/85651 [1] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2016 [2] https://www.mediawiki.org/wiki/Google_Summer_of_Code_2016 [3] https://www.mediawiki.org/wiki/Outreachy/Round_12 [4] https://meta.wikimedia.org/wiki/Technical_Collaboration [5] https://meta.wikimedia.org/wiki/Technical_Collaboration/Goals [6] https://phabricator.wikimedia.org/T124420 [7] https://meta.wikimedia.org/wiki/Technical_Collaboration/Strategy [8] https://www.mediawiki.org/wiki/Developer_Relations/Planning -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

1 0

Prefix search refactoring
by Stas Malyshev 12 Feb '16

12 Feb '16

Hi! In order to make prefix search better, and to bring all variants of prefix search under one roof, we did some refactoring in the search engine implementation, so that various prefix searches now use the same code path and all use the SearchEngine class. The changes are as follows: SearchEngine gets the following new API functions: * public function completionSearch( $search ) - implements prefix completion search, returns SearchSuggestionSet * public function completionSearchWithVariants( $search ) - implements prefix completion search including variants handling, returns SearchSuggestionSet. * public function defaultPrefixSearch( $search ) - basic prefix search without fuzzy matching, etc., to be used in scenarios like special pages search, etc. Returns Title[]. The implementation does not have to implement all three methods differently, they can all use the same code if needed. The default implementation still supports the PrefixSearchBackend hook but we plan to deprecate it, and the CirrusSearch implementation does not use it anymore. Instead, there is a private function, protected function completionSearchBackend( $search ), which implementations (including CirrusSearch) should implement to provide search results. SearchEngine implementations can make use of services provided by the base SearchEngine including: - namespace resolution and normalization. The PrefixSearchExtractNamespace hook is still supported for engines wishing to implement namespace lookup not featured in the standard implementation. - fetching titles for result sets (the implementing engine does not have to fetch titles from DB for suggestions) - result reordering to ensure exact matches are on top - basic prefix search implementation using the database - Special: namespace search implementation == Deprecations == We plan to deprecate the PrefixSearchBackend hook and classes TitlePrefixSearch and StringPrefixSearch. We will keep those classes around for basic search fallback implementation and for old extensions, but no new code should be using these classes, instead they should use SearchEngine APIs described above. Mediawiki code has already been fixed to do that. Extensions implementing search engines should also extend SearchEngine and override the APIs above. CirrusSearch is the example of how to do it. == Show me the code == The patches implementing the refactoring are linked from: https://phabricator.wikimedia.org/T121430 Pretty version of the same: https://www.mediawiki.org/wiki/User:Smalyshev_(WMF)/Suggester If you have questions on this, please contact the Discovery team: https://www.mediawiki.org/wiki/Wikimedia_Discovery#Communications -- Stas Malyshev smalyshev(a)wikimedia.org

1 0

Beta Cluster not updating
by Greg Grossmeier 11 Feb '16

11 Feb '16

All, The Beta Cluster[0] is currently not being updated with new code[1]. That normally happens every ~10 minutes. The issue is that we were not prepared for the php5.3 -> 5.5 transition; the deployment bastion that runs the code update ('scap') is a Precise image. We (Tyler) are currently working on rebuilding it with a Trusty image to solve this problem. I know how frustrating this is for everyone (believe me) and I apologize on behalf of Release Engineering for the second fallout of the php5.3 -> 5.5 transition (the first being yesterday's CI resource exhaustion). I will update the list when Beta Cluster is back to normal. Greg [0] https://www.mediawiki.org/wiki/Beta_Cluster [1] https://phabricator.wikimedia.org/T126573 with a blocker on https://phabricator.wikimedia.org/T126568 -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

1 1

CI response time slow today
by Greg Grossmeier 11 Feb '16

11 Feb '16

Hello all, I want to apologize for the current slowness in our CI stack. Everything is working fine in the sense that nothing is broken. We simply did not adequately prepare for the switch to php5.5 being the new minimum version for MediaWiki. This resulted in a large influx of jobs on a set of our Jenkins executers that was not a large enough pool. Timo and Antoine have created more executers to help mitigate [0]. More explicitly: due to package version availability in Ubuntu, our php5.3 jobs run on our pool of Ubuntu Precis hosts, while the php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts. Both sets of jobs are in our pipeline to migrate to the new Nodepool system that allows burstable availability of executers. Again, apologies for the delays but know that the jobs will complete just not as fast as you are accustomed to. Greg [0] https://phabricator.wikimedia.org/T126423 -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

2 2

[GSoC2016] Project to get started with.
by Arushi Dogra 11 Feb '16

11 Feb '16

Hi everyone, I am a third year Computer Science undergraduate from India. I am interested in contributing to Wikimedia Foundation.Can anyone help me get started? I am currently working on making a Wikipedia Search Engine for ~50 GB of Wikipedia data . The project involves applying various indexing and retrieving techniques to make the engine scalable and efficient. Thank you. Regards, Arushi Dogra

3 2

Download link to properly packed extension
by Stephan Gambke 11 Feb '16

11 Feb '16

It is possible to download extensions in ZIP format from the WMF repo using a link like this: http://git.wikimedia.org/zip/?r=mediawiki/extensions/FooBar.git&h=someTag&f… However, this will produce a package with the extension's files in its root folder. An unsuspecting user will probably simply extract this package into the .../extensions folder with all its files ending up there instead of in the extensions subfolder. The ExtensionDistributor will provide correctly built packages, but it is apparently only working on MW release tags, e.g. REL1_26. Is there any way to download correctly built packages with tags other than the MW relases? Stephan

3 5

Re: [Wikitech-l] 2016-02-10 Scrum of Scrums notes
by Grace Gellerman 10 Feb '16

10 Feb '16

correcting date in subj line On Wed, Feb 10, 2016 at 2:57 PM, Grace Gellerman <ggellerman(a)wikimedia.org> wrote: > https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-02-10 > > = 2015-02-10 = > > == Technology == > > === Analytics === > * '''*Blocking'''*: Nobody we know of > * '''*Blocked'''*: By nothing we're aware of > * '''*Updates*'''*:* > > - ** organizing work to replace limn dashboards with dashiki > > > - ** wikimetrics program global metrics deployed > > > - ** moved wikimetrics from vagrant to docker-compose and it was a > pleasant experience > > > - ** working on cleaning up / organizing analytics files on > dumps.wikimedia.org/other > > > - ** estimating uniques with last access cookies is in the final > stages of being productionized > > > - ** Joseph's sanitization of pageview data (for long term retention > and privacy) is up for review by halfak and anyone else interested > > > === Architecture === > * '''*Blocking'''*: ??? > * '''*Blocked'''*: ??? > * '''*Updates*'''*:* > ** ??? > > === Performance === > * '''*Blocking'''*: ??? > * '''*Blocked'''*: ??? > * '''*Updates*'''*:* > ** ??? > > === Release Engineering === > * '''*Blocking'''*: > ** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to > Jessie and Node 4.2 > * '''*Blocked'''*: none > * '''*Updates*'''*:* > ** Train deploy rolling out wmf.13 this week > ** CI had some trouble due to lack of precise executioners on Monday, > should be fixed now > ** Lots of scap changes in puppet getting lined up: provider, > scap::target, service::node > > === Research === > * '''*Blocking'''*: > * '''*Blocked'''*: > * '''*Updates*'''*:* > ** Reader survey will go out (most probably, if all the technical > components work smoothly) as part of the deployment train tomorrow > (Thursday) > *** Working with the mobile web team > *** Survey will run for a week in enwiki, using QuickSurveys and we expect > to collect between 200-500K responses > *** Sampling rates will be finalized today > *** Documentation: > https://meta.wikimedia.org/wiki/Research:Characterizing_Wikipedia_Reader_Be… > > === Security === > * '''*Blocking'''*: > ** Maps: Security review for PHP schema composer module - > https://gerrit.wikimedia.org/r/#/c/269325/ > ** Graphs: Security review for SVG sanitization - > https://github.com/cure53/DOMPurify > * '''*Blocked'''*: Waiting to see how SessionManger goes before we deploy > a bunch of patches > * '''*Updates*'''*:* > ** 2FA for Keystone done, working on Horizon integration next > > === Services === > * '''*Blocking'''*: ??? > * '''*Blocked'''*: > ** https://phabricator.wikimedia.org/T125003 - move CI slaves in Beta to > Jessie and Node 4.2 > ** https://phabricator.wikimedia.org/T107307 - move CXServer to SCB > ** https://phabricator.wikimedia.org/T107306 - Apertium pkgs for Jessie > * '''*Updates*'''*:* > ** separated out the framework part of RESTBase into > https://github.com/wikimedia/hyperswitch > ** Testing Brotli compression for Cassandra in staging, seeing OOMs > ** restbase1007 bootstrapping > *** rb1001 and rb1002 might run out of disk space because of it > ** separate AQS off of RESTBase - > https://phabricator.wikimedia.org/T126294 > > === Technical Operations === > * '''*Blocking'''*: ORES > * '''*Blocked'''*: none known > * '''*Updates*'''*:* > ** OTRS upgrade to 5.0.6 > ** MultiDC work ongoing with the rest of Technology > > == Product == > === Community Tech === > * '''*Blocking'''*: none > * '''*Blocked'''*: none > * '''*Updates*'''*:* > ** Continuing work on Gadgets 2.0 after recieving further feedback > ** Continuing work on PageAssessments extension > > === Discovery === > * '''*Blocking'''*: none > * '''*Blocked'''*: > ** Maps: Security review for PHP schema composer module - > https://gerrit.wikimedia.org/r/#/c/269325/ > ** Graphs: Security review for SVG sanitization - > https://github.com/cure53/DOMPurify > * '''*Updates*'''*:* > ** Completion suggester changes done, probably not deployed yet due to > SessionManager things > ** Tuning suggester & morelike performance > ** TextCat language detection merged, will work on making A/B test with it > next > ** Working on next year plans/budgets > ** WDQS Blazegraph 2.0 update rolled back due to serious bug, > investigating the bug > ** Maps: blocked on security to deploy Kartographer extension > ** Graphs: disabled http / https protocols, use custom protocols now > ** Data: begining data storage planning (Yuri will add task # here :) > > === Editing === > ==== Collaboration ==== > * '''*Blocking'''*: > ** Dry run patch for external store migration is merged. Now we need to > set External Store up on Beta, then test the dry run patch there: > https://phabricator.wikimedia.org/T119567 > * '''*Blocked'''*: > ** Flow dump generation on dumps.wikimedia.org: > https://phabricator.wikimedia.org/T119511 > * '''*Updates*'''*:* > ** Human-readable name patch is almost done: > https://phabricator.wikimedia.org/T121936 > ** Cross-wiki notifications will soon be rolled out to "real" wikis > (currently in production it is only on test and test2): > *** 'testwikidatawiki' > *** 'wikidatawiki' > *** 'mediawikiwiki' > *** 'commonswiki' > *** 'hewiki' > *** 'hewikibooks' > *** 'hewikinews' > *** 'hewikiquote' > *** 'hewikisource' > *** 'hewikivoyage' > *** 'hewiktionary' > *** 'frwiki' > *** 'frwikibooks' > *** 'frwikinews' > *** 'frwikiquote' > *** 'frwikisource' > *** 'frwikiversity' > *** 'frwikivoyage' > *** 'frwiktionary' > > ==== Language ==== > * '''*Blocking'''*: > ** I have got some pings outside SoS, but don't know if those are blocking > ** Packages for Jessie requested to speed up > * '''*Blocked'''*: nope > * '''*Updates*'''*:* > ** > > ==== Multimedia ==== > > * '''*Blocking'''*: Not that I know of. > * '''*Blocked'''*: *Metrics work* currently paused pending some burning > questions for *halfak*, MarkTraceur to sit down with him this week. *Look > out, Security*, we're hoping to get security review on a *new extension* (see > below) in the next few months. > * '''*Updates*'''*:* > ** Labs instance with *ImageTweaks* (name WIP) going up today, *email to > (wikitech|commons|multimedia)-l* later tonight > ** Some work on *NewFiles*, in particular *filtering by tag*, but it's > not going super. MatmaRex wanted some extra eyes on it, but I think he's > contacted someone already. > > > ==== Parsing ==== > * '''*Blocking'''*: ??? > * '''*Blocked'''*: ??? > * '''*Updates*'''*:* > ** The templatedata-based serialization patch ( > https://gerrit.wikimedia.org/r/#/c/264043/ ) didn't go out this week > after all -- yet to be reviewed. Possible it will get reviewed this week. > ** Parsoid testing services (parsoid rt-testing; parsoid <--> php-parser > html visual diff testing) fully puppetized and operational on ruthenium. > Work ongoing to enable visual diff testing between production mediawiki and > a non-production mediawiki (to test for example changes to parsing) using > labs hardware ( https://phabricator.wikimedia.org/T120345 ). > *** Would appreciate help from anyone familiar with hiera (to configure > labs hardware to use existing puppet roles, and making any additional > tweaks). > ** This week, Arlo will start Parsoid-side work to separate out data-mw > into its own RESTBase bucket -- deployment is still some ways away. We need > to finalize REST API versioning policy ( > https://phabricator.wikimedia.org/T124365 ) and also have all the various > Parsoid clients updated. We will create phab tickets for dependent projects > once T124365 is resolved. > *** Last week, Flow, Services, and us had a meeting and figured out a > strategy for Flow since they use VE, but also talk with Parsoid (and not > RESTBase). > ** Parsoid native implementation for <gallery> is underway (heads up: CX, > VE, Flow, Reading) > *** https://phabricator.wikimedia.org/P2506 is the spec for Parsoid's > output if you want to provide feedback > *** https://gerrit.wikimedia.org/r/#/c/264026/ is the WIP patch > ** Tim working to collect source location information in the PHP > preprocessor for errors / warnings and a way to expose them. Useful in > general, but, also specifically in the context of work related to > replacying Tidy. > > ==== VisualEditor ==== > * '''*Blocking'''*: ??? > * '''*Blocked'''*: ??? > * '''*Updates*'''*:* > ** ??? > > === Fundraising Tech === > No blockers, most work is continued from last week > * updates for new CiviCRM financial tracking > * fixes and enhancements for backup credit card processor > * prep for Latin America fundraising expansion > * debugging email clickthrough tracking > > === Reading === > > ==== Android ==== > * Updates: beta v2.1.140 released but will not be promoted due Wikipedia > Zero incompatibilities. v2.1.141 coming soon. > * Blocked: the options API was not designed for storing collection-like > data. Gather collections don't support inter-wiki pages. > > ==== Reading Infrastructure ==== > * SessionManager is coming back this week. Please let Brad, Bryan, and > Gergő know if you spot any problems. Preferably before Friday afternoon. > > > >

1 0

← Newer
1
...
4
5
6
7
8
9
10
11
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l February 2016