Hi Everyone,
It's time for Wikimedia Tech Talks 2019 Episode 6!
Next month's talk will take place July 10, 2019 at 4:00 PM UTC.
Topic: A Deployment Pipeline Overview
Speaker: Alexandros Kosiaris, Senior Operations Engineer
Summary: The deployment pipeline project has been ongoing for a while,
sometimes with more resources poured into it, sometimes less, but it's
finally in a state that is ready to be used (it's already being used!).
This tech talk is about a presentation to wider technical audiences,
discussing the goals of the project, the implementation decisions and
how it's meant to be used and adopted by the deployers of services
(and eventually MediaWiki) in the coming months.
YouTube stream for viewers: https://www.youtube.com/watch?v=i0FTcG7PxzI
During the live talk, you are invited to join the discussion on IRC
at #wikimedia-office
Those in the SF office can watch this in R124.
You can watch past Tech Talks here:
https://www.mediawiki.org/wiki/Tech_talks
If you are interested in giving your own tech talk, you can learn more here:
https://www.mediawiki.org/wiki/Project:Calendar/How_to_schedule_an_event#Te…
Subbu.
(Standing in for Sarah Rodlund who is currently away on sabbatical)
I'm getting a new error today that was not happening with the identical
config yesterday.
With MW 1.31.1 and 1.31.2 I'm getting errors after installing dependencies
and extensions with Composer. I have the following composer.local.json:
```
{
"require": {
"mediawiki/semantic-media-wiki": "3.0.0",
"mediawiki/semantic-result-formats": "3.0.0",
"mediawiki/semantic-compound-queries": "1.2.0",
"mediawiki/sub-page-list": "1.5.0",
"mediawiki/maps": "6.0.3",
"pear/net_smtp": "1.8.0" <-- remove this when using MW 1.31.2
},
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json",
"extensions/Elastica/composer.json"
]
}
}
}
```
SyntaxHighlight_GeSHi and Elastica are both tracking their REL1_31
branches, which haven't changed in about a year. Despite that, yesterday
running `composer update` caused no issues. Today I started getting the
following error:
```
[Wed Jun 26 16:16:18.364715 2019] [php7:warn] [pid 12304] [client
my.ip.add.ress:46156] PHP Warning:
require(/opt/htdocs/mediawiki/vendor/composer/../jetbrains/phpstorm-stubs/PhpStormStubsMap.php):
failed to open stream: No such file or directory in
/opt/htdocs/mediawiki/vendor/composer/autoload_real.php on line 70,
referer: https://example.com/
[Wed Jun 26 16:16:18.364896 2019] [php7:error] [pid 12304] [client
my.ip.add.ress:46156] PHP Fatal error: require(): Failed opening required
'/opt/htdocs/mediawiki/vendor/composer/../jetbrains/phpstorm-stubs/PhpStormStubsMap.php'
(include_path='/opt/htdocs/mediawiki/vendor/pear/console_getopt:/opt/htdocs/mediawiki/vendor/pear/mail:/opt/htdocs/mediawiki/vendor/pear/mail_mime:/opt/htdocs/mediawiki/vendor/pear/mail_mime-decode:/opt/htdocs/mediawiki/vendor/pear/net_smtp:/opt/htdocs/mediawiki/vendor/pear/net_socket:/opt/htdocs/mediawiki/vendor/pear/pear-core-minimal/src:/opt/htdocs/mediawiki/vendor/pear/pear_exception:.:/usr/share/pear:/usr/share/php')
in /opt/htdocs/mediawiki/vendor/composer/autoload_real.php on line 70,
referer: https://example.com/
```
Using MW 1.32.1 and 1.31.2 I did the following:
```
rm -f ./composer.lock && composer install && sudo systemctl reload httpd
# no mention of phpstorm-stubs in output
# page loads fail
rm -f ./composer.lock && composer install --no-dev && sudo systemctl reload
httpd
# output includes: Removing jetbrains/phpstorm-stubs (dev-master)
# page loads succeed (no phpstorm-stubs)
composer install && sudo systemctl reload httpd
# output includes: Installing jetbrains/phpstorm-stubs (dev-master
9d01ce3): Cloning 9d01ce3476
# page loads succeed (phpstorm-stubs @ 9d01ce3)
composer update && sudo systemctl reload httpd
# output includes: Updating jetbrains/phpstorm-stubs dev-master (9d01ce3 =>
1b99060): Checking out 1b9906084d
# page loads fail (phpstorm-stubs @ 1b99060)
```
I need the httpd reload to invalidate opcache in my config. Based upon the
above it seems that the phpstorm-stubs that is getting required from
non-dev dependencies doesn't have issues, but the one from dev dependencies
does. Or perhaps something to do with Composer or with the Composer merge
plugin is causing one version of phpstorm-stubs to be checked out but the
autoloader to be setup for the other version. Any thoughts?
Thanks,
James
I wanted to restart the thank you Tuesday thread for this week. So heres my list of thanks:
Paladox, and releng, for all you do for maintaining Gerrit, Phabricator, and CI so we can keep making so many needed changes to mediawiki and other software/projects.
Andre, for sorting through hundreds if not thousands of tickets every day, without him, I’m sure phab would be a mess of unsorted tasks!
Finally, thanks to all the volunteers and staff that are constantly deploying, and creating patches to make MediaWiki and related projects as bug-free and user friendly as possible.
--
Devin “Zppix” CCENT
Volunteer Wikimedia Developer
Africa Wikimedia Developers Member and Mentor
Volunteer Mozilla Support Team Member (SUMO)
Quora.com Partner Program Member
enwp.org/User:Zppix
**Note: I do not work for Wikimedia Foundation, or any of its chapters. I also do not work for Mozilla, or any of its projects. **
Hi all!
tl;dr: Gerrit HTTP token auth has been re-enabled. To use it you'll need to
generate a token via your preferences page[0].
Gerrit HTTP token auth was disabled in mid-march due to concerns about its
implementation[1].
Thanks to the work of Paladox and Gerrit upstream in Gerrit 2.15.14[2] we've
re-enabled HTTP token authentication.
I previously removed all HTTP auth tokens, so in order to use HTTP token auth
you'll need to generate a fresh token via your preference page[0]
Your Lowly Gerrit Fiddler,
-- Tyler
[0]. <https://gerrit.wikimedia.org/r/#/settings/http-password>
[1]. <https://phabricator.wikimedia.org/T218750>
[2]. <https://www.gerritcodereview.com/2.15.html#21514>
Hi all!
I invite you to try out "Project Ruprecht"[1][2], a tool that measures the
"tangledness" of PHP code, and provides you with a "naughty list" of things to fix.
For now, you will have to install this locally. I hope however to soon have this
run automatically against core on a regular basis, perhaps by integrating it
with SonarQube. Maybe some day we can also integrate it with CI, to generate a
warning when a new cyclic dependency is about to be introduced.
So that was the tl;dr. Now for some context, history, and shout-outs. And some
actual real world science, too!
For a while now, I have been talking about the how much of a problem cyclic
dependencies are in MediaWiki core: When two components (classes, namespaces,
libraries, whatever) depend on each other, directly or indirectly, this means
that one cannot be used without the other, nor can it be tested, understood, or
modified without also considering the other. So, in effect, they behave as *one*
component, not two. Applied to MW core, this means that roughly half of our 1600
classes effectively behave like a single giant class. This makes the code rather
hard to deal with.
To fix this, I have been looking for tools that let me identify "tangles" of
classes that depend on each other, and metrics' to measure the progress of
"untangling" the code. However, the classic code quality metrics focus on
"local" properties of the code, so they can't tell us much about the progress of
untangling. And the tools I found that would detect cyclic dependencies in PHP
code would all choke on MediaWiki core: they would try to list all detected
cycles - which, by the super-exponential nature of possible paths through a
graph, would be millions and millions. So, the tools would choke and die. That
approach isn't practical for us.
Two discoveries allowed me to come up with a working solution:
First, I decided to leave the PHP world and turned towards graph analysis tools
built for large data sets. Python's graph-tool did the trick. It's build on top
of boost and numpy, and it's *fast*. It crunched through the 7500 or so class
dependencies in MW core in a split second, and told me that we have 14 "tangles"
(non-trivial strongly connected components), and that 43% of our classes are in
these tangles, with 40% being part of one big tangle that is essentially our
monolith manifest. So now I had a metric to work with: the number of classes in
tangles.
That was great, but still didn't tell me where to start. Graph-tool was still
not fast enough to deal with millions of cycles, and even if it had been, that
data wouldn't be very useful. I needed some smart heuristics. Luckily, I
(totally unintentionally, promise!) nerd sniped[5] Amir Sarabadani one evening
at the WMDE office by telling him about this problem. The next day, he told me
that he had been digging into the problem all night, and he had found a paper
that sounded relevant, and it also came with working code: "Breaking Cycles in
Noisy Hierarchies"[3] by J. Sun, D. Ajwani, P.K. Nicholson, A. Sala, and S.
Parthasarathy. I played with the code a bit, and yes! It spat out a list of 290
or so dependencies[4] that it thought were bad - and I agree for a good number
of them. It's not a clean working list, but it gives a very good idea of where
to start looking.
I find it quite fascinating that this works so well for cleaning up a codebase.
After all, the heuristic wasn't design for this - it was designed for fixing
messy ontologies. Indeed, one of their test data sets was (English language)
Wikipedia's category system! I'd love to see what it does with Wikidata's
subclass hierarchy :)
But I suppose it makes sense - dependencies in software are conceptually a lot
like an ontology, and the same strategies of stratification and abstraction
apply. And the same difficulties, too - it's easy enough to spot a problematic
cycle, but often hard to say where it should be cut. And how to cut it - often,
the solution is not to just remove the dependency, but to introduce a new
abstraction that allows the relationship to exist without a cycle. I'd love to
see the research continue in that direction!
So, a big shout out to the researchers, and to Amir who found the paper!
I hope my ramblings have made you curious to play with Ruprecht, and see what it
has to say about other code bases. There's also another feature to play with
which I haven't discussed here: detection of risky classes using the Page Rank
algorithm. Fun!
Cheers,
Daniel
[1] https://phabricator.wikimedia.org/diffusion/MTDA/repository/master/
[2]
https://gerrit.wikimedia.org/r/admin/projects/mediawiki/tools/dependency-an…
[3] https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchies
[4] https://phabricator.wikimedia.org/P8513
[5] https://xkcd.com/356/
--
Daniel Kinzler
Principal Software Engineer, Core Platform
Wikimedia Foundation
Hello,
This is the weekly update from the Search Platform team for the week
starting 2019-06-10.
As always, feedback and questions are welcome.
== Discussions ==
=== Search ===
* Trey has finished his first analysis of a proposal to strip
diacritics in Slovak for searching. Slovak searchers don't always use
them, but removing them causes new problems for stemming. Read more on
MediaWiki or Phabricator —and join the discussion if you speak Slovak!
[0] [1]
* Mathew worked on creating a Cookbook to restart WDQS after applying
updates or doing a deployment [2]
* The team worked with Analytics to port usage of
mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request [3]
* When a new item is created, we have functionality to instant-index
it synchronously, even if with partial information, so it appears
quickly in the index - Stas worked on getting the un-redirected items
to be instant-indexed [4]
* CirrusSearch\SearcherTest::testSearchText PHPUnit tests take a long
time to run and is slowing down change processing in CI - David helped
to make it faster [5]
* It is currently not possible to evaluate all items without labels,
so we investigated adding haslabel:all and determined that matching an
_all field and a descriptions_all field worked well [6]
* We removed old settings ($wgLexemeUseCirrus and
$wgLexemeDisableCirrus) as they were both set to true for production
(no longer applicable / out of date) [7]
* David fixed an issue where suggested article are no longer shown,
API Usage for the morelike search endpoint has dropped since train
deploy of 1.34.0-wmf.7 -- by accident. [8]
* Geodata not being returned for some files - David merged in a few
patches to be sure that we don't lose CoordinatesOutput when multiple
slots are available [9]
* An old request to fix where Special:Search doesn't use labels and
descriptions for suggestions but just the item ID was worked on by
Stas and was enabled [10]
* A PHP notice in MW-Vagrant when searching with CirrusSearch was
happening in the action API and web serach interface. Thanks for
fixing it, Ottomata!
[0] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Folding_Diacritics_i…
[1] https://phabricator.wikimedia.org/T223787#5260102
[2] https://phabricator.wikimedia.org/T221832
[3] https://phabricator.wikimedia.org/T222268
[4] https://phabricator.wikimedia.org/T206256
[5] https://phabricator.wikimedia.org/T225184
[6] https://phabricator.wikimedia.org/T224611
[7] https://phabricator.wikimedia.org/T225183
[8] https://phabricator.wikimedia.org/T224879
[9] https://phabricator.wikimedia.org/T223767
[10] https://phabricator.wikimedia.org/T55652
----
Subscribe to receive on-wiki (or opt-in email) notifications of the
Discovery weekly update.
https://www.mediawiki.org/wiki/Newsletter:Discovery_Weekly
The archive of all past updates can be found on MediaWiki.org:
https://www.mediawiki.org/wiki/Discovery/Status_updates
Interested in getting involved? See tasks marked as "Easy" or
"Volunteer needed" in Phabricator.
[1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R
[2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R
Yours,
Chris Koerner (he/him)
Community Relations Specialist
Wikimedia Foundation