Wikidata-tech July 2013

wikidata-tech@lists.wikimedia.org

26 participants
25 discussions

New awesome code coverage tracking
by Jeroen De Dauw 22 Jul '13

22 Jul '13

Hey, This weekend I stumbled upon Coveralls [0], a service that provides code coverage statistics for open source projects. Currently I added it to the TravisCI build of both Ask and Diff, which was really easy to do [1]. You can find reports on the coveralls site [2], and can also get a quick idea of the coverage status via the badge on the documentation page of the relevant component [3]. I've been wanting this kind of reporting for a long time, so we can track the general trends of our coverage, and spot commits that significantly lower it. So this coveralls thing is quite awesome \o/ [0] https://coveralls.io and https://coveralls.io/info/features [1] https://gerrit.wikimedia.org/r/#/c/74960/ [2] https://coveralls.io/r/wikimedia/mediawiki-extensions-Diff [3] https://github.com/wikimedia/mediawiki-extensions-Diff/ Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

2 1

Incorrect language code?
by Hazard-SJ 21 Jul '13

21 Jul '13

Hello, As far as I checked, we should be using "nb" as the language code of "nowiki". As is known, (see bugs 46455 and 37459), Wikidata allows both codes. I've just come upon a bot request that is over 2 months old that I'm willing to tackle, just that I need verification: terms should use the language code "nb" rather than "no", correct? As for statistics, there are 534741 uses of the "no" code at present, and only 294725 uses of the "nb" code (at present). As I said, I'm willing to have my bot make the necessary fixes, but I just need to verify i advance. Thanks. Hazard-SJ

3 3

Agenda for call tonight
by Denny Vrandečić 18 Jul '13

18 Jul '13

Agenda as per Etherpad <http://etherpad.wmflabs.org/pad/p/Wikidata> * Deployment prep / Wikivoyage on Tuesday, July 23 * Solr/ElasticSearch * SpamBlackList + Wikidata -> backwardscompatibility of SpamBlackList with pre-ContentHandler MW? * to do for Berlin team: document all modules, their deployment status, etc. -> Started it, but where actually to put it? * https://bugzilla.wikimedia.org/show_bug.cgi?id=41586 -> "On wikidata.org, api login returns wrong lgtoken" -> update? If there is anything to add,just add it to the Etherpad or write me. Until later, Denny -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

1 0

DataValues components rethinking
by Jeroen De Dauw 17 Jul '13

17 Jul '13

Hey, While considering the moving of the components currently in the DataValues git repository [0], it occurred to me the current components are not all that nicely dividing the code. All of these components, with the exception of DataTypes and ValueView, bundle whole inheritance hierarchies. A classic package design mistake that leads to various problems. My current plan here is to have a new component that holds the interfaces, exceptions and trivial implementations for all of these components (again, except DTs and VV). Many of the components currently dependent on one of the already existing ones, would then just need this one, which is much more abstract/stable, tackling most of the pains of the current component design. Two questions: * Any suggestions on what to call this new component to hold the interfaces? * Any alternate approaches that improve upon the current situation? There is some urgency to this, as the git repo split should happen soon, to alleviate the pains currently caused by not having independently deployable and released versions. [0] DataValues, ValueParsers, ValueValidators, DataTypes, etc Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

2 10

Jenkins failing for no good reason.
by Daniel Kinzler 16 Jul '13

16 Jul '13

There seems to be an issue with Jenkins. It appears to use an old version of other extensions under some circumstances. It's like this: If you submit change 33 for extension A, which needs change 44 in extension B (which isn't merged yet), jenkins will fail correctly fail. BUT: When change 44 got merged into extension B, and you force Jenkins to re-run (e.g. by rebasing change 33), it will *still* fail, apparently using an old version of extension B. It seems this is only the case for the "testextensions-master" job, not the standalone "repo" and "client" jobs. Here are some examples: https://gerrit.wikimedia.org/r/#/c/72962/ fails for no good reason https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-mast… https://gerrit.wikimedia.org/r/#/c/73772/ fails for no good reason https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-mast… Please gather more evidence/insights if you come across this issue. Thanks, Daniel

2 2

Dispatch script stopped working? [urgent]
by Denny Vrandečić 14 Jul '13

14 Jul '13

Hey all, it seems the dispatcher stopped working about 17-18 hours ago (i.e. Friday 5pm Berlin / 8am SF). Could we please, as fast as possible investigate and get it running again? (DNS changes have been suggested as the culprits, I have no clue) Cheers, Denny -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

3 3

Master is broken!
by Jeroen De Dauw 12 Jul '13

12 Jul '13

Hey, Something broke Wikibase master recently. See https://gerrit.wikimedia.org/r/#/c/73439/ ( https://integration.wikimedia.org/ci/job/mwext-Wikibase-repo-tests/1986/con…) Anyone an idea what caused this? Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

3 2

Search plugins for Firefox and Chrome
by Jeroen De Dauw 11 Jul '13

11 Jul '13

Hey all, Today I had a day off and decided to play around with Firefox plugins and Chrome extensions. The result is one of each being created, both for searching against Wikidata. They are both very simple, though hopefully helpful to you. Firefox plugin: https://addons.mozilla.org/en-US/firefox/addon/wikidata-search/ Chrome extension: https://chrome.google.com/webstore/detail/wikidata-search/ingjkjibhnkhomoml… A more verbose version of this email: http://www.bn2vs.com/blog/2013/07/12/wikidata-search-plugins/ Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

One component per repository
by Jeroen De Dauw 11 Jul '13

11 Jul '13

Hey, A month ago we had some discussion on our internal list on only having a single component per git repo. I now want to follow up on this. Since there is no reason to do this on the internal list, I'm now doing it here and including the original mail. WikibaseQueryEngine and WikibaseQuery have been split off, and more excitingly, we finally managed to do this with the DataModel component as well. Now the dust has settled from those changes, I think it is time to look at the next one: DataValues. Right now this repo contains 6 components. Splitting this up thus means 5 new repos. We can do this one by one, though it probably makes sense to do it close together. I suggest starting with ValueView and DataTypes, as these are currently being the most awkward dependency wise. Any objections against starting work on this next week (ie the one that starts tomorrow)? (Do read my original mail (below) first in case you have not yet already done so.) Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- ---------- Forwarded message ---------- From: Jeroen De Dauw <jeroendedauw(a)gmail.com> Date: 9 June 2013 01:28 Subject: One component per repository To: Wikidata-intern <wikidata-intern(a)wikimedia.de> Hey, As you all know, right now we have git repositories containing multiple components. There are two main reasons for this: * We started with different repos for client, lib and repo and then had a lot of hassle with moving code between them. This reason seems mainly historical at this point. This kind of code moving happening only very rarely at this point, at least in all components that are not client, lib or repo. It might still be going on to some extend in those three, though I suspect this has decreased since the start of the project, probably due to us more clearly defining the component boundaries. In fact, I think the primary reason we ended up moving so many things in these components was caused by those boundaries being so poorly defined. * Installation hassle. Burdening regular users of the software to know about all the dependencies, the required versions and having to load them in a specific version in LocalSettings is a bad idea. Ideally they should not have to know about the dependencies at all. So putting everything in a single repo is a reasonable idea if you lack a package management system. Luckily we are now very close to having Composer work for wikibase, which enables us to have as many packages as we deem fit, without bothering the users with them. So that will not only mitigate making the installation process more complex, it will reduce current complexity (since right now people do need to care about the dependencies (Diff, Ask, DataValues, ...)). In short, the main reasons for multiple components per repo no longer apply in our current situation. OTOH there are some clear disadvantages: * People in the PHP world tend to assume there is only one component in a git repo. This is reflected in mainstream tools such as Composer and TravisCI having this assumption build in. It is not possible (AFAIK) to properly use Composer if you have all your packages in a single repo. For instance, it is currently not possible to list ValueParsers as a distinct component from DataValues in the package repositories. It is also not easily doable to have Travis do a build for each component, you'd have to add a two tier build process, which is a lot of added complexity. * Since the packages contain multiple components, it is less easy to keep track of changes to a single component. Or to make use of just a single component, without having to care about the other ones. Reusability and likelihood of outside contributions are decreased if your package contains a ton of not applicable stuff to the person that just wants to use one of the components in there. * The notion that we cannot create new repositories for new components promotes harmful practices. These components will just be put somewhere based on what currently needs them, and possibly need to be moved closer to the root of the dependency tree. A good example of this is the ValueView code. If I'm not mistaken, some of this was originally in repo, then in lib (same package but different component), and then in the DataValues repo. The last step was made since we happened to be loading the DataValues repo everywhere we need this JavaScript code. ValueView certainly does not belong in the same repo however. Both components do very different things and are even written in different languages, so they will have different consumers, different contributors and different reasons for change. * More clear and explicit dependency structure. Makes messing up of dependencies easier to detect, and makes understanding the individual components easier. ... Given that, I suggest we stop the earlier pattern of putting multiple components in the same repo, and work on moving misplaced components into new repos. I will be doing this for the Database, Query and QueryEngine components in the coming days, as these are not needed by any functionality currently enabled anywhere. I suggest we do the same for the components in DataValues and for DataModel. We can probably leave lib, repo and client as they are, at least for now. Those are not really reusable in their current form anyway, and are the hardest ones to make improvements to. Plus we might want to tackle fixing issues with lib first. Putting DataModel in its own repo is currently blocked by remaining legacy dependencies, which need to be resolved first. That leaves the components in the DataValues repo as the thing to start with. I suggest we tackle that one as soon as everyone of the team got a more clear idea of how this will work with Composer and when there are no new commits on gerrit against the relevant component(s). "Oh no, we will end up with a million git repos now!" Not really. We have 14 components right now, and this number has not increased a lot lately. The only recent additions are Database and QueryEngine. We might end up with a few more over time, but not hundreds, or even dozens. "I still don't want to care about 14 different repos!" The situation for developers will not get worse (else I'd not be a fan of the idea to begin with), and will actually improve in places. For instance, when working on the QueryEngine component, you will now no longer need all this unrelated client, lib, repo, datatypes, valueparsers, valuevalidators, valueformatters and valueview code. If someone breaks the build for one of those, you can happily not care, since that code is not relevant to the work you are doing. You'll also have better CI support. And if you are not working on any of the components you depend on, for instance when working just on client, you can use Composer much like a regular user and not give a damn about all indirect dependencies. </walloftext><other people raging> Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

5 6

New Serialization component
by Jeroen De Dauw 10 Jul '13

10 Jul '13

Hey, While working on the serialization and deserialization code for Ask (which is now finally done - yay!), I figured we'd have use for the same interfaces, exceptions and basic utilities almost everywhere we do this kind of work. After bouncing of the idea to put this into its own component of some people, I went ahead with this. The new component is named Serialization (we failed to come up with a better name) and is now in a WMF git repo. You can browse the code at https://github.com/wikimedia/mediawiki-extensions-Serialization For now, the only direct dependency on this component is the Ask library. It'll need to be included for deployment as soon as we deploy the WikibaseQuery extension. Or as soon as we decide to load Ask for some other reason. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech July 2013