Wikidata-tech

wikidata-tech@lists.wikimedia.org

1 participants
622 discussions

Things to get merged before the branch next week
by Daniel Kinzler 05 Dec '14

05 Dec '14

Hey! Here's a few performance relevant changes I think should get merged before we branch next week: https://gerrit.wikimedia.org/r/#/c/170961/ "Determine update actions based on usage aspects." <--- the last bit missing for usage tracking https://gerrit.wikimedia.org/r/#/c/176650/ "Use wb_terms table for label lookup." <--- should improve memory consumption a lot, and possibly also speed. https://gerrit.wikimedia.org/r/#/c/167224/ "Defer entity deserialization" <--- should reduce memory footprint and improve speed of trivial operations like checkign whether something is a redirect. Are there any other performance improvements that we should get in? I imagine that this will be the last time we branch until the third week of January. -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

1 0

SiteStore loading (memcached traffic)
by hoo 04 Dec '14

04 Dec '14

Hi Everyone, just a wanted to post a quick summary of what I did today in order to significantly reduce the SiteStore related memcached traffic. I stumbled upon this comment https://phabricator.wikimedia.org/T58602#808530 and thus poked a bit at when we load sites from memcached. During that I found that we still were loading the sites basically all the time. To get the number of that down, I uploaded the following patches that have been reviewed, merged and even deployed yet (thanks for the reviews Daniel and Katie): * Don't lookup Sites from mc for the 'languageLinkSiteGroup' setting: https://gerrit.wikimedia.org/r/177419 * Don't load all sites for LangLinkHandler: https://gerrit.wikimedia.org/r/177429 * Don't access sites on WikibaseClient::getEntityIdForTitle: https://gerrit.wikimedia.org/r/177434 That lead to a noticeable memcached traffic change (see attachment). Memcached traffic graphs: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Memcached% 20eqiad&m=cpu_report&r=hour&s=by% 20name&hc=4&mc=2&st=1417641558&g=network_report&z=large I still have https://gerrit.wikimedia.org/r/177416 in review, which slightly changes the behavior of the other projects sidebar, but I think that this change also has quite some potential to reduce memcached traffic even further. Would be great if we could get that ready for backporting until Monday. Cheers, Marius

1 0

Introducing Gerrie - A crawler for Googles code review system Gerrit
by Andy Grunwald 28 Nov '14

28 Nov '14

Hey everyone, i want to introduce Gerrie [1] to you. Gerrie is a crawler for Googles code review system Gerrit. As far as i know, the Wikimedia community, use Gerrit to improve and develop various products like Mediawiki, Wiki-Extensions, Infrastructure and so on. The Gerrit instance is located at gerrit.wikimedia.org [2]. During this activity a lot of interesting data will be created in the background. Gerrie is a tool to transform this data into a RDBMS like MySQL. After this you can create analysis with it with simple sql queries. To retrieve the data the SSH API of Gerrit will be used. The benefit from this is you can use every Gerrit system to analyze like TYPO3 [3]. I assume that most of you are registered at gerrit.wikimedia.org [2] and entered their SSH public key into the system. Gratulations, you are ready to crawl the data as well. Gerrie is written in PHP and complete documented [4]. Even the database scheme is documented [5] to help you to analyze the data. For a quickstart you can follow the documented Getting started guide [6]. Results of this analysis are wide spreaded. One use case can be for example a gamification analysis like the Activity Monitor build upon TYPO3s Gerrit data for the TYPO3.CMS (main content management system) product: http://metrics.andygrunwald.com/statistics/gerrit/activity-monitor/analysis… In this analysis every activity will results into points. Based on the sum of the points a score list will be created. The background color is determined by a hash function based on the users name. This mean: Same Name = Same color. Please take in mind: This does not reflect the contribution to the community. This displays only activity by a user in a specific system. I would love to see you as a kind of tester for this tool. If you need help, do not hesitate to ask. I try to help as much as possible. Have fun during testing. Cheers, Andy [1] Gerrie: https://github.com/andygrunwald/Gerrie [2] gerrit.wikimedia.org: https://gerrit.wikimedia.org/ [3] Gerrit @ TYPO3: https://review.typo3.org/ [4] Documentation: http://gerrie.readthedocs.org/en/latest/ [5] Database Schema: http://gerrie.readthedocs.org/en/latest/database/index.html#schema [6] Getting started: http://gerrie.readthedocs.org/en/latest/getting_started/index.html

1 0

Fixing OCP violations
by Gordon P. Hemsley 23 Nov '14

23 Nov '14

Hey all, I'd like to help with the effort to implement the ability to support additional Entity types beyond Item and Property, but I'm not entirely sure where to start. One potential avenue of contribution is the elimination of OCP violations. According to the list of dependencies [1] on bug 73496 [2], some candidates for this include EntityContent [3], ChangeOps [4], and EntityViewFactory [5]. However, as the codebase layout is currently in flux (as I imagine it always is), I'm not clear on what code is good and what code is bad, nor where new good code should live. I'm very interested in the ability to add additional entity types, and I'd like to help speed the work along without stepping on anyone's toes or duplicating any work. What is the status of these efforts? Any advice or feedback is welcome. Regards, Gordon [1]: https://bugzilla.wikimedia.org/showdependencytree.cgi?id=73496&hide_resolve… https://old-bugzilla.wikimedia.org/showdependencytree.cgi?id=73496&hide_res… [2]: https://bugzilla.wikimedia.org/show_bug.cgi?id=73496 https://old-bugzilla.wikimedia.org/show_bug.cgi?id=73496 [3]: https://bugzilla.wikimedia.org/show_bug.cgi?id=67238 https://old-bugzilla.wikimedia.org/show_bug.cgi?id=67238 [4]: https://bugzilla.wikimedia.org/show_bug.cgi?id=73500 https://old-bugzilla.wikimedia.org/show_bug.cgi?id=73500 [5]: https://bugzilla.wikimedia.org/show_bug.cgi?id=73559 https://old-bugzilla.wikimedia.org/show_bug.cgi?id=73559 -- Gordon P. Hemsley me(a)gphemsley.org http://gphemsley.org/

1 0

Usage of "instanceof StatementListProvider"
by Jeroen De Dauw 10 Nov '14

10 Nov '14

Hey Hoo and Katie, A few days ago, on IRC I mentioned that I could not see any immediate problems with doing something like if ( instanceof StatementListProvider ) { do stuff with statements } I started wondering why this is not bad, since I've definitely seen some code that got seriously bad by doing this. And I've come to the conclusion that this is fine: function storeStatementsOfEntities( EntityDocument[] ) { if ( instanceof StatementListProvider ) { store statements } } While this is not: function storeAllPartsOfEntities( EntityDocument[] ) { if ( instanceof StatementListProvider ) { store statements } if ( instanceof FingerprintProvider ) { store fingerprint } // ... } In other words, if the context is specifically about one thing that an entity can have, then it is fine. If on the other hand, you need some general handling for whole entities, such as for diffing them, serializing them, etc, then a different approach is needed. The pseudo code above should make it clear why this is the case. (The second code suffers from a big Open Closed Principle violation. You'd need to modify this if an extension adds a new type of entity containing a new type of field, which you cannot do, since the dependency would go in the wrong direction. So you become unable to define new types of entities via an extension mechanism.) IIRC the lua handling code does something with whole entities, and thus falls into the second category. If that is the case, then you probably need to do something like what I outlined here: https://lists.wikimedia.org/pipermail/wikidata-tech/2014-August/000546.html Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3

2 1

Cached label lookup
by hoo 01 Nov '14

01 Nov '14

Hi Everyone, I saw that we know have a LabelLookup[0] which is awesome (thanks, aude). I would like to use the some kind of label lookup in our Lua bindings, as getting labels there is insanely heavy (we get the entity, then push the whole thing into Lua and then extract the label). Sadly the current label lookup is uncached and as far as I can see we neither have a change to fix that nor a bug for the issue, yet. It would be great to hear about why we don't have this yet (or am I missing something?) and whether you think it's ok to use it yet (don't want to knock over the DB by hitting wb_terms a lot). Cheers, Marius [0]: https://gerrit.wikimedia.org/r/169330

2 1

Breaking changes to wikibase.RepoApi
by Adrian Lang 28 Oct '14

28 Oct '14

Hi everyone, in the current deployment, we made a breaking change to wikibase.RepoApi. We will do another breaking change to it in the next deployment. If you use RepoApi on wikidata.org or any Wikidata client wiki, please read these instructions on how to migrate your code. == Use case 1: RepoApi.get, RepoApi.post == You exclusively use RepoApi.get or RepoApi.post. You can and will have to use a mw.Api directly. If you are on wikidata.org, just use the following: > var api = new mw.Api(); > // api.get( { … } ); > // api.post( { … } ); If you are on a Wikidata client wiki, use the new wikibase.client.getMwApiForRepo: > mw.loader.using( [ 'wikibase.client.getMwApiForRepo' ] ).done( function() { > var api = wikibase.client.getMwApiForRepo(); > // api.get( { … } ); > // api.post( { … } ); > } ); In this case, I would encourage you to look into the features RepoApi provides. Chances are RepoApi could build your api call for you. == Use case 2: RepoApi's wrapper methods == You use one of RepoApi.createEntity, RepoApi.editEntity, RepoApi.formatValue, RepoApi.getEntities, RepoApi.getEntitiesByPage, RepoApi.parseValue, RepoApi.searchEntities, RepoApi.setLabel, RepoApi.setDescription, RepoApi.setAliases, RepoApi.setClaim, RepoApi.createClaim, RepoApi.removeClaim, RepoApi.getClaims, RepoApi.setClaimValue, RepoApi.setReference, RepoApi.removeReferences, RepoApi.setSitelink, or RepoApi.mergeItems. You should continue using RepoApi, but have to provide it a mw.Api instance. If you are on wikidata.org, just use the following: > mw.loader.using( [ 'wikibase.RepoApi' ] ).done( function() { > var mwApi = new mw.Api(); > var repoApi = new wikibase.RepoApi( mwApi ); > // mwApi.get( { … } ); > // mwApi.post( { … } ); > // repoApi.setClaim( … ); > } ); If you are on a Wikidata client wiki, use the new wikibase.client.getMwApiForRepo: > mw.loader.using( [ 'wikibase.client.getMwApiForRepo' ] ).done( function() { > var mwApi = wikibase.client.getMwApiForRepo(); > var repoApi = new wikibase.RepoApi( mwApi ); > // mwApi.get( { … } ); > // mwApi.post( { … } ); > // repoApi.setClaim( … ); > } ); If you have any questions, feel free to contact me. Bye, Adrian

1 0

PHP library for the JSON dump
by Jeroen De Dauw 22 Oct '14

22 Oct '14

Hey, I've just created two tiny PHP components for reading Wikibase JSON dumps. * https://github.com/JeroenDeDauw/JsonDumpReader * https://github.com/JeroenDeDauw/JsonDumpData Hope this is of use to others :) Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3

1 0

Description(s)
by Magnus Manske 19 Oct '14

19 Oct '14

Hi all, in the dump file wikidatawiki-20140912-pages-articles.xml.bz2 I seem to find some items with a key "description", some with "descriptions". For example, near the beginning of the file: Q15 seems to have key "description" Q17 seems to have key "descriptions" This is rather unhelpful when running e.g. my stats script. a) Can someone please confirm that I'm not crazy? I mean, in this instance. b) Is this a bug, or a feature? c) If a bug, is it already fixed for the next dump? Which key will it be? (If a feature: why?) Thanks, Magnus

6 11

Parsing Entity IDs
by Jeroen De Dauw 17 Oct '14

17 Oct '14

Hey, I just noticed this commit [0], which gets rid of a pile of direct BasicEntityIdParser usages for performance reasons. It also addresses a problem which is apparently more widespread than I thought: using an EntityIdParser that only works for Item IDs and Property IDs. Unless you are only dealing with Item IDs or with Property IDs, an EntityIdParser should always be injected. This allows the thing constructing the object graph to add support for all required entity types, and gives extensions to Wikibase Repo (or Wikibase Client) a chance to register new entity types. Of course this also means that no new code that introduces such occurrences should be allowed through review, even if it contains a "fix this later" TODO (for new code there is no excuse to do it wrong). [0] https://gerrit.wikimedia.org/r/#/c/167136/ Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3

2 1

← Newer
1
...
38
39
40
41
42
43
44
...
63
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech