Wikidata-tech

wikidata-tech@lists.wikimedia.org

1 participants
622 discussions

New names for Wikibase / Wikidata deployment branches
by Katie Filbert 24 Sep '14

24 Sep '14

We are now on MediaWiki 1.25 deployment branches of core. For Wikibase / Wikidata branches, we are using a new naming scheme: Instead of "mw1.25-wmf1" as in the past, we now have "wmf/1.25wmf1" which is the same naming scheme as used by core and other extensions. This allows Jenkins to test our branches against the corresponding version of core, if available, instead of testing against master. Thanks to hashar for making this possible. :) Cheers, Katie -- Katie Filbert Wikidata Developer Wikimedia Germany e.V. | Tempelhofer Ufer 23-24, 10963 Berlin Phone (030) 219 158 26-0 http://wikimedia.de Wikimedia Germany - Society for the Promotion of free knowledge eV Entered in the register of Amtsgericht Berlin-Charlottenburg under the number 23 855 as recognized as charitable by the Inland Revenue for corporations I Berlin, tax number 27/681/51985.

2 1

Removing 3rd party dependencies from WikibaseQueryEngine
by Daniel Kinzler 19 Sep '14

19 Sep '14

Hi. The call with the WMF today turned up a pretty big issue. WikibaseQueryEngine (WQE) depends on two big 3rd party libraries: * Symphony (used for implementing a command line client for the query engine, which we use to generate database tables) * doctine/dbal (a database abstraction layer we use to run queries and create database tables) In order to deploy these, the WMF would require a line-by-line review of something around 50thousand lines of code (or maybe it was 30thousand, or 80thousand - a lot, in any case). This is not feasible. We (Katie, Jeroen, Chris, Nik, me, etc) have come up with a plan to get rid of these dependencies: 1) Split the command line interface into a separate component (WQE-CLI perhaps) that would not be deployed. Symphony is then out of the picture. 2) Go back to using MediaWiki's DB abstraction for running queries. This should be easy. 3) For generating DB tables (aka schema creation), we create a separate component (WQE schema generator or something) that would use dbal to generate staic sql files for the supported db systems (most importantly, mysql and sqlite). This would be part of our build step, but the code relying on dbal would not be deployed. Only the generated sql files would be used for deployment (either through the update script, or, in the case of WMF, manually). In the end, WQE could use either MW or dbal for running queries, so we could deploy it without dbal on the WMF cluster, but it could also be used outside MediaWiki. If there are no objections, I propose to take this on for the next sprint. Perhaps we can start to split this into tasks on bugzilla already. Cheers, Daniel -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

6 13

DB performance review: Wikibase Usage Tracking
by Daniel Kinzler 17 Sep '14

17 Sep '14

Hi all! The Wikibase team would like to allow data from any item to be used on any client page. To do this, we need to track which item is being used where, so we can purge the appropriate pages when the item changes. We would like to have people with database experience to look at our proposal and let us know about any concerns, especially wrt performance. Here you find a proposal for two database tables for tracking the usage of entities across wikis: https://gerrit.wikimedia.org/r/#/c/158078/9/usagetracking/includes/Usage/Sq… https://gerrit.wikimedia.org/r/#/c/158078/9/subscription/includes/Subscript… The "entity_usage" table would be on every client, recording wich entity is used on which page (kind of like the iwlinks table). The "entity_per_client" table would be on the repo, and track which wiki ("client") is interested in changes to which entity. Please have a look and let me know if you have any questions or suggestions, especially with regards to the following use cases: The following would happpen when editing/re-parsing a page on a client wiki (e.g. wikipedia): * get all entities used on a given page from entity_usage * delete rows based on a page id and a list of entity ids from entity_usage * insert rows for a page / entity pair into entity_usage * queriy rows for a set of entities from entity_usage (with no page id specified). * add rows for a set of (newly used) entites to the entity_per_client table * remove rows for a set of (no longer used) entites from the entity_per_client table The following would happen when dispatching a change from wikibase: * looking up interested wikis for a list of entities from the entity_per_client table. * (notification via the job queue) * looking up pages to be purged/updated based on a list of entity ids (and possibly an aspect id) in the entity_usage table. -- daniel -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

2 2

Re: [Wikidata-tech] Removing 3rd party dependencies from WikibaseQueryEngine
by Daniel Kinzler 10 Sep '14

10 Sep '14

Hi Katie, hi Rob! Thanks for your swift replies. Am 10.09.2014 04:59, schrieb Katie Filbert: > On Tue, Sep 9, 2014 at 7:25 PM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de > <mailto:daniel.kinzler@wikimedia.de>> wrote: ... > So, there are two things we need to know in order to make an informed decision: > > 1) can we use the Ubuntu LTS packages for symfony and dbal? > > > which versions are they? php-doctrine-dbal (2.3.4-1) php-symfony-console (2.3.1+dfsg-1) Both are "universe" packages, no not "officially" supported by Ubuntu. But so is php5-memcached and other stuff we use. > are they hhvm-compatible? > > http://hhvm.h4cc.de/package/doctrine/dbal The hhvm status page for dbal lists everything before 2.5 as "not tested", so we just don't know whether it would work with hhvm. Worth a try, I suppose. I can't find anything about hhvm compatibility on symfony.com, but I found some blog posts from 2013 discussing symfony performance on hhvm, so that should work fine. Am 10.09.2014 07:34, schrieb Rob Lanphier: ... > I'm kinda regretting bringing up the apt repo case, because basically > a loophole in our review strategy, not something I'm sure we want to > encourage or something I believe will get a great deal of traction. > There's an outside chance that people shrug and say "sure, why not?", > but I don't think it's going to be generally attractive. I wouldn't call that a loophole, but rather an inconsistency. Nobody is going to review libpcre, or the mysql client library, or Lucene. I find it confusing that anything written in PHP apparently needs full review, while we just trust stuff written in C or Java. I'm not saying we should deploy just any 3rd party code without review. I'm saying we should have better criteria than the language the library is written in. One criterion could be the size of the install base - many eyes.But then, heartbleed happened... >> 1) can we use the Ubuntu LTS packages for symfony and dbal? > > Per above, probably not likely. But using, say, php5-curl or php5-oauth is fine?... I don't want to whine and complain about the deployment/review policy; I'm trying to find out whether there is a policy, and if so, what it is, and what the rationale it is based on. >> 2) when is 14.04 going to be rolled out? > > Concurrently with the HHVM upgrade. In the coming weeks. We don't > have a set end date for when all machines will be converted, but we > should be well underway by the end of this month. There's probably > going to be a stubborn service or two that will stick around quite a > bit longer than that, though. Sadly, Wikibase seems to have some compat issues with hhvm. At least that would be an explanation for the breakage on beta. We currently blame it on class aliases, but nobody really knows. It seems kind of random... >> Who can answer these questions? How do we poke TechOps? > > The Wikimedia Operations list is a good place to ask about this type of thing. Yay, yet another mailing list! Thanks, Daniel -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

1 0

congratulations
by Gerard Meijssen 10 Sep '14

10 Sep '14

Hoi, The latest statistics show that we may finally say that "only 50% of our items have zero, one or two statements". No longer can we say more that 50% of our items have zero or one statement. THAT is in my opinion a great indication of our progress of bringing data to Wikidata. As a consequence we become more information and useful. It is important to know that the first statements are in many ways the most difficult but also the most important. They often include the instances of or subclasses. Congratulations !!! Thanks, GerardM

1 0

Parser cache update/migration strategies
by Daniel Kinzler 09 Sep '14

09 Sep '14

Hi all! tl;dr: How to best handle the situation of an old parser cache entry not containing all the info expected by a newly deployed version of code? We are currently working to improve our usage of the parser cache for Wikibase/Wikidata. E.g., We are attaching additional information related to languagelinks the to ParserOutput, so we can use it in the skin when generating the sidebar. However, when we change what gets stored in the parser cache, we still need to deal with old cache entries that do not yet have the desired information attached. Here's a few options we have if the expected info isn't in the cached ParserOutput: 1) ...then generate it on the fly. On every page view, until the parser cache is purged. This seems bad especially if generating the required info means hitting the database. 2) ...then invalidate the parser cache for this page, and then a) just live with this request missing a bit of output, or b) generate on the fly c) trigger a self-redirect. 3) ...then generated it, attach it to the ParserOutput, and push the updated ParserOutput object back into the cache. This seems nice, but I'm not sure how to do that. 4) ...then force a full re-rendering and re-caching of the page, then continue. I'm not sure how to do this cleanly. So, the simplest solution seems to be 2, but it means that we invalidate the parser cache of *every* page on the wiki potentially (though we will not hit the long tail of rarely viewed pages immediately). It effectively means that any such change requires all pages to be re-rendered eventually. Is that acceptable? Solution 3 seems nice and surgical, just injecting the new info into the cached object. Is there a nice and clean way to *update* a parser cache entry like that, without re-generating it in full? Do you see any issues with this approach? Is it worth the trouble? Any input would be great! Thanks, daniel -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

3 2

Parser cache update/migration strategies
by Daniel Kinzler 09 Sep '14

09 Sep '14

1 0

Location of JSON-only dump files
by Markus Krötzsch 04 Sep '14

04 Sep '14

Hi, The JSON-only dumps are at http://dumps.wikimedia.org/other/wikidata/ Is this a preliminary location? I am asking since this does not follow the directory structure conventions of the rest of the WMF dump site, which is normally something like http://some-base-url/projectname/filename The projectname here would be "wikidatawiki" but even http://dumps.wikimedia.org/other/wikidatawiki/ would obviously not be a good location if more projects would export JSON. For example, where would Commons put its JSON in the future? Cheers, Markus

2 4

Changes to CI / Jenkins for Wikidata/base
by addshorewiki＠gmail.com 04 Sep '14

04 Sep '14

Hi all! Over the past week or so I have been working with Hashar to move our CI stuff from our own jenkins instance that we created so that we could run tests with composer back to the wmf jenkins and zuul! One the whole this has gone well! All Wikibase tests as now running through the wmf jenkins with the Wikidata build tests chasing behind (not quite complete yet) See: - https://gerrit.wikimedia.org/r/156715 - https://gerrit.wikimedia.org/r/156568 - https://bugzilla.wikimedia.org/show_bug.cgi?id=70250 Our setup now is that we have 3 jenkins slave instances on labs, each with 5 test runners. See: - https://integration.wikimedia.org/ci/computer/wikidata-jenkins1 - https://integration.wikimedia.org/ci/computer/wikidata-jenkins2 - https://integration.wikimedia.org/ci/computer/wikidata-jenkins3 These are located within the wikidata-build projects on labs (which has the small amount of documentation needed to add more instances. See: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-build As a result of this our tests in general should now run 3 times faster than they did with wdjenkins, and infact they run even faster than when we were using the old wmfjenkins setup! We also have gate-submit jobs again! That is all :) Addshore

2 1

Factoring code for repo-client interaction into packages
by Daniel Kinzler 03 Sep '14

03 Sep '14

WikibaseLib is a horrible kitchen sink, and I don't want to add more to the mess. So I want to put the usage tracking code into sensible packages. However, I'm a bit at a loss as to how to best split the different responsibilities into packages. Here are some of the communication needs we have, implying which code needs to be shared between repo and client: The client needs to: * load entity data ** need to share entity storage code ** but should not know about EntityContent ** and should have no write access * look up properties by label, and look up labels of items ** need to share term storage code ** no need for write access ** no need for code for constraints checks, etc. ** should not have related related maintenance scripts or schema update code * look up data types for properties ** need to share property info storage code ** no need for write access ** should not have related maintenance scripts or schema update code * load change details ** need to share change table storage code and value objects ** no need for write access ** no need for dispatching logic ** also should not have schema update code * look up sitelinks by page title ** need to share link table storage code ** no need for write access ** should not have related maintenance scripts or schema update code * update notification subscriptions ** need to share subscription storage code ** should not have related maintenance scripts or schema update code So, there are 6 things the client and the repo both need to access. But the write logic, or at least the maintenance logic, should not be bundled with the leaner "read only" package. So I see 12 new packages... dependency hell. So, what to do? Have 6 read level packages, and stuff the maintenance logic into a single package not used by the client? Also ugly. Ideas? -- daniel -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

4 4

← Newer
1
...
40
41
42
43
44
45
46
...
63
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech