Wikidata-tech March 2014

wikidata-tech@lists.wikimedia.org

7 participants
8 discussions

TravisCI integration with Gerrit
by Jeroen De Dauw 14 Jun '14

14 Jun '14

Hey Steffen and Andy, Continuing what I started on Twitter here, as some more characters might be helpful :) It seems that both our projects (FLOW3 and Wikidata) are in a similar situation. We are using Gerrit as CR tool, and TravisCI to run our tests. And we both want to have Travis run tests for all patchsets submitted to Gerrit, and then +1 or -1 on verified based on the build passing or failing. To what extend have you gotten such a thing to work on your project? Is there code available anywhere? If both projects can use the same code for this, I'd be happy to contribute to what you already have. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

2 2

GSoC 2014: Wikidata Annotation tool
by Amanpreet Singh 24 Mar '14

24 Mar '14

Hello everyone, I have applied for GSoC 2014 with Mediawiki aiming to create a plugin that can annotate statements in various websites and feed then as statements ( with references taken as website url and author in the case of Google books, Wikisource etc. Project Proposal is currently hosted at : https://www.mediawiki.org/wiki/Wikidata_annotation_tool I need to get more feedback from community such as what extra features we require from the tool which can prove this project more useful. I hope this project is considered useful by the community. So kindly, take a look at proposal and provide any valuable comments you can. Thanks Note: This mail has been crossposted to wikidata-l and wikitech-l -- Amanpreet Singh, IIT Roorkee

1 0

Status of the new (de)serialization code
by Jeroen De Dauw 19 Mar '14

19 Mar '14

Hey all, A quick update on the status of the new serialization and deserialization code. For a long time now, we've had two serialization formats. One intended for external usage, and one intended for internal usage. The former one is the format our web API uses. The later one is the format in which entities are stored in the Wikibase Repo database, and is what ends up in the current dumps. The code responsible for serialization and deserialization is for both formats not reusable and has a high degree of associated technical debt. To tackle both the reusability and design problems, we started work on two new components. One for the external format and one for the internal one. * External one: https://github.com/wmde/WikibaseDataModelSerialization * Internal one: https://github.com/wmde/WikibaseInternalSerialization They are now both essentially feature complete. The next step will be to replace usage of our existing code with usage of these new components, so the old code can be removed. Once this is done, the components will be 100% feature complete and very well tested, at which point their 1.0 versions will be released. For a description of the responsibility of both components and an explanation of how to use them, see their README files. Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3

1 0

Managing dependencies when extending Wikibase
by Daniel Kinzler 13 Mar '14

13 Mar '14

The folks of the Wikidata.lib project at the Hasso Plattner Institut have developed an extension to Wikibase that allows us to suggest properties to add to items, based on the properties already present (a very cool project, btw). This is, conceptually, and extension to the Wikibase extension. This raises problems for managing dependencies: * conceptually, the extension (property suggester) depends *on* wikibase. * practically, we want to install the property suggester as an optional dependency (feature/plugin/extension) *of* wikibase. So, how do we best express this? How can composer handle this? I think the most obvious/important thing to do is to have a separate module for the interface wikibase exposes to plugins/extensions. This would include the interfaces of the service objects, and some factory/registry classes for accessing these. What's the best practice for exposing such a set of interfaces? How is this best expressed in terms of composer manifests? What are the next steps to resolve the circular/inverse dependencies we currently have? -- daniel PS: Below is an email in which Moritz Finke listed the dependencies the property suggester currently has: -------- Original-Nachricht -------- Betreff: PropertySuggester Dependencies Datum: Thu, 6 Mar 2014 11:07:56 +0000 Von: Finke, Moritz <Moritz.Finke(a)student.hpi.uni-potsdam.de> An: Daniel Kinzler <daniel.kinzler(a)wikimedia.de> Hi, unten sind die Abhängigkeiten des PropertySuggesters nach Klassen sortiert... Grüße Moritz Abhängigkeiten PropertySuggester: GetSuggestions: use Wikibase\DataModel\Entity\ItemId; use Wikibase\DataModel\Entity\Property; use Wikibase\DataModel\Entity\PropertyId; use Wikibase\EntityLookup; use Wikibase\Repo\WikibaseRepo; use Wikibase\StoreFactory; use Wikibase\Utils; use ApiBase; use ApiMain; use DerivativeRequest; WikibaseRepo::getDefaultInstance()->getEntityContentFactory(); StoreFactory::getStore( 'sqlstore' )->getEntityLookup(); StoreFactory::getStore()->getTermIndex()->getTermsOfEntities( $ids, 'property', $language ); Utils::getLanguageCodes() 'type' => Property::ENTITY_TYPE ) SuggesterEngine: use Wikibase\DataModel\Entity\Item; use Wikibase\DataModel\Entity\PropertyId; Suggestion: use Wikibase\DataModel\Entity\PropertyId; SimplePHPSuggester: use Wikibase\DataModel\Entity\Item; use Wikibase\DataModel\Entity\PropertyId; use DatabaseBase; use InvalidArgumentException; GetSuggestionsTest: use Wikibase\Test\Api\WikibaseApiTestCase; SimplePHPSuggesterTest: use Wikibase\DataModel\Entity\PropertyId; use Wikibase\DataModel\Entity\Item; use Wikibase\DataModel\Claim\Statement; use Wikibase\DataModel\Snak\PropertySomeValueSnak; use DatabaseBase; use MediaWikiTestCase; JavaScript: wikibase.entityselector wbEntityId

1 1

Reusable "Internal" serialization component
by Jeroen De Dauw 05 Mar '14

05 Mar '14

Hey all, We recently finished with the initial version of a reusable component capable of serializating and deserializating our DataModel [0]. This covers the serialization format we use for our web API, and not the one we use "internally" for storing our entities. This happens to be the format which is in our current dumps, and is thus dealt with by various people and tools outside of our actual core software. We are now starting with the creating of a serialization component for this format [1]. As before, everyone is welcome to contribute to this component. The work is very self contained and requires little knowledge of our wider codebase. The advice I have for the DataModelSerialization component also applies here, so have a look at if you want to contribute [2]. [0] https://github.com/wmde/WikibaseDataModelSerialization [1] https://github.com/wmde/WikibaseInternalSerialization [2] http://lists.wikimedia.org/pipermail/wikidata-tech/2013-December/000356.html Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3

1 0

About:precision in global coordinates
by Fredo Erxleben 03 Mar '14

03 Mar '14

Hey there, While testing I found that the notations of precision in the internal dumpfiles is not always what I expected: My example is the item Zwickau (json and stacktrace attached): The precision in the geolocation is denoted as an decimal value, while the implementation of GlobalCoordinatesValueImpl requires a long value encoding a precision. How should this discrepancy be handled best? Greetings, Fredo PS: I am at the moment checking all the errors that occur while converting an earlier dumpfile, so I am likely to stumble upon more problems soon.

3 2

Re: [Wikidata-tech] Composer, Wikibase
by Jeroen De Dauw 01 Mar '14

01 Mar '14

Hey, (Included wikidata-tech) Also, while you're at it, if you have a sec, you might like to look at the > tentative idea for using a separate Wikibase store for the features we're > working on: > https://www.mediawiki.org/wiki/Editor_campaigns/Notes#Structured_data_store > The spirit is similar in many ways to what's in the EP extension, though > we're targeting a much wider range of use cases, and the code should be > much more modular. Also hoping to achieve a very slick UX (or UX's) with > help from designers. > Will just reply to this part here. >From what I read on this page, I'd not be thinking about using Wikibase. If anything, Semantic MediaWiki seems like a better fit (though perhaps still not a good one - I do not suggest you try using it here). * Wikibase does not support queries and visualizations at this point - you'd just have entity storage * Our entity storage code is not reusable at this point * Our two applications, Wikibase Repo and Wikibase Client are both in bad shape design wise, making it hard to have them suit your needs (which are different than the requirements for Wikidata.org). This is something that should change regardless as it is slowing the Wikidata team down a lot, though it might well take some time before we got rid of the big issues there. I'm also not sure what you'd really gain by having our current entity storage rather than following a more conventional approach. You do not have a user defined "schema" right? I'm getting the impression that what you really want to do is store the data in tripplestore format. In case of the Wikibase applications, we'll be using Wikibase QueryEngine for this, which will maintain a second copy of the data that can be queried. Since this component is not bound to the Wikibase applications, you could potentially use it at some point (initial development has not quite finished yet). https://www.mediawiki.org/wiki/Wikibase/Components Using Wikibase QueryEngine would gain you some amount of interoperability, esp as you'd essentially be using our DataModel. I'm rather skeptical our DataModel makes much sense for your use case. You don't need Statements (that have a rank and references) and perhaps not even claims (which have qualifiers). This means you'll not have use for Item entities, which means you'd need a new type of entity. This does not mean you have to give up on interoperability altogether - you can still use lower level components. The wiki page mentions geo and time. You can indeed use our implementations for those by using the appropriate DataValues libraries (also linked from the Wikibase/Components page). There definitely is opportunity for reuse and interop here. I suggest we identify the reusable components of our system that are useful to you, as well as functionality that is not reusable yet, though would be beneficial for you if it was. Quoting from the wiki page itself: > The code produced during the first few sprints will probably be pretty straightforward. Still, it would be nice to have an architectural plan to aim for. Here's a rough proposal: I'm happy to see you are thinking about this. The diagram and text look reasonable enough, though neither explains the boundaries (arguably the most important part of an architecture) in detail. The dependencies presumably all go inwards towards the "campaigns business logic" part. In other words, this part does not depend on any of the others. The top layer contains parts that would each have their own application and domain logic, so these should definitely not be whole blobs that depend on the "reusable UI logic" in the layer below. I cannot stress enough how important it is to get that right - it makes a huge impact on the maintenance cost of the project and how flexible and powerful it ends up being. (Just look at the EducationProgram extension or at MW core to see how it can go wrong.) Quoting from the wiki page itself: > Question: Do we want to use existing frameworks at all? Composer? Anything from Symfony? Composer is a dependency manager (tool), not a framework. If your code is binding to it, you are very likely doing something wrong. I'm assuming you will be building this on top of MediaWiki, in which case you already have a framework. Generally it is good design practice to bind as little to a framework as possible (frameworks ought to take the same place as the database and the UI - at the edge, as an implementation detail that could theoretically be replaced). It is nicer to depend on libraries, which your application will control, rather than on a framework, which will control your application. One of the worst kind of bindings you can have to a framework is to derive from its concrete classes and put in your domain and application logic. This is what practically all MediaWiki extensions are doing with special pages, api modules, jobs, actions, etc. Those APIs are lousily designed, making binding to them even more problematic. For the Wikibase applications we made this same mistake, which was identified as such though review by external experts. In case of Symfony, which mostly adheres to sane design principles, the framework is based on libraries. Making use of those can definitely be beneficial to you. There is however no reason to limit yourself to Symfony - there are many great libraries out there that are not part of the Symfony project. Good luck with your project - I hope it turns into a gem rather than a ball of mud :) Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

Typing of globe-coordinate
by Maarten Dammers 01 Mar '14

01 Mar '14

Hi everyone, I ran into some pretty annoying problems with coordinates specifically the precision. The valid values of precision aren't properly documented and enforced at the moment, see https://bugzilla.wikimedia.org/show_bug.cgi?id=62105 I would like to know what is supposed to be valid so I know what direction to go in. Maarten

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech March 2014