Wikidata-tech

wikidata-tech@lists.wikimedia.org

1 participants
622 discussions

On interface stability and forward compatibility
by Daniel Kinzler 05 Feb '16

05 Feb '16

Hi all! In the context of introducing the new "math" and "external-id" data types, the question came up whether this introduction constitutes a breaking change to the data model. The answer to this depends on whether you take the "English" or the "German" approach to interpreting the format: According to <https://en.wikipedia.org/wiki/Everything_which_is_not_forbidden_is_allowed>, in England, "everything which is not forbidden is allowed", while, in Germany, the opposite applies, so "everything which is not allowed is forbidden". In my mind, the advantage of formats like JSON, XML and RDF is that they provide good discovery by eyeballing, and that they use a mix-and-match approach. In this context, I favour the English approach: anything not explicitly forbidden in the JSON or RDF is allowed. So I think clients should be written in a forward-compatible way: they should handle unknown constructs or values gracefully. In this vein, I would like to propose a few guiding principles for the design of client libraries that consume Wikibase RDF and particularly JSON output: * When encountering an unknown structure, such as an unexpected key in a JSON encoded object, the consumer SHOULD skip that structure. Depending on context and use case, a warning MAY be issued to alert the user that some part of the data was not processed. * When encountering a malformed structure, such as missing a required key in a JSON encoded object, the consumer MAY skip that structure, but then a warning MUST be issued to alert the user that some part of the data was not processed. If the structure is not skipped, the consumer MUST fail with a fatal error. * Clients MUST make a clear distinction of data types and values types: A Snak's data type determines the interpretation of the value, while the type of the Snak's data value specifies the structure of the value representation. * Clients SHOULD be able to process a Snak about a Property of unknown data type, as long as the value type is known. In such a case, the client SHOULD fall back to the behaviour defined for the value type. If this is not possible, the Snak MUST be skipped and a warning SHOULD be issued to alert the user that some part of the data could not be interpreted. * When encountering an unknown type of data value (value type), the client MUST either ignore the respective Snak, or fail with a fatal error. A warning SHOULD be issued to alert the user that some part of the data could not be processed. Do you think these guidelines are reasonable? It seems to me that adopting them should save everyone some trouble. -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

3 4

Long QIDs in Wikidata dump
by David Przybilla 08 Jan '16

08 Jan '16

While playing with the dump, I came accross some entries like the following: <http://www.wikidata.org/entity/Q12258SCD97A47E-A0CA-453F-B01A-DEE8829139BF> <http://www.wikidata.org/entity/P646v> "/m/021821" . what does the `Q12258SCD97A47E-A0CA-453F-B01A-DEE8829139BF` stand for ?

2 1

Fwd: 'help'
by Jorge Hernandez 30 Dec '15

30 Dec '15

---------- Forwarded message ---------- From: Jorge Hernandez <jorgeah2(a)uci.edu> Date: Wed, Dec 30, 2015 at 9:03 AM Subject: 'help' To: wikidata-tech-owner(a)lists.wikimedia.org Hello Wiki Community, My name is Jorge, and I am a researcher conducting a small project on Wikidata. I truly enjoy learning about this new dimension of knowledge, but like most of us, am having problems with querying and even installing Lua scribunto etc. My aim is to be able to learn how to extract geo-spatial attributes ( e.g., coordinates) for any given statement. Also, it will come in very handy if there was a link to a list or register of all the property and values available on Wikidata ( e.g., P31, Q2). Is this information available somewhere? I am still getting used to this listing and e-mail system. Hope to hear from you soon, Have a great day!!! Kindest Regards, Jorge Hernandez -- *Jorge A. HernandezResearch Intern* *University of California, Irvine* Schools of Social & Information Science Member, Sociology Honors Research Fellow , 2014-2015 Member, Dean's Council :: Academic | Philanthropic| Research|Social *University of Southern California* Spatial Science Institute Email:Jorgeah2@uci.edu Email:Jorgeahe@usc.edu Tel: 951-452-0395 -- *Jorge A. HernandezResearch Intern* Speedfind Technologies *University of California, Irvine* Schools of Social & Information Science Member, Sociology Honors Research Fellow , 2014-2015 Member, Dean's Council :: Academic | Philanthropic| Research|Social *University of Southern California* Spatial Science Institute Email:Jorgeah2@uci.edu Email:Jorgeahe@usc.edu Tel: 951-452-0395

1 0

schema of wikidata keys used in wikidata objects
by Luigi Assom 24 Dec '15

24 Dec '15

Hello, I am building a web site for knowledge discovery, and would like to understand more the structure of wikidata in order to expose selected keys for a wikidata object. *Is there any schema describing all the keys used in wikidata objects?* As example, this demo shows all possible keys for 'Richard_Feynman' object. http://earldouglas.github.io/wdqs-explorer/Richard_Feynman/ It looks like that for each property belonging to 'Richard_Feynman' a query must be made against the property_Id , so to fetch a human readable label. That is, about ~500Kb must be downloaded. I would like to understand better how to consume wikidata API. - which API exposes a simple schema 'key' : 'value' , like: 'birth_date' : 'Los Angeles' (similar to what you see at wikidata site) ? - which API expose the value of a relationships between objects ? (e.g. Feynman is connected to Cornell University by 'employer' relationship) - Is there a table describing all keys currently present in wikidata, so that is possible to know which one to filter ? thank you for your debrief, Luigi

2 1

Last call for objections against DataModel changes.
by Daniel Kinzler 07 Dec '15

07 Dec '15

A couple of weeks ago, I proposed to change our PHP data model bindings to allow extra info to be attached using the concept of "facets" simmilar to the "role object" and "extension object" pattern. Code experiments showcasing this idea can be found on github: * https://github.com/wmde/WikibaseDataModel/pull/576 * https://github.com/wmde/WikibaseDataModelSerialization/pull/174 This is the final call for objections against using this approach. The rationale behind it can be found on <https://phabricator.wikimedia.org/T118860> and related tickets. Implementation details can still change later, but after nearly 3 months, we finally need a decision on the conceptual level. If there are no substantial objections, this will become definite on Tuesday, December 8. -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

4 3

Re: [Wikidata-tech] Wikidata-tech Digest, Vol 31, Issue 8
by Christopher Johnson 30 Nov '15

30 Nov '15

OK, I try to make this clear. The use case is to be able to simply identify with a SPARQL query and/or count "unreferenced statements" using fastRangeCount. How do we do this with the current implementation? What this gains is a useful method for measuring data quality that does not exist now (as far as I can understand). And, it could also provide a stable URI (UUID) for references that would fix the questionable use of the "unstable" reference hash as a resource URI and facilitate the implementation of reusability for them. Thanks, Christopher On 30 November 2015 at 13:00, <wikidata-tech-request(a)lists.wikimedia.org> wrote: > Send Wikidata-tech mailing list submissions to > wikidata-tech(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/wikidata-tech > or, via email, send a message with subject or body 'help' to > wikidata-tech-request(a)lists.wikimedia.org > > You can reach the person managing the list at > wikidata-tech-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Wikidata-tech digest..." > > > Today's Topics: > > 1. Re: Wikidata-tech Digest, Vol 31, Issue 5 (Stas Malyshev) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 29 Nov 2015 16:54:29 -0800 > From: Stas Malyshev <smalyshev(a)wikimedia.org> > To: Wikidata technical discussion <wikidata-tech(a)lists.wikimedia.org> > Subject: Re: [Wikidata-tech] Wikidata-tech Digest, Vol 31, Issue 5 > Message-ID: <565B9E45.6090106(a)wikimedia.org> > Content-Type: text/plain; charset=utf-8 > > Hi! > > > In Blazegraph, this could be supported by Quads or RDR (Reification Done > > Right). > > We considered using RDR but decided against it because RDR is not > standard and existing tools and libraries would not understand it. So in > the interest of better data integration we decided to use regular RDF > representation that can be queries by standard SPARQL. > > > One possible approach using triples for the use case could be to assign > > a blank node to a reference placeholder and introduce the valid range > > class for prov:wasDerivedFrom (prov:entity) with the canonical reference > > UUID like this: > > I'm not sure I understand - what would doing this earn us? This looks > like just adding one more join to the lookups. > -- > Stas Malyshev > smalyshev(a)wikimedia.org > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Wikidata-tech mailing list > Wikidata-tech(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata-tech > > > ------------------------------ > > End of Wikidata-tech Digest, Vol 31, Issue 8 > ******************************************** >

1 0

Re: [Wikidata-tech] Wikidata-tech Digest, Vol 31, Issue 5
by Christopher Johnson 29 Nov '15

29 Nov '15

Thank you for the explanation. The content negotion for an Item IRI is clear. Any request for http://www.wikidata.org/entity/Q... requires an Accept application/rdf+xml header in order to get the RDF. The default response is JSON and Accept text/html returns a 200 response delivering the UI page. For statement resolution in the Item RDF, is not this a fragment? So in the Item context, the IRI for a statement resource would be http://www.wikidata.org/entity/Q16521#Statement_UUID. Otherwise, the statement IRI http://www.wikidata.org/entity/statement/Statement_UUID could just return the statement as a separate entity. On the topic of references, a use case is to measure data quality by counting the number of "unreferenced statements". At https://phabricator.wikimedia.org/T117234#1834728, I propose the possibility of using blank reference nodes to identify these "bad" statements. Having an object to count greatly expedites the query process because of the estimated cardinality feature of Blazegraph. The only alternative to this is to count distinct statements with the prov:wasDerivedFrom predicate, and this is extremely slow (in fact, it may not be possible without a huge amount of memory). I do not know what would be involved in implementing blank reference nodes and what performance consequences may also occur. It seems to me that the pairing of statements and references is a core feature of the data model, and it is odd that there can exist statements that have no associated reference node in the RDF. Cheers, Christopher On 27 November 2015 at 13:00, <wikidata-tech-request(a)lists.wikimedia.org> wrote: > Send Wikidata-tech mailing list submissions to > wikidata-tech(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/wikidata-tech > or, via email, send a message with subject or body 'help' to > wikidata-tech-request(a)lists.wikimedia.org > > You can reach the person managing the list at > wikidata-tech-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Wikidata-tech digest..." > > > Today's Topics: > > 1. RDF Item, Statement and Reference IRI Resolution? > (Christopher Johnson) > 2. Re: RDF Item, Statement and Reference IRI Resolution? > (Markus Krötzsch) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 27 Nov 2015 07:21:10 +0100 > From: Christopher Johnson <christopher.johnson(a)wikimedia.de> > To: wikidata-tech(a)lists.wikimedia.org, wikimedia-de-tech > <wikimedia-de-tech(a)wikimedia.de> > Subject: [Wikidata-tech] RDF Item, Statement and Reference IRI > Resolution? > Message-ID: > <CACzuuKvGK1dM1+dn4ypocjhO= > psuk4LLtWngZp1yFVP6wmVqFA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > After looking at the RDF format closely, I am asking if the item, statement > and reference IRIs could/should be directly resolvable to XML/JSON > formatted resources. > > It seems that currently http://www.wikidata.org/entity/.... redirects to > the UI at https://www.wikidata.org/wiki/ which is not what a machine > reader > would expect. > Without a simple method to resolve the IRIs (perhaps a RESTful API?), these > RDF data objects are opaque for parsers. > > Of course, with wbgetclaims, it is possible to get the statement like this: > > https://www.wikidata.org/w/api.php?action=wbgetclaims&format=xml&claim=Q209… > > but the API expected GUID format does not match the RDF UUID representation > (there is a $ or "%24" after the item instead of a -) and it returns both > the statement and the references. > > Since the reference is its own node in the RDF, it can be queried > independently. For example, to ask "return all of the statements where > reference R is bound." But then, the return value is a list of statement > IDs and a subquery or separate query is then required to return the > associated statement node. > > I am also wondering why item, statement and reference "UUIDs" are not in > canonical format in the RDF. This is a question of compliance with IETF > guidelines, which may or may not be relevant. > > Item: Q20913766 > Statement: Q20913766-CD281698-E1D0-43A1-BEEA-E2A60E5A88F1 > Reference: 39f3ce979f9d84a0ebf09abe1702bf22326695e9 > > See: https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format > See: http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml > and http://tools.ietf.org/html/rfc4122 for information on urn:uuid > guidelines. > > Thanks for your feedback, > Christopher >

2 2

RDF Item, Statement and Reference IRI Resolution?
by Christopher Johnson 27 Nov '15

27 Nov '15

Hi, After looking at the RDF format closely, I am asking if the item, statement and reference IRIs could/should be directly resolvable to XML/JSON formatted resources. It seems that currently http://www.wikidata.org/entity/.... redirects to the UI at https://www.wikidata.org/wiki/ which is not what a machine reader would expect. Without a simple method to resolve the IRIs (perhaps a RESTful API?), these RDF data objects are opaque for parsers. Of course, with wbgetclaims, it is possible to get the statement like this: https://www.wikidata.org/w/api.php?action=wbgetclaims&format=xml&claim=Q209… but the API expected GUID format does not match the RDF UUID representation (there is a $ or "%24" after the item instead of a -) and it returns both the statement and the references. Since the reference is its own node in the RDF, it can be queried independently. For example, to ask "return all of the statements where reference R is bound." But then, the return value is a list of statement IDs and a subquery or separate query is then required to return the associated statement node. I am also wondering why item, statement and reference "UUIDs" are not in canonical format in the RDF. This is a question of compliance with IETF guidelines, which may or may not be relevant. Item: Q20913766 Statement: Q20913766-CD281698-E1D0-43A1-BEEA-E2A60E5A88F1 Reference: 39f3ce979f9d84a0ebf09abe1702bf22326695e9 See: https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format See: http://www.iana.org/assignments/urn-namespaces/urn-namespaces.xhtml and http://tools.ietf.org/html/rfc4122 for information on urn:uuid guidelines. Thanks for your feedback, Christopher

2 1

Using the Role Object Pattern to represent derived information in the data model
by Daniel Kinzler 17 Nov '15

17 Nov '15

Hi all! For weeks and months now, we have been discussion how to best represent "extra" information in (or associated with) the wikibase data model. After some more discussion and a bit of research, I think I have found what we need: The Role Object Pattern aka Role Class Model, see <https://en.wikipedia.org/wiki/Role_Class_Model>. Please have a look at https://phabricator.wikimedia.org/T118860 and let me know if you have any objections. If not, let's use this sprint to discuss the details of the implementations, and do a task breakdown. PS: I came across quite a few famous names when during my research. Looks like we are not first in having this need... -- Daniel Kinzler Senior Software Developer Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.

1 0

EntityStore 1.0 and TermStore 1.0 released
by Jeroen De Dauw 16 Nov '15

16 Nov '15

Hey all, I've created two new small PHP libraries that provide persistence and lookup services for Wikibase data. You can read about them here http://www.bn2vs.com/blog/2015/11/14/entitystore-and-termstore-for-wikibase… Cheers -- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate ~=[,,_,,]:3

1 0

← Newer
1
...
32
33
34
35
36
37
38
...
63
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech