August 2013 - Wikidata-tech - lists.wikimedia.org

TravisCI integration with Gerrit
by Jeroen De Dauw 14 Jun '14

14 Jun '14

Hey Steffen and Andy, Continuing what I started on Twitter here, as some more characters might be helpful :) It seems that both our projects (FLOW3 and Wikidata) are in a similar situation. We are using Gerrit as CR tool, and TravisCI to run our tests. And we both want to have Travis run tests for all patchsets submitted to Gerrit, and then +1 or -1 on verified based on the build passing or failing. To what extend have you gotten such a thing to work on your project? Is there code available anywhere? If both projects can use the same code for this, I'd be happy to contribute to what you already have. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

2 2

UI for badges
by Michał Łazowik 06 Sep '13

06 Sep '13

Hi! I'm getting closer and closer to writing UI for badges in repo. Does anyone have any ideas? The only thing I came with is a additional row beneath each site link which behaves (and looks) like aliases editing currently does, but I think that this would take too much space. Thanks, Michał

4 12

IRI-value or string-value for URLs?
by Denny Vrandečić 03 Sep '13

03 Sep '13

We are planning to deploy URLs as data values rather soon (i.e. September 9, if all goes well). There was a discussion on wikidata-l mailing list: <http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg02664.html> The current implementation for URLs uses a string data value. There was also a IRI data value developed (for this use case), but in a previous (internal) discussion it was decided to use string value instead. The above thread included a few strong arguments by Markus for using the IRI data value. If we want to do this, we need to decide that very quickly, and change it accordingly. Let's see if we can make the decision here on this list. We need to make the decision by Monday latest, better earlier. Here are my current thoughts (check also the above mentioned thread if you did not have already). Currently I have a preference to using the string value, just to point out my current bias, but I want wider input. * I do not see the advantage of representing ' http://www.ietf.org/rfc/rfc1738.txt' as a structured data value of the form { protocol : 'http', hierarchicalpart : 'www.ietf.org/rfc/rfc1738.txt', query : '', fragment : '' }. * If we use string value, a number of necessary features come for free, like the diffing, displaying it in the diffs, etc. Sure, there is the argument that we can use the getString method for these, but then what is the use case that we actually serve by using the structured data? * I understood the advantages of being able to *identify* whether the value of a snak is a string or a URL, but that seems to be the same advantages as for knowing whether the value of a snak is a Commons media file name or a string. None of the the use cases though have been explaining why using the above data structure is advantageous over a simple string value. Please let us collect the arguments for and against using the IRI data value *structure* here (not for being able to *identify* whether a string is an IRI or a string). Not completely independent of that, there are a few questions that need to be answered but that are not as immediate, i.e. do not have to be decided by next week: * should, in the external JSON structure, for every snak the data value type be listed (as it currently is)? I.e. should it state "string" instead of "Commons media filename"? * should, in the external JSON structure, for every snak the data type of the property used be listed? This would then say URL, and this would solve all the use cases mentioned by Markus, which rely on *identifying* this distinction, not on the actual IRI data structure. * should, in the internal JSON structure, something be changed? The external JSON structure is the one used when communicating through the API. The internal JSON structure is the one that you get when using the dumps. We need to have an export of the whole Wikidata knowledge base in the external JSON format, rather sooner than later, and hopefully also in RDF. The lack of these dumps should not influence our decision right now, imho :) Cheers, Denny -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

5 13

Improvements to the implementation of Entity
by Jeroen De Dauw 02 Sep '13

02 Sep '13

Hey, Now the design issues of EntityId have been fixed, it's Entity's turn :) (Note: this is about domain layer implementation details. Not considering changing anything visible to the user here.) While working on the QueryEntity together with addshore, we ran into a number of issues with the current implementation of Entity. The main problem is that Entity objects are constructed from their internal "serialization". The constructor, which is marked as protected, takes in this "serialization" (in array form). This is rather awkward, consider how we now typically construct a Property: $property = Property::newEmpty(); $property->setDataTypeId( $id ); (We also have a static newFromDataTypeId which wraps this.) There is a bunch of code that assumes one can create empty Entity objects. Esp in tests. I now think it was a mistake to allow this at all for Property, which should not be constructed without a dataTypeId. The same goes for QueryEntity, which should not be constructed without a Ask\Query. It'd be much nicer if people could just use the constructors of the objects and have these enforce the list of required parameters. They'd just take the actual objects and not serializations. $property = new Property( $id ); $queryEntity = new QueryEntity( $askQuery ); And since these things are enforced, one now gets back a string when calling getDataTypeId, and a Ask\Query when calling getQueryDefinition, rather then either that type or null. Serialization and deserialization code can also go into dedicated service objects. This is already done in QueryEntity, which is using the same serializers as the ones the web API will use, saving us implementation of a second format, which would not be of much help here anyway (I'd save some disk space...). There is also a lot of room to be more strict about things. Right now you can happily construct a Entity that has ints as aliases, or as language code for labels. On top of that, there are currently still TODOs from the first months of the project in Entity related to normalization and handling of duplicates. We might want to clearly define responsibilities at this point :) Oh and, of course Entity, Item, Property and Query each should go into their own git repo. Any objections or concerns about the above rambling? There lately has also been some talking about doing things with Entity that we did not consider before. Such as entities that contain other entities. Is there a list of such thoughts? If not, lets compile one here so these can be held into account. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

2 3

DataValues and related components reorganization
by Jeroen De Dauw 30 Aug '13

30 Aug '13

Hey, This email is meant to provide an overview of the plans regarding the reorganization of the components in the DataValues.git repository. Current component situation: * DataValues * ValueParsers, depends on DataValues * ValueValidators, depends on DataValues * ValueFormatters, depends on DataValues * DataTypes, depends on all the above * ValueView, depends on all the above All of these are bundling inheritance hierarchies and are both defining interfaces as well as complex implementations. Reorganization plans: * DataValues, will hold interfaces, exceptions and trivial implementations of current DataValues * DataValues interfaces (still need a good name for this), will hold interfaces, exceptions and trivial implementations of ValueParsers, ValueFormatters and ValueValidators. Depends on DataValues * DataValues implementations (still need a good name for this), will hold common non-trivial implementations of the interfaces defined by the above two components * DataTypes, unchanged, now only dependent on DataValues and DataValues interfaces * ValueView, unchanged, now only dependent on DataTypes, DataValues and DataValues interfaces Dependencies are thus minimized and users are no longer forced to depend on unstable concrete classes for no reason. Coincidentally the number of components also drops by one. Git repository wise, everything is currently in a single repository. Each component will go into its own repo, with the exception of ValueView and DataTypes, which we'll at least initially put together. This means creation of 3 new git repos. The DataTypes git repo has already been created and we are awaiting removal of the old DataTypes code from DataValues.git which currently is blocked by WMF configuration update. Once this is done we can proceed with the remaining two repos. When this reorganization is done and the components reside in their own repos, we can make the two abstract ones releasable. These are the ones most dependent upon, and some of the current users have their own releases blocked due to the lack of any released version of their DataValues dependencies. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 2

Build step before deployment?
by Denny Vrandečić 30 Aug '13

30 Aug '13

Hello, currently, there is a strong interdependence between the Wikibase development and Wikidata deployment, which caused and causes friction. Sorry for that. A suggestion that was made yesterday by Rob was to introduce a build-step for deployment of Wikidata (something ULS already seems to be doing). If done right, this would allow to decouple the way components are split from the way components are deployed. If I understood it correctly, we would basically have either a deployment module or a deployment branch on an existing module, which (preferably) automatically gathers all dependencies, e.g. into a lib/ or dependency/ folder, or similar, and remains the sole module to be deployed. There are a number of details to be decided obviously, but I wanted first to gather consensus on whether this is the way forward for the short-term. If so, we would start to work on this very soon. Cheers, Denny -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

1 0

Re: [Wikidata-tech] Clarifying role of different Wikidata extensions (Re: Loading DataTypes in WMF config)
by Rob Lanphier 29 Aug '13

29 Aug '13

Hi Denny, On Tue, Aug 27, 2013 at 4:27 AM, Denny Vrandečić < denny.vrandecic(a)wikimedia.de> wrote: > 2013/8/27 Rob Lanphier <robla(a)wikimedia.org> >> On Mon, Aug 26, 2013 at 9:26 AM, Denny Vrandečić >> <denny.vrandecic(a)wikimedia.de> wrote: >> > We are indeed not yet using the RFC process, but I would prefer if we >> > could >> > agree to move to this process in the future, as this particular >> > discussion >> > is going on already a bit longer than what I would expect for a >> > discussion >> > regarding the question about how to organize code. >> >> >> This is not just about how we organize code. This is about how we >> package our work. It's unusual for us to have extensions with hard >> dependencies on other extensions, let alone the complicated hierarchy >> you all have chosen. It may be ok to do that, but we should discuss >> it before setting the precedent. >> > > This was already discussed here: > <http://www.mail-archive.com/wikitech-l@lists.wikimedia.org/msg69983.html> Yes, it was discussed. I was traveling the week that thread broke out, and never fully caught up on that particular thread (only fully read it this morning), so my apologies. That said, it didn't appear to me that we actually resolved anything there. Did I miss something? > Scribunto and Babel are other extensions that have dependencies, and there > are more of them. We are not setting a sole precedent here. Scribunto has optional integration with WikiEditor, SyntaxHighlight GeSHi, CodeEditor. The latter three all are useful extensions to end users in their own right, and Scribunto works fine without them. Moreover, I'm comfortable that I could walk around the office and get a coherent and mostly correct explanation about what all of these extensions do from many (if not most) developers here. I'm not as familiar with the dependencies that Babel has. In my cursory look, the situation looks similar to Scribunto's dependencies. The description for Babel could be a bit better, but it's sufficient for me to know what end-user functionality it's providing. I'm not saying that, because we haven't really managed dependencies between internal libraries before that we can't do it now. However, I don't think your examples support your case that this is already standard practice. >> This gets exposed to end users via Special:Version: >> https://en.wikipedia.org/wiki/Special:Version >> >> Since MediaWiki administrators often use this as a means of >> understanding how to configure their wikis "like Wikipedia", it would >> be nice if we didn't clutter that page up with a lot of the internals >> of our systems. Each of the links should point to a page that does a >> good job of describing what the extension does, and the vast majority >> of them do. Unfortunately, for most of the Wikidata extensions right >> now, the pages are pretty much boilerplaite plus a one-liner in many >> cases. >> > > I just checked a random sample of other extensions, and most of them are > just boilerplate plus a one-liner. I just started at the bottom: > <https://www.mediawiki.org/wiki/Extension:ZeroRatedMobileAccess> > <https://www.mediawiki.org/wiki/Extension:WikimediaMessages> > <https://www.mediawiki.org/wiki/Extension:WikimediaShopLink> Starting at the bottom isn't a representative sample. Moreover, the documentation for ZeroRatedMobileAccess is mostly in the clearly linked README which is not the worst place for it. Most of the spot checking I did on other extensions, there was sufficient information there for me to understand the essential functionality provided. > Also, this is a very different point than raised before, and we have, in > several places tried to improve our documentation, as e.g. by creating this > page: > <https://www.mediawiki.org/wiki/Wikibase> > But the quality of our documentation seems to be a shift in the focus of > this discussion. I would be happy if we could define well what the actual > point of discussion is, so that we can resolve it as soon. I'm trying to understand the breakup of the extensions, and in our continuing discussion, you've pointed me at varying bits of documentation that don't answer the questions that I have. I'd like to understand what exactly is so terrible about the status quo that you all are blocked on your development. If this refactoring is really so urgent, why can't you clearly and concisely state not only what you are doing, but *why* you are doing it. >> Rather than get too far into the implementation details of what your >> extensions are doing, I think maybe I'll hold off until someone on my >> team has more time to think about this and comment on it. >> > > As long as this does not contradict your other mail, where we said not to > further delay the refactoring of the DataValues-related extensions, sure. Well, we can move forward with this specifically: https://gerrit.wikimedia.org/r/#/c/76481/ I would hope you can hold off on the other refactoring until there is at least one person on my team who can confidently explain what the role of each of the extensions you're proposing is. I was going to bite the bullet and just get my head around it myself, but I need to be realistic about the level of effort I can expend with this. Rob

2 1

Fwd: [Wikitech-l] New search backend live on mediawiki.org
by Daniel Kinzler 28 Aug '13

28 Aug '13

Did I say it'll take until next year before we have the new search infrastructure? Looks like I was wrong, it's being beta tested on mediawiki.org already. Time to check how well it works with wikibase, then! Let's ask about that in the call tomorrow. -- daniel -------- Original-Nachricht -------- Betreff: [Wikitech-l] New search backend live on mediawiki.org Datum: Wed, 28 Aug 2013 14:20:10 -0400 Von: Nikolas Everett <neverett(a)wikimedia.org> Antwort an: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> An: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> Today we threw the big lever and turned on our new search backend at mediawiki.org. It isn't the default yet but it is just about ready for you to try. Here is what is we think we've improved: 1. Templates are now expanded during search so: 1a. You can search for text included in templates 1b. You can search for categories included in templates 2. The search engine is updated very quickly after articles change. 3. A few funky things around intitle and incategory: 3a. You can combine them with a regular query (incategory:kings peaceful) 3b. You can use prefix searches with them (incategory:norma*) 3c. You can use them everywhere in the query (roger incategory:normans) What we think we've made worse and we're working on fixing: 1. Because we're expanding templates some things that probably shouldn't be searched are being searched. We've fixed a few of these issues but I wouldn't be surprised if more come up. We opened Bug 53426 regarding audio tags. 2. The relative weighting of matches is going to be different. We're still fine tuning this and we'd appreciate any anecdotes describing search results that seem out of order. 3. We don't currently index headings beyond the article title in any special way. We'll be fixing that soon. (Bug 53481) 4. Searching for file names or clusters of punctuation characters doesn't work as well as it used to. It still works reasonably well if you surround your query in quotes but it isn't as good as it was. (Bugs 53013 and 52948) 5. "Did you mean" suggestions currently aren't highlighted at all and sometimes we'll suggest things that aren't actually better. (Bugs 52286 and 52860) 6. incategory:"category with spaces" isn't working. (Bug 53415) What we've changed that you probably don't care about: 1. Updating search in bulk is much more slow then before. This is the cost of expanding templates. 2. Search is now backed by a horizontally scalable search backend that is being actively developed (Elasticsearch) so we're in a much better place to expand on the new solution as time goes on. Neat stuff if you run your own MediaWiki: CirrusSearch is much easier to install than our current search infrastructure. So what will you notice? Nothing! That is because while the new search backend (CirrusSearch) is indexing we've left the current search infrastructure as the default while we work on our list of bugs. You can see the results from CirrusSearch by performing your search as normal and adding "&srbackend=CirrusSearch" to the url parameters. If you notice any problems with CirrusSearch please file bugs directly for it: https://bugzilla.wikimedia.org/enter_bug.cgi?product=MediaWiki%20extensions… Nik Everett _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

2 1

Database schema updating code review
by Jeroen De Dauw 27 Aug '13

27 Aug '13

Hey, I wrote up a little design spike for a bunch of code needed for the store schema update functionality. This is nothing big or out of the ordinary [0]. Since I am only continuing with this code later this week, and no real implementation has been done so far, this commit now lends itself well to a design level review. https://gerrit.wikimedia.org/r/#/c/81254/3/src/TableDefinitionReader.php [0] Someone please suggest writing a formal RFC, have 5 calls about it, a dedicated architecture review, a dedicated security review, and perhaps its own mailing list. Cheers -- Jeroen De Dauw http://www.bn2vs.com *Don't panic*. Don't be evil. ~=[,,_,,]:3 --

1 0

Loading DataTypes in WMF config
by Jeroen De Dauw 27 Aug '13

27 Aug '13

Hey, We have a bunch of development TODOs which are blocked by WMF deployment config being updated. Aude made the following commit 3 weeks ago. Can someone with the appropriate rights please have a look at it? https://gerrit.wikimedia.org/r/#/c/76481/ Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

3 8

Clarifying role of different Wikidata extensions (Re: Loading DataTypes in WMF config)
by Rob Lanphier 27 Aug '13

27 Aug '13

Hi Denny, Comments below: On Mon, Aug 26, 2013 at 9:26 AM, Denny Vrandečić <denny.vrandecic(a)wikimedia.de> wrote: > 2013/8/26 Rob Lanphier <robla(a)wikimedia.org> >> On Fri, Aug 23, 2013 at 8:03 AM, Denny Vrandečić >> <denny.vrandecic(a)wikimedia.de> wrote: >> > for Semantic MediaWiki we had this already listed for a while here: >> > <https://meta.wikimedia.org/wiki/Wikidata/Notes/SMW_and_Wikidata> >> > The list is not completely up-to-date, it reflects the previous package >> > structure. >> >> It would be helpful if there was a wiki page that reflected the >> current state of things. For MediaWiki core, we're trying to get into >> the practice of having RFCs for changes of this magnitude, and >> something of that level of rigor would be nice here too. > > > The current state is described here: > <http://www.mediawiki.org/wiki/Wikibase> , more specifically here: > <http://www.mediawiki.org/wiki/Wikibase#Packages_and_code_structure> > > The new structure was described in Emails on Wikidata-tech, most recently > here: > <http://www.mail-archive.com/wikidata-tech@lists.wikimedia.org/msg00200.html> > > We are indeed not yet using the RFC process, but I would prefer if we could > agree to move to this process in the future, as this particular discussion > is going on already a bit longer than what I would expect for a discussion > regarding the question about how to organize code. This is not just about how we organize code. This is about how we package our work. It's unusual for us to have extensions with hard dependencies on other extensions, let alone the complicated hierarchy you all have chosen. It may be ok to do that, but we should discuss it before setting the precedent. This gets exposed to end users via Special:Version: https://en.wikipedia.org/wiki/Special:Version Since MediaWiki administrators often use this as a means of understanding how to configure their wikis "like Wikipedia", it would be nice if we didn't clutter that page up with a lot of the internals of our systems. Each of the links should point to a page that does a good job of describing what the extension does, and the vast majority of them do. Unfortunately, for most of the Wikidata extensions right now, the pages are pretty much boilerplaite plus a one-liner in many cases. Rather than get too far into the implementation details of what your extensions are doing, I think maybe I'll hold off until someone on my team has more time to think about this and comment on it. Rob

1 0

EntityId changes
by Jeroen De Dauw 26 Aug '13

26 Aug '13

Hey, After discussion with Danwe and Denny I decided to rewrite EntityId. It was the highest risk class according to PHPUnit (290 CRAP) in the DataModel component, which really is bad considering this is one of our most basic classes, which a lot of other code binds itself to. The goal of rewriting it is getting rid of needless complexity, needless bindings and making non-problematic use of the class easier. The rewrite currently includes: * Dropping of id prefix configurability (as previously discussed) * Separation of the id class and the entity id DataValue (EntityId no longer derives from DataValue. A new EntityIdDataValue composites in a EntrityId) * The EntityId class now has as fields entity type (string) and id serialization (string) * New ItemId and PropertyId derivatives, which in the constructor only take the id serialization. The long term goal is to make EntityId abstract * Serialization format compatibility where required. The format of the DataValue will remain the same for now Comments on the work in progress are welcome: https://gerrit.wikimedia.org/r/#/c/80394/ It is possible this change will be done by tomorrow. More likely it'll only be merged on Monday or later. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

3 13

Re: [Wikidata-tech] [Wikidata-l] claims Datatypes inconsistency suspicion
by Daniel Kinzler 26 Aug '13

26 Aug '13

Am 25.08.2013 19:19, schrieb Markus Krötzsch: > If we have an IRI DV, considering that URLs are special IRIs, it seems clear > that IRI would be the best way of storing them. The best way of storing them really depends on the storage platform. It may be a string or something else. I think the real issue here is that we are exposing something that is really an internal detail (the data value type) instead of the high level information we actually should be exposing, namely property type. I think splitting the two was a mistake, and I think exposing the DV type while making the property type all but inaccessible makes things a lot worse. In my opinion, data should be self-descriptive, so the *semantic* type of the property should be included along with the value. People expect this, and assume that this is what the DV type is. But it's not, and should not be used or abused for this purpose. Ideally, it should not matter at all to any 3rd party if use use a string or IRI DV internally. The (semantic) property type would be URL, and that's all that matters. I'm quite unhappy about the current situation; we are beginning to see the backlash of the decision not to include the property type inline. If we don't do anything about this now, I fear the confusion is going to get worse. -- daniel

3 4

Drop id prefix configurability
by Jeroen De Dauw 26 Aug '13

26 Aug '13

Hey, Are there any objections against, or concerns with, dropping the configurability of entity id prefixes? Hardcoding these in their associated entity classes will make a lot of things easier. To clarify, entity id prefixes are things such as the "Q" in "Q42" for item ids or the "P" in "P1337" in property ids. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

3 2

Registering Value Formatters
by Daniel Kinzler 26 Aug '13

26 Aug '13

Since this topic came up again during the daily standup today, I'd like to share a few links. We have discussed how to best manage and register formatters before. A very brief summary of our discussion is here: * https://meta.wikimedia.org/wiki/Wikidata/Development/Data_values The relevant tickets are: * https://bugzilla.wikimedia.org/show_bug.cgi?id=49265 * https://bugzilla.wikimedia.org/show_bug.cgi?id=51799 Please consider these when working on value formatters. Thanks! -- daniel

1 0

Using @covers tags
by Jeroen De Dauw 25 Aug '13

25 Aug '13

Hey, I'd like to reiterate over the importance of using @covers tags [0] for test cases rather then writing "Test for SomeClass". These tags make coverage reporting a lot more accurate, and thus make it easier to spot areas lacking tests. Today I added a bunch of @covers tags in the DataModel component. The reported coverage dropped from 85% to 57%, quite a difference. [0] http://phpunit.de/manual/current/en/appendixes.annotations.html#appendixes.… Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

3 2

Fwd: Java-based Wiktionary Library (JWKTL) 1.0.0 released as open source software
by Sumana Harihareswara 23 Aug '13

23 Aug '13

Forwarding to the Wikidata tech list in case this makes a future Wiktionary collaboration easier. -------- Original Message -------- Subject: [Wiki-research-l] Java-based Wiktionary Library (JWKTL) 1.0.0 released as open source software (Wiki-research-l Digest, Vol 96, Issue 22) Date: Tue, 20 Aug 2013 14:20:56 +0000 From: Judith Eckle-Kohler <eckle-kohler(a)ukp.informatik.tu-darmstadt.de> Reply-To: wiki-research-l(a)lists.wikimedia.org To: wiki-research-l(a)lists.wikimedia.org <wiki-research-l(a)lists.wikimedia.org> [Apologies for X-posting] We are pleased to announce the release of the Java-based Wiktionary Library (JWKTL) 1.0.0 - an application programming interface for Wiktionary. Project homepage: http://code.google.com/p/jwktl/ == Overview == JWKTL (Java-based Wiktionary Library) is an application programming interface for the free multilingual online dictionary Wiktionary (http://www.wiktionary.org). JWKTL enables efficient and structured access to the information encoded in the English, the German, and the Russian Wiktionary language editions, including sense definitions, part of speech tags, etymology, example sentences, translations, semantic relations, and many other lexical information types. The Russian JWKTL parser is based on Wikokit (http://code.google.com/p/wikokit/). Prior to being available as open source software, JWKTL has been a research project at the Ubiquitous Knowledge Processing (UKP) Lab of the Technische Universität Darmstadt, Germany. The following people have mainly contributed to this project: Yevgen Chebotar, Iryna Gurevych, Christian M. Meyer, Christof Müller, Lizhen Qu, Torsten Zesch. == Publications == A detailed description of Wiktionary and JWKTL is available in our scientific articles: * Christian M. Meyer and Iryna Gurevych: Wiktionary: A new rival for expert-built lexicons? Exploring the possibilities of collaborative lexicography, Chapter 13 in S. Granger & M. Paquot (Eds.): Electronic Lexicography, pp. 259291, Oxford: Oxford University Press, November 2012. (http://www.ukp.tu-darmstadt.de/publications/details/?no_cache=1&tx_bibtex_p…) * Christian M. Meyer and Iryna Gurevych: OntoWiktionary Constructing an Ontology from the Collaborative Online Dictionary Wiktionary, chapter 6 in M. T. Pazienza and A. Stellato (Eds.): Semi-Automatic Ontology Development: Processes and Resources, pp. 131161, Hershey, PA: IGI Global, February 2012. (http://www.ukp.tu-darmstadt.de/publications/details/?no_cache=1&tx_bibtex_p…) * Torsten Zesch, Christof Müller, and Iryna Gurevych: Extracting Lexical Semantic Knowledge from Wikipedia and Wiktionary, in: Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC), pp. 16461652, May 2008. Marrakech, Morocco. (http://www.ukp.tu-darmstadt.de/publications/details/?no_cache=1&tx_bibtex_p…) == License and Availability == The latest version of JWKTL is available via Maven Central. If you use Maven as your build tool, then you can add JWKTL as a dependency in your pom.xml file: <dependency> <groupId>de.tudarmstadt.ukp.jwktl</groupId> <artifactId>jwktl</artifactId> <version>1.0.0</version> </dependency> JWKTL is available as open source software under the Apache License 2.0 (ASL). The software thus comes "as is" without any warranty (see license text for more details). JWKTL makes use of Berkeley DB Java Edition 5.0.73 (Sleepycat License), Apache Ant 1.7.1 (ASL), Xerces 2.9.1 (ASL), JUnit 4.10 (CPL). Some classes have been taken from the Wikokit project (available under multiple licenses, redistributed under the ASL license). See NOTICE.txt for further details. == Contact == Please direct any questions or suggestions to https://groups.google.com/forum/#!forum/jwktl-users Group E-Mail: jwktl-users(a)googlegroups.com Best wishes, Christian M. Meyer -- Christian M. Meyer, M.Sc. Doctoral Researcher Ubiquitous Knowledge Processing (UKP Lab) FB 20 Computer Science Department Technische Universität Darmstadt Hochschulstr. 10, D-64289 Darmstadt, Germany Phone [+49] (0)6151 16-5386, fax -5455, room S2/02/B113 meyer(a)ukp.informatik.tu-darmstadt.de www.ukp.tu-darmstadt.de<http://www.ukp.tu-darmstadt.de> Web Research at TU Darmstadt (WeRC) www.werc.tu-darmstadt.de<http://www.werc.tu-darmstadt.de>

1 0

Fwd: Re: [Wikitech-l] unexpected error info in HTML
by Jeremy Baron 23 Aug '13

23 Aug '13

---------- Forwarded message ---------- From: "Sumana Harihareswara" <sumanah(a)wikimedia.org> Date: Aug 22, 2013 7:06 PM Subject: Re: [Wikitech-l] unexpected error info in HTML To: "Wikimedia developers" <wikitech-l(a)lists.wikimedia.org>, "Jiang BIAN" < bianjiang(a)google.com> Cc: > On 08/01/2013 03:08 AM, Jiang BIAN <bianjiang(a)google.com> wrote: > > Hi, > > > > I noticed some pages we crawled containing error message like this; > > > > <div id="mw-content-text" lang="zh-CN" dir="ltr" class="mw-content-ltr"><p > > class="error">Failed to render property P373: > > Wikibase\LanguageWithConversion::factory: given languages do not have the > > same parent language</p> > > > > > > But when I open the url in browser, there is no such message. And using > > index.php can also get normal content without error messages. > > > > Here are examples you can retry: > > > > bad > > $ wget 'http://zh.wikipedia.org/zh-cn/Google' > > > > good > > $ wget 'http://zh.wikipedia.org/w/index.php?title=Google' > > > > > > Looks like something is wrong on Wikipedia side, anything need to fix? > > > > > > > > Thanks > > I checked with Jiang Bian and found out that this is still happening -- > can anyone help Google out here? :-) > > -- > Sumana Harihareswara > Engineering Community Manager > Wikimedia Foundation > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Updated doxygen
by Katie Filbert 22 Aug '13

22 Aug '13

I have updated doxygen documentation for Wikibase and added it for some related extensions. (some query-related components are still missing) http://wikidata-docs.wikimedia.de/ If this is useful, I can add remaining components, make it look nicer and set it up to update regularly. Cheers, Katie -- Katie Filbert Wikidata Developer Wikimedia Germany e.V. | NEW: Obentrautstr. 72 | 10963 Berlin Phone (030) 219 158 26-0 http://wikimedia.de Wikimedia Germany - Society for the Promotion of free knowledge eV Entered in the register of Amtsgericht Berlin-Charlottenburg under the number 23 855 as recognized as charitable by the Inland Revenue for corporations I Berlin, tax number 27/681/51985.

2 1

Looking for stuff to do? Here you go!
by Lydia Pintscher 20 Aug '13

20 Aug '13

Hey :) For the unlikely case that some readers of this list aren't aware of this yet: There are a lot of Wikidata bugs in bugzilla that could use some help. They're marked with the keyword need-volunteer. You can see them here: https://bugzilla.wikimedia.org/buglist.cgi?keywords=need-volunteer%2C%20&ke… (Obviously you're welcome to work on other stuff too but if you don't know what to put time into this is the perfect list.) If you're working on any of those please leave a comment in the bug report so others know you're on it. In case you're more into writing a bot then http://www.wikidata.org/wiki/Wikidata:Bot_requests is the place to look. If you need help here and the IRC channel #wikimedia-wikidata on freenode is a good place to ask. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Community Communications for Technical Projects Wikimedia Deutschland e.V. Obentrautstr. 72 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

1 0

badges support - decision needed?
by Lydia Pintscher 16 Aug '13

16 Aug '13

Hey :) It seems that badges support is stalled on a decision about how exactly to define the set of available badges if I read https://bugzilla.wikimedia.org/show_bug.cgi?id=40810 correctly. Can we make a decision and move forward? It's the most voted on bug we have. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Community Communications for Technical Projects Wikimedia Deutschland e.V. Obentrautstr. 72 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

3 3

Introduced 'order' parameter in ByPropertyListSerializer
by Jeroen De Dauw 15 Aug '13

15 Aug '13

Hey, Regarding https://gerrit.wikimedia.org/r/#/c/77862/4 The approach implemented in PS4 is different then the one I was thinking of originally when we discussed this. It has { qualifiers: { snaks: {}, order: [] } } rather then { qualifiers: {}, qualifiers-order: [] } While I agree that the former is nicer, it is a breaking change in our API. I'm thus not merging it as long as it is not clear that we are all well aware of this breakage and find it an acceptable price for having a nicer format. If we decide to go forward with this, it should also be announced to our API users and properly documented. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

How to convert Wikidata RDF to be consumable by the Wikidata edit API?
by Claus Stadler 14 Aug '13

14 Aug '13

Hi everyone, Summary: We are looking for ways of how to map individual triples from DBpedia to something that can be consumed by the Wikidata API. Long story: Denis Lukovnikov is a Google summer of code student, working on new and pretty user interface for DBpedia. A demo of the work in progress can be found at [1]. Besides the new design, a further goal of the project is to offer a framework for triple-level actions. One of the planned actions is about pushing DBpedia triples to Wikidata: For example, I notice that the Wikidata page for my home town "Berndorf in Lower Austria" does not contain the population: http://www.wikidata.org/wiki/Q666615 Looking at the corresponding DBpedia entry, this information actually exists there: http://dbpedia.org/resource/Berndorf,_Lower_Austria The new DBpedia interface should offer a button next to the "population 8728" triple which enables transfer of this information to Wikidata. In another GSoC project, Hady Elsahar is working on mappings between the wikidata RDF vocabulary and the DBpedia vocabulary. This means, we can in principle map DBpedia RDF data to Wikidata RDF. However, looking at the Wikidata API [2] there is action=wbcreateclaim * with the example: api.php?action=wbcreateclaim&entity=q42&property=p9001&snaktype=novalue&token=foobar&baserevid=7201010 So the core question is, how can we map e.g. properties such as wikidata:population (if that existed) to their respective Wikidata property identifier (Q12345)? This goes for any property that may occur in an RDF dump, such as: http://www.wikidata.org/wiki/Special:EntityData/Q666615.nt Ideally I think we are looking for either a Wikidata service or a dataset similar to [3] what is what we have done in the LinkedGeoData Project: In this project, we map OpenStreetMap (OSM) data to RDF, and we maintain our RDF mappings in the database [3] together with the original OSM data. Then we use an RDB2RDF mapper configured with the view definitions in [4] to expose the relational database as Linked Data [5] and a virtual SPARQL endpoint [6]. Cheers, Claus [1] WiP demo: http://dbpv.cstadler.aksw.org/#/page/Berndorf,_Lower_Austria (currently loading may take a while for pages with many inverse statements). [2] WikiData API: http://www.wikidata.org/w/api.php [3] Raw-Data to RDF mappings for OpenStreetMap: https://github.com/GeoKnow/LinkedGeoData/blob/master/linkedgeodata-core/src… [4] RDB2RDF view definitions: https://github.com/GeoKnow/LinkedGeoData/blob/master/linkedgeodata-core/src… [5] Eiffel Tower resource: http://linkedgeodata.org/page/triplify/way5013364 [6] Explanation of the unterlying SQL: http://linkedgeodata.org/vsnorql/?query=Explain+Select+*+{+%3Chttp%3A%2F%2F… -- Dipl. Inf. Claus Stadler Department of Computer Science, University of Leipzig Research Group: http://aksw.org/ Workpage & WebID: http://aksw.org/ClausStadler Phone: +49 341 97-32260

3 4

Dump problem?
by Markus Krötzsch 09 Aug '13

09 Aug '13

Hi, there seems to be a problem with the dumps or with the way in which we interpret the dumps. Currently, daily dumps come with a maxrevid.txt file that is supposed to give the largest revision number in the dump. For example, daily dump of 1st Aug 2013 has max id 62860640 [1]. I guess that's true. The wda scripts (that also create the statistics and digested dumps we publish) use this number to figure out if a daily is still relevant or if it is already contained in the latest full dump. For this, we need to get the maximal revision id of the full dump. We do this by reading the file site_stats.sql.gz, where we look for the line starting with INSERT INTO `site_stats` and take a revision number from there (third number in the insert tuple). For example, for the dump of 27 July 2013, this number is 63069374 [2]. There is a problem here, since the maximal revision in the dumps of 27 July 2013 is not actually that high (the history dump of that date is incomplete, but the current revs dump is done and has max rev 61983867 [3]). Thus, our scripts ignore several days of dailies. Before I go and work on this, my question is whether this is an error in our script (i.e., the number we take from sitestats is not supposed to be the max revision) or an error in the dumps (i.e., sitestats was exported wrongly). Cheers, Markus [1] http://dumps.wikimedia.org/other/incr/wikidatawiki/20130801/maxrevid.txt [2] http://dumps.wikimedia.org/wikidatawiki/20130727/wikidatawiki-20130727-site… [3] This can be seen in the comments for the dump at http://dumps.wikimedia.org/wikidatawiki/20130727/ -- Markus Kroetzsch, Departmental Lecturer Department of Computer Science, University of Oxford Room 306, Parks Road, OX1 3QD Oxford, United Kingdom +44 (0)1865 283529 http://korrekt.org/

1 0

Dependency status badge
by Jeroen De Dauw 07 Aug '13

07 Aug '13

Hey, While looking at some projects on GitHub, I came accross one that had a "dependencies up to date badge". This badge was provided by https://www.versioneye.com/ and based on the package.json file of the JS project in question. Since this service also supports Composer, I went ahead and tried using this for Ask. https://github.com/wikimedia/mediawiki-extensions-Ask/blob/master/README.md \o/ Also see the page you get when clicking the badge: https://www.versioneye.com/php/ask:ask Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

JavaScript implementation of Ask
by Jeroen De Dauw 06 Aug '13

06 Aug '13

Hey, I wanted to play a bit with some JavaScript things and decided to start implementing the Ask language in JS. Though I have not gotten far yet in actual implementation, I had a lot of fun setting up the build process and CI tools. https://github.com/JeroenDeDauw/AskJS The tests can be run on the command line (unlike the ones we have for Wikibase), they are run by TravisCI as well, and code coverage reports are created by each Travis build using coveralls.io \o/ I plan to try out Grunt and see how that compares to Jake, and look into how to make this work in browsers as well, and setting up the CI so it verifies it does. (I am currently doing this in my free time, not as WMDE employee) Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --

1 0

Re: [Wikidata-tech] [Wikitech-l] Is assert() allowed?
by Daniel Kinzler 05 Aug '13

05 Aug '13

My take on assertions, which I also tried to stick to in Wikibase, is as follows: * A failing assertion indicates a "local" error in the code or a bug in PHP; They should not be used to check preconditions or validate input. That's what InvalidArgumentException is for (and I wish type hints would trigger that, and not a "fatal error"). Precondition checks can always fail, never trust the caller. Assertions are things that should *always* be true. * Use assertions to check postconditions (and perhaps invariants). That is, use them to assert that the code in the method (and maybe class) that contains the assert is correct. Do not use them to enforce caller behavior. * Use boolean expressions in assertions, not strings. The speed advantage of strings is not big, since the expression should be a very basic one anyway, and strings are awkward to read, write, and, as mentioned before, potentially dangerous, because they are eval()ed. * The notion of "bailing out" on "fatal errors" is a misguided remnant from the days when PHP didn't have exceptions. In my mind, assertions should just throw an (usually unhandled) exception, like Java's AssertionError. I think if we stick with this, assertions are potentially useful, and harmless at worst. But if there is consensus that they should not be used anywhere, ever, we'll remove them. I don't really see how the resulting boiler plate would be cleaner or safer: if ( $foo > $bar ) { throw new OMGWTFError(); } -- daniel Am 31.07.2013 00:28, schrieb Tim Starling: > On 31/07/13 07:28, Max Semenik wrote: >> I remeber we discussed using asserts and decided they're a bad >> idea for WMF-deployed code - yet I see >> >> Warning: assert() [<a href='function.assert'>function.assert</a>]: >> Assertion failed in >> /usr/local/apache/common-local/php-1.22wmf12/extensions/WikibaseDataModel/DataModel/Claim/Claims.php >> on line 291 > > The original discussion is here: > > <http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/59620> > > Judge for yourself. > > -- Tim Starling > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >

7 6

302 vs 303 response code
by Rob Lanphier 03 Aug '13

03 Aug '13

Hi folks, On our WMF/WMDE Wikidata coordination call, Daniel suggested we revisit bug 44098 [1], asserting that /entity/Q1234 (for example) should trigger a 303, not a 302 HTTP status code. In reading the HTTP spec for this, it would appear that 303 expressly prohibits caching the response[2]. I'm not sure if that's going to be a problem, since I don't know if we do any 302 response caching (note it's just the 303 redirect itself that can't be cached, not the target). However, the spec is also pretty ambiguous about whether 302 or 303 is more appropriate. At any rate, Daniel was concerned that this has been an issue for a while, but the only artifact I'm seeing of active conversation on this topic is bug 44098, which is closed. Daniel, to make sure we track this, assuming we don't quickly come to a resolution in this thread, could you file a new bug requesting the 302 response be updated to 303? Thanks Rob [1] https://bugzilla.wikimedia.org/44098 [2] http://tools.ietf.org/html/rfc2616#section-10.3.4

3 2

You must not expect the generic exception class
by Daniel Kinzler 02 Aug '13

02 Aug '13

A quick heads up: I just tried runnign our phpunit tests with PHPUnit 3.6.10, and got about 60 of these errors: InvalidArgumentException: You must not expect the generic exception class. So, apparently, phpunit 3.6 doesn't want the expected exception to be "Exception". That's annoying. Apparently this will be fixed in 3.7... Well, I thought I'd let you know. -- daniel

2 3

Re: [Wikidata-tech] [Wikitech-l] Is assert() allowed?
by Tyler Romeo 01 Aug '13

01 Aug '13

On Wed, Jul 31, 2013 at 7:28 PM, Tim Starling <tstarling(a)wikimedia.org>wrote: > The php.ini option assert.bail is 0 by default. So? It's the same way in Java. You have to turn on assertions. It's kind of natural to assume that if assertions are off the won't cause fatal errors. *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech August 2013