Hi!
I am developing functionality that will allow WDQS query to get data from Mediawiki API [1].
Currently, the design is as follows: - The API should have a pre-defined template - The template also specifies which results are available from the API
The need for template is currently because we need to convert data from API's treelike format to tabular format that SPARQL needs, and the template allows to specify how the conversion is done. See https://www.wikidata.org/wiki/Wikidata:WDQS_and_Mediawiki_API for detailed description of how it works.
The prototype implementation is running on http://wdqs-test.wmflabs.org/ (only Categories API described in the page above is configured now, but more will be soon). I'd like to hear feedback about this: - does template model make sense at all? Is it enough? - what APIs would we want to expose? - any other features that would be useful?
Other comments and ideas on the matter are of course always welcome. Please comment on the talk page[2] or reply to this message.
[1] https://phabricator.wikimedia.org/T148245 [2] https://www.wikidata.org/w/index.php?title=Wikidata_talk:WDQS_and_Mediawiki_...
Thanks,
Stas,
One thing that would be extremely useful right away would be an integration of the free text search from MediaWiki API. That is one area SPARQL does not handle well, Mediawiki does well, and its pretty important for many applications. If there were some clever way of mixing (fast!) free text search with sparql it would be quite powerful. Imagine e.g. building type-ahead query boxes given semantic constraints.
On Thu, Apr 27, 2017 at 2:40 PM, Stas Malyshev smalyshev@wikimedia.org wrote:
Hi!
I am developing functionality that will allow WDQS query to get data from Mediawiki API [1].
Currently, the design is as follows:
- The API should have a pre-defined template
- The template also specifies which results are available from the API
The need for template is currently because we need to convert data from API's treelike format to tabular format that SPARQL needs, and the template allows to specify how the conversion is done. See https://www.wikidata.org/wiki/Wikidata:WDQS_and_Mediawiki_API for detailed description of how it works.
The prototype implementation is running on http://wdqs-test.wmflabs.org/ (only Categories API described in the page above is configured now, but more will be soon). I'd like to hear feedback about this:
- does template model make sense at all? Is it enough?
- what APIs would we want to expose?
- any other features that would be useful?
Other comments and ideas on the matter are of course always welcome. Please comment on the talk page[2] or reply to this message.
[1] https://phabricator.wikimedia.org/T148245 [2] https://www.wikidata.org/w/index.php?title=Wikidata_talk: WDQS_and_Mediawiki_API&action=edit
Thanks,
Stas Malyshev smalyshev@wikimedia.org
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Hi!
One thing that would be extremely useful right away would be an integration of the free text search from MediaWiki API. That is one
This is on the agenda, though it has two issues to solve: 1. Wikidata search is not great as such. We're working on it, but will take time to get it all. 2. Once we do the above, we may use internal Blazegraph API to have much tighter intergration with search, not going through Mediawiki API, which would allow better control over search parameters and flexibility. This will require some work too, though.
We could of course use search in Wikipedia and other wikis right now, which is in much better shape, but then we need to do some work to get back to Wikidata IDs. Doable, but we need to think which option is better. Maybe there would be several of them. But yes, this is definitely something that is on the agenda.
Wow, this looks awesome! Thanks a lot for working on this!
I think the configuration model could use some more documentation, but in principle I’d say it looks pretty reasonable. It’s also probably powerful enough for most reasonable needs (I read “XPath” and my eyes lit up :D ).
As for which APIs should be exposed: one thing I can remember that was felt as missing from WDQS several times is access and historical data, so I think that exposing action=query with prop=pageviews and prop=revisions might be useful.
Cheers, Lucas
On 27.04.2017 23:40, Stas Malyshev wrote:
Hi!
I am developing functionality that will allow WDQS query to get data from Mediawiki API [1].
Currently, the design is as follows:
- The API should have a pre-defined template
- The template also specifies which results are available from the API
The need for template is currently because we need to convert data from API's treelike format to tabular format that SPARQL needs, and the template allows to specify how the conversion is done. See https://www.wikidata.org/wiki/Wikidata:WDQS_and_Mediawiki_API for detailed description of how it works.
The prototype implementation is running on http://wdqs-test.wmflabs.org/ (only Categories API described in the page above is configured now, but more will be soon). I'd like to hear feedback about this:
- does template model make sense at all? Is it enough?
- what APIs would we want to expose?
- any other features that would be useful?
Other comments and ideas on the matter are of course always welcome. Please comment on the talk page[2] or reply to this message.
[1] https://phabricator.wikimedia.org/T148245 [2] https://www.wikidata.org/w/index.php?title=Wikidata_talk:WDQS_and_Mediawiki_...
Thanks,
Hi Stas,
This is *really* exciting news -- thank you so much for your work on this.
I can see it being valuable in so many ways -- the kind of things that people have put tickets in for so far, eg categories, image sizes, page stats etc, are just the tip of the iceberg.
One question: were there particular reasons why you went for wrapping a SERVICE round the Mediawiki API, rather than eg a SPARQL layer round the SQL tables, suitable for SPARQL federation?
Are there efficiency issues? Or did you think that the SERVICE approach was simply more user-friendly for the most commons APIs ?
-- James.
On 27/04/2017 22:40, Stas Malyshev wrote:
Hi!
I am developing functionality that will allow WDQS query to get data from Mediawiki API [1].
Currently, the design is as follows:
- The API should have a pre-defined template
- The template also specifies which results are available from the API
The need for template is currently because we need to convert data from API's treelike format to tabular format that SPARQL needs, and the template allows to specify how the conversion is done. See https://www.wikidata.org/wiki/Wikidata:WDQS_and_Mediawiki_API for detailed description of how it works.
The prototype implementation is running on http://wdqs-test.wmflabs.org/ (only Categories API described in the page above is configured now, but more will be soon). I'd like to hear feedback about this:
- does template model make sense at all? Is it enough?
- what APIs would we want to expose?
- any other features that would be useful?
Other comments and ideas on the matter are of course always welcome. Please comment on the talk page[2] or reply to this message.
[1] https://phabricator.wikimedia.org/T148245 [2] https://www.wikidata.org/w/index.php?title=Wikidata_talk:WDQS_and_Mediawiki_...
Thanks,
Hi!
One question: were there particular reasons why you went for wrapping a SERVICE round the Mediawiki API, rather than eg a SPARQL layer round the SQL tables, suitable for SPARQL federation?
Yes, we don't have any idea how to do such SPARQL layer :)
Are there efficiency issues? Or did you think that the SERVICE approach was simply more user-friendly for the most commons APIs ?
Probably there will be efficiency issues too, in general RDF data model is rather different from relational database date model, so translating between them is not trivial, unless you take extremely naive approach, in which case performance probably will be horrible.
SERVICE otoh is pretty easy to implement and it's flexible enough to allow talking to many external APIs. Current proposal seems to be OK, though may be changed if more user-friendly ideas come up.
Hi Stas,
Thanks for working on this. I tried to run the examples, but I get the following error when I hit the button: Query is malformed: QName 'mwapi:titles' uses an undefined prefix
And here you can see more details about the error output: https://pastebin.com/yNd7vPw4
I hope you can fix it :)
Thanks Micru
On Thu, Apr 27, 2017 at 11:40 PM, Stas Malyshev smalyshev@wikimedia.org wrote:
Hi!
I am developing functionality that will allow WDQS query to get data from Mediawiki API [1].
Currently, the design is as follows:
- The API should have a pre-defined template
- The template also specifies which results are available from the API
The need for template is currently because we need to convert data from API's treelike format to tabular format that SPARQL needs, and the template allows to specify how the conversion is done. See https://www.wikidata.org/wiki/Wikidata:WDQS_and_Mediawiki_API for detailed description of how it works.
The prototype implementation is running on http://wdqs-test.wmflabs.org/ (only Categories API described in the page above is configured now, but more will be soon). I'd like to hear feedback about this:
- does template model make sense at all? Is it enough?
- what APIs would we want to expose?
- any other features that would be useful?
Other comments and ideas on the matter are of course always welcome. Please comment on the talk page[2] or reply to this message.
[1] https://phabricator.wikimedia.org/T148245 [2] https://www.wikidata.org/w/index.php?title=Wikidata_talk: WDQS_and_Mediawiki_API&action=edit
Thanks,
Stas Malyshev smalyshev@wikimedia.org
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Hi Micru,
It looks like you tried on https://query.wikidata.org (where I get the same error) and not on http://wdqs-test.wmflabs.org (where the implementation is installed).
Cdlt, ~nicolas
2017-04-28 8:57 GMT+02:00 David Cuenca Tudela dacuetu@gmail.com:
Hi Stas,
Thanks for working on this. I tried to run the examples, but I get the following error when I hit the button: Query is malformed: QName 'mwapi:titles' uses an undefined prefix
And here you can see more details about the error output: https://pastebin.com/yNd7vPw4
I hope you can fix it :)
Thanks Micru
Hi Nicolas,
Indeed! That was it :) A typical case of PEBCAK :P
Cheers, Micru
On Fri, Apr 28, 2017 at 9:19 AM, Nicolas VIGNERON < vigneron.nicolas@gmail.com> wrote:
Hi Micru,
It looks like you tried on https://query.wikidata.org (where I get the same error) and not on http://wdqs-test.wmflabs.org (where the implementation is installed).
Cdlt, ~nicolas
2017-04-28 8:57 GMT+02:00 David Cuenca Tudela dacuetu@gmail.com:
Hi Stas,
Thanks for working on this. I tried to run the examples, but I get the following error when I hit the button: Query is malformed: QName 'mwapi:titles' uses an undefined prefix
And here you can see more details about the error output: https://pastebin.com/yNd7vPw4
I hope you can fix it :)
Thanks Micru
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata