Communicating corrections back to data source - Wikidata - lists.wikimedia.org

List overview All Threads
Download

Communicating corrections back to data source

Wikidata and DBpedia as spatial...

Wikidata activities at Wikimania

Julie McMurry

10 Jun 2016 10 Jun '16

8:31 p.m.

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie

Attachments:

attachment.htm (text/html — 361 bytes)

Reply

Show replies by thread

Benjamin Good

10 Jun 10 Jun

8:43 p.m.

Hi Julie, We've thought a lot about this, but not done anything formally yet. There is an example of this happening to improve the disease ontology presented in this paper [1]. Mechanically, parties interested in a particular swath of data linked to their resource could set up repeated SPARQL queries to watch for changes. Beyond that, the core mediawiki API could be used to create alerts when new discussions are started on articles or items of interest. At some point we hope to produce a reporting site that would aggregate this kind of information in our domain (feedback and changes by the community) as well as changes by our bots and provide reports back to the primary sources and to whoever else was interested. (Maybe we will see a start on that this summer..) This hasn't become a priority yet because we haven't yet generated the community scope to make it a really valuable source of input to the original databases. [1] http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf On Fri, Jun 10, 2016 at 11:31 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Gerard Meijssen

9:11 p.m.

Hoi, It has been discussed a lot. There are ample ways of doing this. It is that it is not hard at all to compare sources. It starts with sharing identifiers and then comparing values that could be / should be the same. We can then report differences and work on quality. When Wikidata has no data, we could import it and identify the source.. It is feasible, experiments were done. There has been only apathy. Thanks, GerardM On 10 June 2016 at 20:31, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Sandra Fauconnier

9:12 p.m.

I have recently read an interview with Herbert Van de Sompel <https://en.wikipedia.org/wiki/Herbert_Van_de_Sompel>, who among others has worked on the OAI-PMH and the Memento project (for those for whom that rings a bell). Recently his team has developed an initiative called ResourceSync <http://www.openarchives.org/rs/toc>, that seems to be addressing exactly this - keeping distributed databases on the web mutually up to date. It’s the closest thing I’ve ever seen that seems to address what we (and the entire interlinked web) would need in this area. I might have missed other initiatives, but this one gave me a big AHA moment! Here’s a short video that explains the principle: https://www.youtube.com/watch?v=ASQ4jMYytsA <https://www.youtube.com/watch?v=ASQ4jMYytsA> In the interview I read. Herbert said that it didn’t see wide adoption yet though. I can imagine that, if the Wikimedia projects’ software would adopt this, it might have a snowball effect. Best, Sandra

On 10 Jun 2016, at 20:43, Benjamin Good <ben.mcgee.good(a)gmail.com> wrote: Hi Julie, We've thought a lot about this, but not done anything formally yet. There is an example of this happening to improve the disease ontology presented in this paper [1]. Mechanically, parties interested in a particular swath of data linked to their resource could set up repeated SPARQL queries to watch for changes. Beyond that, the core mediawiki API could be used to create alerts when new discussions are started on articles or items of interest. At some point we hope to produce a reporting site that would aggregate this kind of information in our domain (feedback and changes by the community) as well as changes by our bots and provide reports back to the primary sources and to whoever else was interested. (Maybe we will see a start on that this summer..) This hasn't become a priority yet because we haven't yet generated the community scope to make it a really valuable source of input to the original databases. [1] http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf <http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf> On Fri, Jun 10, 2016 at 11:31 AM, Julie McMurry <mcmurry.julie(a)gmail.com <mailto:mcmurry.julie@gmail.com>> wrote: It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Luca Martinelli

11 Jun 11 Jun

7:31 a.m.

I do this actually for the institute I work for: I take a moment to check the property violations for P396 and then advice the proper office to merge the codes (which they double-check once noticed). It sometimes ended also with me learning a couple of things on how work goes. :) L. Il 10 giu 2016 20:32, "Julie McMurry" <mcmurry.julie(a)gmail.com> ha scritto:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Gerard Meijssen

9:32 a.m.

Hoi, Resourcesync is unlikely to be adopted by Wikidata for adopting changes from elsewhere. If others want to share data FROM WIkidata there is no problem with providing Resourcesync. What I do not completely understand is how its mechanism for indicating changes may be used. When it can be used to generate reports so that people can actually see the differences it could be really important to improve the quality. The notion that we can just copy in data or change data based on an "authorised" source is problematic. Thanks, GerardM On 10 June 2016 at 21:12, Sandra Fauconnier <sandra.fauconnier(a)gmail.com> wrote:

I have recently read an interview with Herbert Van de Sompel <https://en.wikipedia.org/wiki/Herbert_Van_de_Sompel>, who among others has worked on the OAI-PMH and the Memento project (for those for whom that rings a bell). Recently his team has developed an initiative called ResourceSync <http://www.openarchives.org/rs/toc>, that seems to be addressing exactly this - keeping distributed databases on the web mutually up to date. It’s the closest thing I’ve ever seen that seems to address what we (and the entire interlinked web) would need in this area. I might have missed other initiatives, but this one gave me a big AHA moment! Here’s a short video that explains the principle: https://www.youtube.com/watch?v=ASQ4jMYytsA In the interview I read. Herbert said that it didn’t see wide adoption yet though. I can imagine that, if the Wikimedia projects’ software would adopt this, it might have a snowball effect. Best, Sandra On 10 Jun 2016, at 20:43, Benjamin Good <ben.mcgee.good(a)gmail.com> wrote: Hi Julie, We've thought a lot about this, but not done anything formally yet. There is an example of this happening to improve the disease ontology presented in this paper [1]. Mechanically, parties interested in a particular swath of data linked to their resource could set up repeated SPARQL queries to watch for changes. Beyond that, the core mediawiki API could be used to create alerts when new discussions are started on articles or items of interest. At some point we hope to produce a reporting site that would aggregate this kind of information in our domain (feedback and changes by the community) as well as changes by our bots and provide reports back to the primary sources and to whoever else was interested. (Maybe we will see a start on that this summer..) This hasn't become a priority yet because we haven't yet generated the community scope to make it a really valuable source of input to the original databases. [1] http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf On Fri, Jun 10, 2016 at 11:31 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Sandra Fauconnier

11:55 a.m.

Why? I would enthusiastically welcome an automated system that allows us to stay up to date with all the authority databases for which we have properties. At this moment, I have a hunch that we are already hopelessly out of date with most of the ones we started to add a few years ago. I’d like to hear other suggestions that provide broad scalable solutions for this (not just ‘I’m tracking this one dataset that is important to me’, but ‘this allows us to track all hundreds of external datasets we have some kind of identifiers for’). Sandra

On 11 Jun 2016, at 09:32, Gerard Meijssen <gerard.meijssen(a)gmail.com> wrote: Hoi, Resourcesync is unlikely to be adopted by Wikidata for adopting changes from elsewhere. If others want to share data FROM WIkidata there is no problem with providing Resourcesync. What I do not completely understand is how its mechanism for indicating changes may be used. When it can be used to generate reports so that people can actually see the differences it could be really important to improve the quality. The notion that we can just copy in data or change data based on an "authorised" source is problematic. Thanks, GerardM On 10 June 2016 at 21:12, Sandra Fauconnier <sandra.fauconnier(a)gmail.com <mailto:sandra.fauconnier@gmail.com>> wrote: I have recently read an interview with Herbert Van de Sompel <https://en.wikipedia.org/wiki/Herbert_Van_de_Sompel>, who among others has worked on the OAI-PMH and the Memento project (for those for whom that rings a bell). Recently his team has developed an initiative called ResourceSync <http://www.openarchives.org/rs/toc>, that seems to be addressing exactly this - keeping distributed databases on the web mutually up to date. It’s the closest thing I’ve ever seen that seems to address what we (and the entire interlinked web) would need in this area. I might have missed other initiatives, but this one gave me a big AHA moment! Here’s a short video that explains the principle: https://www.youtube.com/watch?v=ASQ4jMYytsA <https://www.youtube.com/watch?v=ASQ4jMYytsA> In the interview I read. Herbert said that it didn’t see wide adoption yet though. I can imagine that, if the Wikimedia projects’ software would adopt this, it might have a snowball effect. Best, Sandra

On 10 Jun 2016, at 20:43, Benjamin Good <ben.mcgee.good(a)gmail.com <mailto:ben.mcgee.good@gmail.com>> wrote: Hi Julie, We've thought a lot about this, but not done anything formally yet. There is an example of this happening to improve the disease ontology presented in this paper [1]. Mechanically, parties interested in a particular swath of data linked to their resource could set up repeated SPARQL queries to watch for changes. Beyond that, the core mediawiki API could be used to create alerts when new discussions are started on articles or items of interest. At some point we hope to produce a reporting site that would aggregate this kind of information in our domain (feedback and changes by the community) as well as changes by our bots and provide reports back to the primary sources and to whoever else was interested. (Maybe we will see a start on that this summer..) This hasn't become a priority yet because we haven't yet generated the community scope to make it a really valuable source of input to the original databases. [1] http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf <http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf> On Fri, Jun 10, 2016 at 11:31 AM, Julie McMurry <mcmurry.julie(a)gmail.com <mailto:mcmurry.julie@gmail.com>> wrote: It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata>

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata <https://lists.wikimedia.org/mailman/listinfo/wikidata> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Federico Leva (Nemo)

12:09 p.m.

Sandra Fauconnier, 10/06/2016 21:12:

Recently his team has developed an initiative called ResourceSync <http://www.openarchives.org/rs/toc>, that seems to be addressing exactly this - keeping distributed databases on the web mutually up to date. It’s the closest thing I’ve ever seen that seems to address what we (and the entire interlinked web) would need in this area. I might have missed other initiatives, but this one gave me a big AHA moment!

That's more like a backup/mirroring system https://www.openarchives.org/rs/1.0/resourcesync#MotivExamples (similar to LOCKSS etc.), while here we're talking of data reconciliation. Avoiding data loss, linkrot and single points of failure for the web is indeed something that interests Wikimedia; I believe the most advanced proposal in this area is http://brewster.kahle.org/2015/08/11/locking-the-web-open-a-call-for-a-dist… Nemo

Reply

Gerard Meijssen

12:11 p.m.

Hoi, The first reason is obvious. External sources have their issues. They do not necessarily fit our purpose. A case in point is a substance database that indicates that they are registered as a medicine. When it is then known that those substances are proven to be ineffective, what is the value. You do want to differ on those statements and at least indicate that they are proven ineffective. The second reason is that sources have their issues, biases and I doubt that we would ever hand over control. The best we could do and should do is register differences and source what we believe is true. It then becomes a two way street. We learn about their pov and they about ours. They can change their data based on our input. When they do not, it is their database, their effort, their priority. Yet another reason is that the same data is known in many databases. Some only register a year of birth and not a date. Obviously we do not need a year when we have the complete date. When several sources, unrelated sources share the same data, we want to track the differences between them all and indeed know what is what. Thanks, GerardM On 11 June 2016 at 11:55, Sandra Fauconnier <sandra.fauconnier(a)gmail.com> wrote:

Why? I would enthusiastically welcome an automated system that allows us to stay up to date with all the authority databases for which we have properties. At this moment, I have a hunch that we are already hopelessly out of date with most of the ones we started to add a few years ago. I’d like to hear other suggestions that provide broad scalable solutions for this (not just ‘I’m tracking this one dataset that is important to me’, but ‘this allows us to track all hundreds of external datasets we have some kind of identifiers for’). Sandra On 11 Jun 2016, at 09:32, Gerard Meijssen <gerard.meijssen(a)gmail.com> wrote: Hoi, Resourcesync is unlikely to be adopted by Wikidata for adopting changes from elsewhere. If others want to share data FROM WIkidata there is no problem with providing Resourcesync. What I do not completely understand is how its mechanism for indicating changes may be used. When it can be used to generate reports so that people can actually see the differences it could be really important to improve the quality. The notion that we can just copy in data or change data based on an "authorised" source is problematic. Thanks, GerardM On 10 June 2016 at 21:12, Sandra Fauconnier <sandra.fauconnier(a)gmail.com> wrote:

I have recently read an interview with Herbert Van de Sompel <https://en.wikipedia.org/wiki/Herbert_Van_de_Sompel>, who among others has worked on the OAI-PMH and the Memento project (for those for whom that rings a bell). Recently his team has developed an initiative called ResourceSync <http://www.openarchives.org/rs/toc>, that seems to be addressing exactly this - keeping distributed databases on the web mutually up to date. It’s the closest thing I’ve ever seen that seems to address what we (and the entire interlinked web) would need in this area. I might have missed other initiatives, but this one gave me a big AHA moment! Here’s a short video that explains the principle: https://www.youtube.com/watch?v=ASQ4jMYytsA In the interview I read. Herbert said that it didn’t see wide adoption yet though. I can imagine that, if the Wikimedia projects’ software would adopt this, it might have a snowball effect. Best, Sandra On 10 Jun 2016, at 20:43, Benjamin Good <ben.mcgee.good(a)gmail.com> wrote: Hi Julie, We've thought a lot about this, but not done anything formally yet. There is an example of this happening to improve the disease ontology presented in this paper [1]. Mechanically, parties interested in a particular swath of data linked to their resource could set up repeated SPARQL queries to watch for changes. Beyond that, the core mediawiki API could be used to create alerts when new discussions are started on articles or items of interest. At some point we hope to produce a reporting site that would aggregate this kind of information in our domain (feedback and changes by the community) as well as changes by our bots and provide reports back to the primary sources and to whoever else was interested. (Maybe we will see a start on that this summer..) This hasn't become a priority yet because we haven't yet generated the community scope to make it a really valuable source of input to the original databases. [1] http://biorxiv.org/content/biorxiv/early/2015/11/16/031971.full.pdf On Fri, Jun 10, 2016 at 11:31 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered? Julie _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Reply

Lydia Pintscher

3:09 p.m.

On Fri, Jun 10, 2016 at 8:32 PM Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

It is great that WikiData provides a way for data to be curated in a crowd-sourced way. It would be even better if changes (especially corrections) could be communicated back to the original source so that all could benefit. Has this been discussed previously? Considered?

I do want someone to look at this topic more systematically - possibly as a thesis. The task for that is https://phabricator.wikimedia.org/T108292. Feel free to add input to it. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

Reply

Julie McMurry

11:29 p.m.

Thanks Sandra for introducing ResourceSync. The web page and it appears up to date, but according to the video footer, the original presentation was from 2014. I would like to know who if anyone has implemented it so far and how it is going? Has anyone on this listserve heard of it before? My colleagues and I are gearing up to resubmit the identifiers best practice paper and would like to know about all relevant tooling that could help data integrators with change/dependency management. Best, Julie

Reply

2869

days inactive

2870

days old

wikidata@lists.wikimedia.org

Manage subscription

10 comments

7 participants

tags (0)

participants (7)

Benjamin Good
Federico Leva (Nemo)
Gerard Meijssen
Julie McMurry
Luca Martinelli
Lydia Pintscher
Sandra Fauconnier