"Poorly structured" HTML is not all that bad in 2018 thanks to HTML 5 (which builds the "rendering decisions made about broken HTML from Netscape 3" into the standard so that in common languages you can get the same DOM tree as the browser) If you try to use an official or unofficial API to fetch data from some service in 2018 you will have to add some dependencies and you just might open a can of whoop-ass that will make you reinstall Anconda or maybe you will learn something you'll never be able to unlearn about how XML processing changed between two minor versions of the JDK On the other hand I have often dusted off the old HTML-based parser I made for Flickr and found I could get it to work for other media collections, blogs, etc. by just changing the "semantic model" embodied in the application which could be as simple as some function or object that knows something about the structure of the URLs some documents. I cannot understand why so many standards have been pushed to integrate RDF and HTML that have gone nowhere but nobody has promoted the clean solution of "add a css media type for RDF" that marks the semantics of HTML up the way JSON-LD works. Often though if you look it that way much of the time these days matching patterns against CSS gets you most of the way there. I've had cases where I haven't had to change the rule sets much at all but none of them have been more than 50 lines of code, all much less. ------ Original Message ------ From: "Federico Leva (Nemo)" <nemowiki(a)gmail.com> To: "Discussion list for the Wikidata project" <wikidata(a)lists.wikimedia.org>rg>; "Ettore RIZZA" <ettorerizza(a)gmail.com> Sent: 9/26/2018 1:00:53 PM Subject: Re: [Wikidata] Looking for "data quality check" bots _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi Ettore, On 26-09-18 14:31, Ettore RIZZA wrote: Let's have a look at the evolution of automated editing. The first step is to add missing data from anywhere. Bots importing date of birth are an example of this. The next step is to add data from somewhere with a source or add sources to existing unsourced or badly sourced statements. As far as I can see that's where we are right now, see for example edits like https://www.wikidata.org/w/index.php?title=Q41264&type=revision&dif… is . Of course the next step would be to be able to compare existing sourced statements with external data to find differences. But how would the work flow be? Take for example Johannes Vermeer ( https://www.wikidata.org/wiki/Q41264 ). Extremely well documented and researched, but http://www.getty.edu/vow/ULANFullDisplay?find=&role=&nation=&su… and https://rkd.nl/nl/explore/artists/80476 combined provide 3 different dates of birth and 3 different dates of death. When it comes to these kind of date mismatches, it's generally first come, first served (first date added doesn't get replaced). This mismatch could show up in some report. I can check it as a human and maybe do some adjustments, but how would I sign it of to prevent other people from doing the same thing over and over again? With federated SPARQL queries it becomes much easier to generate reports of mismatches. See for example https://www.wikidata.org/wiki/Property_talk:P1006/Mismatches . Maarten _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata