Hi!
My view is that this tool should be extremely cautious when it sees new data structures or fields. The tool should certainly not continue to output facts without some indication that something is suspect, and preferably should refuse to produce output under these circumstances.
I don't think I agree. I find tools that are too picky about details that are not important to me hard to use, and I'd very much prefer a tool where I am in control of which information I need and which I don't need.
What can happen if the tool instead continues to operate without complaint when new data structures are seen? Consider what would happen if the tool was written for a version of Wikidata that didn't have rank, i.e., claim objects did not have a rank name/value pair. If ranks were then added, consumers of the output of the tool would have no way of distinguishing deprecated information from other information.
Ranks are a bit unusual because ranks are not just informational change, it's a semantic change. It introduces a concept of a statement that has different semantics than the rest. Of course, such change needs to be communicated - it's like I would make format change "each string beginning with letter X needs to be read backwards" but didn't tell the clients. Of course this is a breaking change if it changes semantics.
What I was talking are changes that don't break semantics, and majority of additions are just that.
Of course this is an extreme case. Most changes to the Wikidata JSON dump format will not cause such severe problems. However, given the current situation with how the Wikidata JSON dump format can change, the tool cannot determine whether any particular change will affect the meaning of what it produces. Under these circumstances it is dangerous for a tool that extracts information from the Wikidata JSON dump to continue to produce output when it sees new data structures.
The tool can not. It's not possible to write a tool that would derive semantics just from JSON dump, or even detect semantic changes. Semantic changes can be anywhere, it doesn't have to be additional field - it can be in the form of changing the meaning of the field, or format, or datatype, etc. Of course the tool can not know that - people should know that and communicate it. Again, that's why I think we need to distinguish changes that break semantics and changes that don't, and make the tools robust against the latter - but not the former because it's impossible. For dealing with the former, there is a known and widely used solution - format versioning.
This does make consuming tools sensitive to changes to the Wikidata JSON dump format that are "non-breaking". To overcome this problem there should be a way for tools to distinguish changes to the Wikidata JSON dump format that do not change the meaning of existing constructs in the dump from those that can. Consuming tools can then continue to function without problems for the former kind of change.
As I said, format versioning. Maybe even semver or some suitable modification of it. RDF exports BTW already carry version. Maybe JSON exports should too.