Dear Wikidata Community,
We are trying to understand which database technologies and strategies Wikidata uses for
storing, updating, and querying the data (knowledge) it manipulates.
By looking at the
documentation<https://wmde.github.io/wikidata-wikibase-architecture/assets/img/03-dataflow-out.drawio.17c12ee9.svg>
we understood that RDF is only used for the Wikidata Query Service, but we could not find
out exactly how Wikidata stores the information that is translated to RDF during the data
dump.
More specifically, we understood that a MySQL (or is it MariaDB?) relational database is
used as the key persistence component for most of Wikidata services and that the
information that is maintained in this database is periodically exported to multiple
formats, including RDF.
In addition, looking at the relational database schema published in the
documentation<https://www.mediawiki.org/wiki/Manual:Database_layout> we could not
locate tables that are easily mappable to the Wikibase Data
Model<https://www.mediawiki.org/wiki/Wikibase/DataModel>el>.
Thus, we hypothesize that there is some software component (Wikibase Common Data Access?)
that dynamically translates the data contained in those tables to Statements, Entities,
etc. Is that hypothesis, correct?
If yes, does this software component uses any intermediate storage mechanism for caching
those Statements, Entities, ...? Or are those translations always performed at runtime
on-the-fly (be it for querying, adding, or updating Statements, Entities, …)?
Finally, we would like to understand more about how Wikidata REST
API<https://www.wikidata.org/wiki/Wikidata:REST_API> is implemented:
* In which database are the statements that added/retrieved through it stored? Is it
being stored in the central MySQL database or in another database?
* Does it have any support for pagination of statements? For example, if an item has
many statements associated with a property, does the API assumes that both the underlying
database and the network will support the retrieval of all those statements?
* Are you currently considering implementing the support for more flexible querying of
statements, or such requirement has been fully delegated to the Wikidata Query Service?
If there is an updated documentation that could help us answer those questions, could you
kindly point us to it? Otherwise, would you be able to share this information with us?
Best Regards,
Elton F. de S. Soares
Advisory Software Engineer
Rio de Janeiro, RJ, Brazil
IBM Research
E-mail: eltons@ibm.com<mailto:eltons@ibm.com>
