Dear Wikibase/Wikidata Community,
We are trying to understand which database technologies and strategies Wikibase/Wikidata
uses for storing, updating, and querying the data (knowledge) it manipulates.
By looking at the
documentation<https://wmde.github.io/wikidata-wikibase-architecture/assets/img/03-dataflow-out.drawio.17c12ee9.svg>
we understood that RDF is only used for the Wikidata Query Service, but we could not find
out exactly how Wikibase/Wikidata stores the information that is translated to RDF during
the data dump.
More specifically, we understood that a MySQL (or is it MariaDB?) relational database is
used as the key persistence component for most of Wikibase/Wikidata services and that the
information that is maintained in this database is periodically exported to multiple
formats, including RDF.
In addition, looking at the relational database schema published in the
documentation<https://www.mediawiki.org/wiki/Manual:Database_layout> we could not
locate tables that are easily mappable to the Wikibase Data
Model<https://www.mediawiki.org/wiki/Wikibase/DataModel>el>.
Thus, we hypothesize that there is some software component (Wikibase Common Data Access?)
that dynamically translates the data contained in those tables to Statements, Entities,
etc. Is that hypothesis, correct?
If yes, does this software component uses any intermediate storage mechanism for caching
those Statements, Entities, ...? Or are those translations always performed at runtime
on-the-fly (be it for querying, adding, or updating Statements, Entities, …)?
Finally, we would like to understand more about how Wikidata REST
API<https://www.wikidata.org/wiki/Wikidata:REST_API> is implemented:
• In which database are the statements that added/retrieved through it stored? Is
it being stored in the central MySQL database or in another database?
• Does it have any support for pagination of statements? For example, if an item
has many statements associated with a property, does the API assumes that both the
underlying database and the network will support the retrieval of all those statements?
• Are you currently considering implementing the support for more flexible
querying of statements, or such requirement has been fully delegated to the Wikidata Query
Service?
If there is an updated documentation that could help us answer those questions, could you
kindly point us to it? Otherwise, would you be able to share this information with us?
Best Regards,
Elton F. de S. Soares
Advisory Software Engineer
Rio de Janeiro, RJ, Brazil
IBM Research
E-mail: eltons@ibm.com<mailto:eltons@ibm.com>
