GerardM post triggered my interest to post to the mailing list. As you might know I am working on functional quadstore that is quadstore that keeps around old version of data, like a wiki but in direct-acyclic-graph. It only stores differences between commits. It rely on snapshot of the latest version for fast reads. My ultimate goal is to build somekind of portable knowlege base. That is something like WikiBase + blazegraph but that you spinup on regular machine with the press of button.
Enought brag about me. I wont't reply to all the message of the threads one by one but:
Here is what SHOULD BE possible:
- incremental dumps
- time traveling queries
- full dumps
- The federation of wikibase SHOULD BE possible since it stored in a history like GIT and git pull git push are planned in the ROADMAP
And online edition of the quadstore.
Access Control List are not designed yet, I except that this should be enforced by the application layer.
I planned start working on Data Management System (something like CKAN) with search featrure. But I would gadly work with wikimedia instead.
Also, given it modeled after git, one can do merge-request like features, ie. exist the massive import that is crippled.
What I would need is logs possibly with timing of queries (read and write) to do benchmarks.
Maybe I should ask for fund at mediawiki?
FWIW, I got 2 times faster than blazegraph on microbenchmark.
Hoi,
Wikidata grows like mad. This is something we all experience in the really bad response times we are suffering. It is so bad that people are asked what kind of updates they are running because it makes a difference in the lag times there are.
Given that Wikidata is growing like a weed, it follows that there are two issues. Technical - what is the maximum that the current approach supports - how long will this last us. Fundamental - what funding is available to sustain Wikidata.
For the financial guys, growth like Wikidata is experiencing is not something you can reliably forecast. As an organisation we have more money than we need to spend, so there is no credible reason to be stingy.
For the technical guys, consider our growth and plan for at least one year. When the impression exists that the current architecture will not scale beyond two years, start a project to future proof Wikidata.
It will grow and the situation will get worse before it gets better.
Thanks,
GerardM
PS I know about phabricator tickets, they do not give the answers to the questions we need to address.