Hello Sebastian and Stas,
Le mer. 12 juin 2019 à 19:27, Amirouche Boubekki < amirouche.boubekki@gmail.com> a écrit :
Hello Sebastian,
First thanks a lot for the reply. I started to believe that what I was saying was complete nonsense.
Le mer. 12 juin 2019 à 16:51, Sebastian Hellmann < hellmann@informatik.uni-leipzig.de> a écrit :
Hi Amirouche,
Any open data projects that are running open databases with FoundationDB and WiredTiger? Where can I query them?
Thanks for asking. I will set up a wiredtiger instance of wikidata. I need a few days, maybe a week (or two :)).
I could setup FoundationDB on a single machine instead but it will require more time (maybe one more week).
Also, it will not support geo-queries. I will try to make labelling work but with a custom syntax (inspired form SPARQL).
I figured that anything that is not SPARQL will not be convincing. Getting my engine 100% compatible is much work.
The example deployment I have given in the previous message should be enough to convince you that FoundationDB can store WDQS.
The documented limits about FDB states that it to support up to 100TB of data https://apple.github.io/foundationdb/known-limitations.html#database-size. That is 100x times more than what WDQS needs at the moment.
Anyway, I updated my proposal to support wikimedia foundation to transition to a new solution in the wiki https://meta.wikimedia.org/wiki/Grants:Project/WDQS_On_FoundationDB to reflect the new requirements where the required space was reduced from 12T SSD to 6T SSD, it is based on this FDB forum topic https://forums.foundationdb.org/t/sizing-and-pricing/379/2?u=amirouche and an optimisation I will make in my engine. That proposal is biased toward getting a FDB prototype. It could be reworked to emphasize the fact that a benchmarking tool must be put together to be able to tell which solution is best.
My estimations might be off, especially the 1 month of GCP credits.
To be honest, WDQS is a low hanging fruit compared to the goal of building a portable wikidata.
I am offering my full-time services, it is up to you decide what will happen.