Le dim. 9 juin 2019 à 23:18, Amirouche Boubekki < amirouche.boubekki(a)gmail.com&gt; a écrit : I got "feedback" in others threads from the same topic that I will quote and reply to. develop inhouse Big cat, small fish. As wikidata continue to grow, it will have specific needs. Needs that are unlikely to be solved by off-the-shelf solutions. deployments. FoundationDB and WiredTiger are respectively used at Apple (among other companies) and MongoDB since 3.2 all over-the-world. WiredTiger is also used at Amazon. we're working on now. What is the phabricator ticket? Please. I don't understand how this is related to the scaling issues. advantage for us to consider running it in production. That will be vendor lock-in for wikidata and wikimedia along all the poor souls that try to interop with it. First commit <https://github.com/arangodb/arangodb/commit/6577d5417a000c29c9ee7666cbcc3cefae6eee21> is from 2011. It has two backends: MMAP and rocksdb. Wikidata It looks like a bounty. ArangoDB is a multi-model database, it support: - Document - Graph - Key-Value not sure resources into that. I am biased. I would advise against trying arangodb. This is another short term solution. least Incorrect. My solution is in the foreseeable future. It would be interesting to read how much "resource" is poured into keeping all those synchronized: - ElasticSearch - MySQL - BlazeGraph Maybe some REDIS?

Hello Sebastian, First thanks a lot for the reply. I started to believe that what I was saying was complete nonsense. Le mer. 12 juin 2019 à 16:51, Sebastian Hellmann < hellmann(a)informatik.uni-leipzig.de&gt; a écrit : Yes! At least part of it. The domain specific part. Yes, but it is not because of its technical merits. Similarly for PHP. Historically, PHP and MySQL were easy to setup and easy to use but otherwise difficult to work with. This is/was painful enough that nowadays the goto RDBMS is PostgreSQL even if MySQL is still very popular [0][1]. Those are technical reasons. Also, I agree it is not because MySQL had no ACID guarantees when it started 1995, that nowadays it is a bad choice. [0] https://trends.google.com/trends/explore?q=MySQL,PostgreSQL [1] https://stackshare.io/stackups/mysql-vs-postgresql I am giving an example deployment of WiredTiger. WiredTiger is an ordered Key-Value Store that is the storage engine of MongoDB since 3.2. It was created by independent company and later mongodb acquired WiredTiger. It is still GPLv2 or v3. Among the founders there is one of the engineer that created bsddb that Oracle has bought. Also, I am not saying WiredTiger solve all the problems of mongodb. I am just saying that because WiredTiger is the storage backend of MongoDB since 3.2 it has seen widespread usage and testing. Some say it is THE solution for scalability, others have said it was the Some people gave warnings about the technical issues of mongodb before 3.2. Also, Caveat emptor. The situation is better that a few years back. After all that was open source / free software / source available software since the beginning. Like I said above, WiredTiger is not the solution of all problems. I cited WiredTiger as a possible tool for building a cluster similar the current one where machines have full copies of the data. The advantage of WiredTiger is that it is easier to setup (compared to a distributed database) but it still requires fine-tuning / configuration. Also, there is many other Ordered Key-Value store in the wild. I have documented those in the document: https://github.com/scheme-requests-for-implementation/srfi-167/blob/master/… In particular, if WDQS doesn't want to use ACID transactions, there might be a better solution. Other popular options are LMDB (used in OpenLDAP) and RocksDB by Facebook (that is LevelDB fork). But again, that is ONE possibility, my design / database work with any of the libraries described in the above libraries.md url. My recommendation for production cluster is to use FoundationDB. Because it can scale horizontally and provides single / double / triple replication. If a node is down, the write and reads can still continue if you have enough machine up. WiredTiger would better suited for single machine (and my database (can) support both WiredTiger and FoundationDB with the same code base). Do FoundationDB and WiredTiger have any track record for hosting open data tl;dr: I don't know. Like I said previously, WiredTiger is used in many contexts among others it used at Amazon Web Services (AWS). FoundationDB is used at Apple, I don't remember which services rely on it but at least the Data Science team rely on it. The main contributor did a lightning talk about it: Entity Store: A FoundationDB Layer for Versioned Entities with Fine Grained <https://youtu.be/16uU_Aaxp9Y> That is the use-case that looks the more like data. More on popularity contest, it is used at WaveFront (owned by VMWare) that is an analytic tool. Here is a talk: Running FDB at scale <https://youtu.be/M438R4SlTFE> JanusGraph has FDB backend, see the talk: The JanusGraph FoundationDB Storage Adapter <https://youtu.be/rQM_ZPZy8Ck> It is also used at SnowFlake <https://www.snowflake.com/> that is apparently a datawhare house, here is the talk: How FoundationDB powers SnowflakeDB's metadata <https://youtu.be/KkeyjFMmIf8> It is also used at SkuVault as multi-model database, see the forum topic: https://forums.foundationdb.org/t/success-story-foundationdb-at-skuvault/336 <https://youtu.be/KkeyjFMmIf8> Again, I think the popularity of a tool is a hint. For instance, LevelDB is very popular but it far from the best in terms of speed. Similarly, I would not recommend Oracle BerkeleyDB, even if it is owned by Oracle. That said, database configuration and fine tuning is an art. So prolly I did something wrong with bsdd and leveldb. Maybe. But at least, in my proposal it is possible to benchmark several open source vendors. PostgreSQL and MySQL are widely used, e.g. OpenStreetMaps. Virtuoso by Yes, I know. I am waiting for a proposal to run WDQS on top of MySQL or PostgreSQL. I don't know FoundationDB or WiredTiger, but in the past there were often Those are details I can not have. There is a few hints in the case of WiredTiger, in the sense that they are branches named after mongodb e.g. https://github.com/wiredtiger/wiredtiger/tree/mongodb-4.0 so it seems mongodb use a specific branch that is public. For FoundationDB, like I said previously, setting up a cluster is more demanding as it is a distributed database. But it is also more future-proof. I would like to note, that IF WDQS is hosted at openlink, the problem is the same, if not worse. Thanks for asking. I will set up a wiredtiger instance of wikidata. I need a few days, maybe a week (or two :)). I could setup FoundationDB on a single machine instead but it will require more time (maybe one more week). Also, it will not support geo-queries. I will try to make labelling work but with a custom syntax (inspired form SPARQL). I have vaguely read the paper. On the topic of provenance, I would argue to rely a n+1 tuple items. Otherwise, off-topic but I can cite Bernstein MVCC paper which says that read operations don't block write operations and write operations don't block read operations which is what WT and FDB use internally. Reproducibility is a key topic in engineering. That's why there is so much As-A-Service fu around. Delegating the infrastructure to another party would be a risk and my last choice as far as my values as a human being are concerned. That is what I mean by "portable" wikidata. This is somewhat off-topic but I think that making wikimedia infrastructure reproducible _and_ portable should be some kind of priority for the organisation. That's why I value projects like kiwix. Again, thanks for the reply.

Hi Amirouche, On 16.06.19 23:01, Amirouche Boubekki wrote: Don get me wrong, I don want you to set it up. I am asking about a reference project, that has: 1. open data and an open database 2. decent amount of data 3. several years of running it. Like OpenStreetMap and PostreSQL, MediaWiki/Wikipedia -> MySQL, DBpedia -> Virtuoso. This would be a very good point for it. Otherwise I would consider it a sales trap, i.e. some open source which does not work really until you switch to the commercial product, same for Neptune. Now I think, only Apple knows how to use it. Any other reference projects? -- All the best, Sebastian Hellmann Director of Knowledge Integration and Linked Data Technologies (KILT) Competence Center at the Institute for Applied Informatics (InfAI) at Leipzig University Executive Director of the DBpedia Association Projects: http://dbpedia.org, http://nlp2rdf.org, http://linguistics.okfn.org, https://www.w3.org/community/ld4lt <http://www.w3.org/community/ld4lt> Homepage: http://aksw.org/SebastianHellmann Research Group: http://aksw.org

Hi! "Support" is such a multi-faceted word. It can mean "it works very well with such amount of data and is faster than the alternatives" or "it is guaranteed not to break up to this number but breaks after it" or "it would work, given massive amounts of memory and super-fast hardware and very specific set of queries, but you'd really have to take an effort to make it work" and everything in between. The devil is always in the details, which this seemingly simple word "supports" is rife with. I wish you luck with the grant, though I personally think if you expect to have a production-ready service in 6 month that can replace WDQS then in my personal opinion it is a bit too optimistic. I might be completely wrong on this of course. If you just plan to load the Wikidata data set and evaluate the queries to ensure they are fast and produce proper results on the setup you propose, then it can be done. Good luck! -- Stas Malyshev smalyshev(a)wikimedia.org

Le mer. 12 juin 2019 à 19:11, Stas Malyshev &lt;smalyshev(a)wikimedia.org&gt; a écrit : Today, the problem is not the same as the one MySQL, PostgreSQL, blazegraph and openlink had when they started working on their respective databases. See below. that scale is probably not going to happen. It will. > FoundationDB and WiredTiger are respectively used at Apple (among other > > companies) > > and MongoDB since 3.2 all over-the-world. WiredTiger is also used at > Amazon. > I believe they are, but I think for our particular goals we have to > limit themselves for a set of solution that are a proven good match for > our case. See the other mail I just sent. We are a turning point in database engineering history. The very last database systems that were built are all based on Ordered Key Value Store, see Google Spanner paper [0]. Thanks to WT/MongoDB and Apple, those are readily available, in widespread use and fully open source. It is only missing a few pieces for making it work a fully backward compatible way with WDQS (at scale). [0] https://ai.google/research/pubs/pub39966 > > That will be vendor lock-in for wikidata and wikimedia along all the > > poor souls that try to interop with it. > Since Virtuoso is using standard SPARQL, it won't be too much of a > vendor lock in, though of course the standard does not cover all, so > some corners are different in all SPARQL engines. There is a big chance that same thing that happened with the www will happen with RDF. That is one big player own all the implementations. operational aspects, is non-trivial. I agree. more disruptive, so right now we do not seriously consider doing that. I also agree. > As I already mentioned, there's a difference between "you can do it" and > "you can do it efficiently". [...] The tricky part starts when you need to > run millions > of queries on 10B triples database. If your backend is not optimal for > that task, it's not going to perform. I already did small benchmarks against blazegraph. I will do more intensive benchmarks using wikidata (and reduce the requirements in terms of SSD). Thanks for the reply.

Hi Guillaume, On 10.06.19 16:54, Guillaume Lederrey wrote: Yes, sharding is what you need, I think, instead of replication. This is the technique where data is repartitioned into more manageable chunks across servers. Here is a good explanation of it: http://vos.openlinksw.com/owiki/wiki/VOS/VOSArticleWebScaleRDF http://docs.openlinksw.com/virtuoso/ch-clusterprogramming/ Sharding, scale-out or repartitioning is a classical enterprise feature for Open-source databases. I am rather surprised that Blazegraph is full GPL without an enterprise edition. But then they really sounded like their goal as a company was to be bought by a bigger fish, in this case Amazon Web Services. What is their deal? They are offering support? So if you go open-source, I think you will have a hard time finding good free databases sharding/repartition. FoundationDB as proposed in the grant [1]is from Apple [1] https://meta.wikimedia.org/wiki/Grants:Project/WDQS_On_FoundationDB I mean try the sharding feature. At some point though it might be worth considering to go enterprise. Corporate Open Source often has a twist. Just a note here: Virtuoso is also a full RDMS, so you could probably keep wikibase db in the same cluster and fix the asynchronicity. That is also true for any mappers like Sparqlify: http://aksw.org/Projects/Sparqlify.html However, these shift the problem, then you need a sharded/repartitioned relational database.... All the best, Sebastian -- All the best, Sebastian Hellmann Director of Knowledge Integration and Linked Data Technologies (KILT) Competence Center at the Institute for Applied Informatics (InfAI) at Leipzig University Executive Director of the DBpedia Association Projects: http://dbpedia.org, http://nlp2rdf.org, http://linguistics.okfn.org, https://www.w3.org/community/ld4lt <http://www.w3.org/community/ld4lt> Homepage: http://aksw.org/SebastianHellmann Research Group: http://aksw.org

Hi Guillaume, All, As the lead developer for sparql.uniprot.org one of the few sparql endpoints with much more data (7x) than wikidata and significant external users. I can chime in with our experiences of hosting data with Virtuoso. All in all, I am very happy with it and it has made our endpoint possible and useful at a shoe string budget. We like WikiData have an async loading process and allow anyone to run analytics queries on our SPARQL endpoint with generous timeouts. We have two servers each with 256GB of ram and 8TB of raw SSD space (consumer). These are whitebox AMD machines from 2014 and the main cost at the time was the RAM. The setup was relatively cheap (cheaper than what is documented at https://www.mediawiki.org/wiki/Wikidata_Query_Service/Implementation#Hardwa…) Even in 2014 we already had more data than you do now. There is a third multi use server which does the loading of data offline. This is now a larger new Epyc server with more ram and more SSD but used for much more than just the RDF loading). Unlike, most sites we do have our own custom frontend in front of virtuoso. We did this to allow more styling, as well as being flexible and change implementations at our whim. e.g. we double parse the SPARQL queries and even rewrite some to be friendlier. I suggest you do the same no matter which DB you use in the end, and we would be willing to open source ours (it is in Java, and uses RDF4J and some ugly JSPX but it works, if not to use at least as an inspiration). We did this to avoid being locked into endpoint specific features. We use the opensource edition of virtuoso, and do not need the sharding etc. features. We use the CAIS (Cheap Array Of Independent Servers ;) approach to resilience. OpenlinkSW behind Virtuoso can deliver support for the OpenSource edition and if you are interested I suggest you talk to them. Virtuoso 7 has become very resilient over the years, and does not need much hand-holding anymore (in 2015 this was different). Of course we have aggressive auto-restart code but this is rarely triggered these days. While the inbound queries are getting more complex. Some of the tricks you have build into WQS are going to be a pain to redo in virtuoso. But I don't see anything impossible there. Pragmatically, while WDS is a Graph database, the queries are actually very relational. And none of the standard graph algorithms are used. To be honest RDF is actually a relational system which means that relational techniques are very good at answering them. The sole issue is recursive queries (e.g. rdfs:subClassOf+) in which the virtuoso implementation is adequate but not great. This is why recovering physical schemata from RDF data is such a powerful optimization technique [1]. i.e. you tend to do joins not traversals. This is not always true but I strongly suspect it will hold for the vast majority of the Wikidata Query Service case. I hope this was helpful, and I am willing to answer further questions. Regards, Jerven [1] https://research.vu.nl/files/61555276/complete%20dissertation.pdf and associated work that was done by Orri Erling. Which unfortunately has not yet landed in the Virtuoso master branch. On 6/10/19 9:49 PM, Guillaume Lederrey wrote:

and of course not to forget the fully open source SPARQL 1.1 compliant RDF database Apache Jena with TDB. Did you already evaluate Apache Jena for use in wikidata? On Tue, Jun 11, 2019 at 5:07 PM Andra Waagmeester &lt;andra(a)micel.io&gt; wrote: -- --- Marco Neumann KONA

Hi! It would be interesting to know more about this, if this is open source. Is there any more information about it online? If you mean algorithms like A* or PageRank, then yes, they are not used too much (likely also because SPARQL has no standard support for any of these, too), though Blazegraph implements some of them as custom services. Yes, path queries are pretty popular on WDQS too, especially given as many relationships like administrative/territorial placement or ownership are hierarchical and transitive, which often requires path queries. Would be interesting to see if we can apply anything from the article. Thanks for the link! -- Stas Malyshev smalyshev(a)wikimedia.org

Hi! Does this mean no sharding, i.e. each server stores the full DB? This is the model we're using currently, but given the growth of the data it may be non sustainable on current hardware. I see in your tables that Uniprot has about 30B triples, but I wonder how update loads there look like. Our main issue is that the hardware we have now is showing its limits when there's a lot of updates in parallel to significant query load. So I wonder if the "single server holds everything" model is sustainable in the long term. Again, here the question is not too much in "can you load 7bn triples into Virtuoso" - we know we can. What we want to figure out whether given specific query/update patterns we have now - it is going to give us significantly better performance allowing to support our projected growth. And also possibly whether Virtuoso has ways to make our update workflow be more optimal - e.g. right now if one triple changes in Wikidata item, we're essentially downloading and updating the whole item (not exactly since triples that stay the same are preserved but it requires a lot of data transfer to express that in SPARQL). Would there be ways to update the things more efficiently? Replication could certainly be useful I think it it's faster to update single server and then replicate than simultaneously update all servers (that's what is happening now). -- Stas Malyshev smalyshev(a)wikimedia.org

Changing the subject a bit: I am surprised to see how many SPARQL requests go to the endpoint when performing a ShEx validation with the shex-simple Toolforge tool. They are all very simple and quickly complete. For each Wikidata item tested, one of our tests [1] requests tens of times. That is, testing 100 Wikidata items may yield thousands of requests to the endpoint in rapid succession. I suppose that given the simple SPARQL queries, these kinds of requests might not load WDQS very much. [1] https://tools.wmflabs.org/shex-simple/wikidata/packages/shex-webapp/doc/she… Finn http://people.compute.dtu.dk/faan/

Dear, I've seen this suggestion on Quora : https://www.quora.com/Wouldnt-a-mix-database-system-that-handle-both-JSON-d… I'm not qualified enough to know if it is relevant but this could be some brainstorming. Regards Le mer. 19 juin 2019 à 19:45, Finn Aarup Nielsen &lt;faan(a)dtu.dk&gt; a écrit :

In the enterprise where I work as a Data Architect, we approach scaling in many ways, but there's no question that the age old technique of SORTING lines up everything for systems and cpu's to massively ingest and pipeline across IO boundaries. Sometimes this involves more indices, lots of duplicated data, and scatter/gather techniques. Knowing WHAT to sort and HOW to sort will vary widely by the queries that are expected to perform well from a system. So sorted indices of many kinds (where data is duplicated) are necessary to achieve extremely fast IO for a broad database such as Wikidata. Scaling problems can be categorized in a few buckets: 1. Increase of data queries. (READ) 2. Increase of data writes. (WRITE) 3. There is no 3. Because all scale problems boil down to IO (READ/WRITE) and how you approach fast IO. Google is known to replicate data at different levels of abstraction (metaschema, indices, meta-relations) across entire regions of the world in order to achieve fast IO. With a nearly unlimited budget they can MOVE THINGS FAST certainly and afford to be extremely wasteful and smart with data replication techniques. IBM approaches the scale problem via Polymorphic stores that support multiple indices, db structures, both in-memory and graph-like. Essentially, duplicating the hell out of the data in many, many ways and wasting space and memory to result in extremely high performance on queries. https://queue.acm.org/detail.cfm?id=3332266 Juan Sequeda (now at data.world) and team at Capsenta also seem to use polymorpic storage to bridge SPARQL and relational DB's. But I'm unsure of the actual architecture but would love to hear more about it. I've followed Juan for some time. https://www.zdnet.com/article/data-world-joins-forces-with-capsenta-to-brin… It is unfortunate that Wikidata doesn't have the hardware resources to duplicate and sort data in myriad ways to achieve better scale. On the software(s) side, we all know what the capabilities are of various stacks, but we often don't have the "time" or "hardware" to truly flex the "software" stack muscles to allow fast IO. Thad https://www.linkedin.com/in/thadguidry/

Hi! Agreed, if we are to get any solution that is not constrained by hardware limits of a single server, we can not avoid looking at sharding. Thanks, very interesting article. I'd certainly would like to know how this works with database in the size of 10 bln. triples and queries both accessing and updating random subsets of them. Updates are not covered very thoroughly there - this is, I suspect, because many databases of 10 bln. size do not have as active (non-append) update workload as we do. Maybe they still manage to solve it, if so, I'd very much like to know about it. Given how the original data is stored (JSON blob inside mysql table) it would not be very useful. In general, graph data model and Wikitext data model on top of which Wikidata is built are very, very different, and expecting same storage to serve both - at least without very major and deep refactoring of the code on both sides - is not currently very realistic. And of course moving any of the wiki production databases to Virtuoso would be a non-starter. Given than original Wikidata database stays on Mysql - which I think is a reasonable assumption - there would need to be a data migration pipeline for data to come from Mysql to whatever is the WDQS NG storage. Yes, relational-RDF bridges are known but my experience is they usually are not very performant (the difference in "you can do it" and "you can do it fast" is sometimes very significant) and in our case, it would be useless anyway as Wikidata data is not really stored in relational database per se - it's stored in JSON blob opaquely saved in relational database structure that knows nothing about Wikidata. Yes, it's not the ideal structure for optimal performance of Wikidata itself, but I do not foresee this changing, at least in any short term. Again, we could of course have data export pipeline to whatever storage format we want - essentially we already have one - but the concept of having single data store is probably not realistic at least within foreseeable timeframes. We use separate data store for search (ElasticSearch) and probably will have to have separate one for queries, whatever would be the mechanism. Thanks, -- Stas Malyshev smalyshev(a)wikimedia.org

Hi! Yes, but the way the data are stored now is JSON blob within a text field in MySQL. I do not see how RDF View over ODBC would help it any - of course Virtuoso would be able to fetch JSON text for a single item, but then what? We'd need to run queries across millions of items, fetching and parsing JSON for every one of them every time is unfeasible. Not to mention this JSON is not an accurate representation of the RDF data model. So I don't think it is worth spending time in this direction... I just don't see how any query engine could work with that storage. -- Stas Malyshev smalyshev(a)wikimedia.org

On 6/10/19 10:54 AM, Guillaume Lederrey wrote: Virtuoso Cluster Edition is as described by Sebastian in an earlier post to this thread [1]. Online that's behind our LOD Cloud cache which hosts 40 Billion+ triples, but still using ridiculously cheap hard-ware for the share-nothing cluster. As Jerven has already articulated [2], the single-server open source edition of Virtuoso can also scale to 40 Billion+ triples as demonstrated by Uniprot amongst others. There's a publicly available Google Spreadsheet that provides insights into a variety of Virtuoso configurations that you can also look at regarding resource requirements [3]. Bottom line, Virtuoso has no fundamental issues with performance, scale, or security (most haven't hit this bump yet, but its coming!) regarding RDF-data deployed in line with Linked Data principles. We are always opened to collaboration with anyone (or group) seeking to fully exploit the power and promise of a Semantic Web derived from Linked Data :) Links: [1] https://lists.wikimedia.org/pipermail/wikidata/2019-June/013132.html -- Sebastian Hellman comment [2] https://lists.wikimedia.org/pipermail/wikidata/2019-June/013143.html -- Jerven Bolleman comment [3] https://docs.google.com/spreadsheets/d/1-stlTC_WJmMU3xA_NxA1tSLHw6_sbpjff-5… <https://docs.google.com/spreadsheets/d/1-stlTC_WJmMU3xA_NxA1tSLHw6_sbpjff-5OITtrbFw/edit?ouid=112399767740508618350&usp=sheets_home&ths=true> -- Virtuoso configurations sample spreadsheet [4] https://hub.docker.com/u/openlink/ -- Docker Hub offerings [5] https://aws.amazon.com/marketplace/pp/B00ZWMSNOG -- Amazon Marketplace BYOL Edition [6] https://aws.amazon.com/marketplace/pp/B011VMCZ8K -- Amazon Marketplace PAGO Edition [7] https://github.com/openlink/virtuoso-opensource -- Github [8] http://download.openlinksw.com -- Download Site -- Regards, Kingsley Idehen Founder & CEO OpenLink Software Home Page: http://www.openlinksw.com Community Support: https://community.openlinksw.com Weblogs (Blogs): Company Blog: https://medium.com/openlink-software-blog Virtuoso Blog: https://medium.com/virtuoso-blog Data Access Drivers Blog: https://medium.com/openlink-odbc-jdbc-ado-net-data-access-drivers Personal Weblogs (Blogs): Medium Blog: https://medium.com/@kidehen Legacy Blogs: http://www.openlinksw.com/blog/~kidehen/ http://kidehen.blogspot.com Profile Pages: Pinterest: https://www.pinterest.com/kidehen/ Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen Twitter: https://twitter.com/kidehen Google+: https://plus.google.com/+KingsleyIdehen/about LinkedIn: http://www.linkedin.com/in/kidehen Web Identities (WebID): Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this

Hi Stas, thanks for the elaboration. I can understand the background much better. I have to admit, that I am also not a real expert, but very close to the real experts like Vidal and Rahm who are co-authors of the SWJ paper or the OpenLink devs. I am also spoiled, because OpenLink solves the hosting for DBpedia and also DBpedia-live with ca. 130k updates per day for the English Wikipedia. I think, this one is the recent report: https://medium.com/virtuoso-blog/dbpedia-usage-report-as-of-2018-01-01-8cae… Then again DBpedia didn't grow for a while, but we made a "Best of" now [1]. But  will not host it all. [1] https://svn.aksw.org/papers/2019/ISWC_FlexiFusion/public.pdf I also see that your context is difficult. Maybe you can custom shard/scaleout Blazegraph based on the queries and then replicate the sharded clusters. Like a mix between sharding and replication, maybe just 3 * 3 servers instead of 9 replicated ones or 9 servers full of shards. Not so many options here, if you have the OS requirement. I guess you are caching static content already as much as possible. This also seems pretty much what I know, but it really is all second hand as my expertise is more focused on what's inside the database. All the best, Sebastian On 10.06.19 22:02, Stas Malyshev wrote: -- All the best, Sebastian Hellmann Director of Knowledge Integration and Linked Data Technologies (KILT) Competence Center at the Institute for Applied Informatics (InfAI) at Leipzig University Executive Director of the DBpedia Association Projects: http://dbpedia.org, http://nlp2rdf.org, http://linguistics.okfn.org, https://www.w3.org/community/ld4lt <http://www.w3.org/community/ld4lt> Homepage: http://aksw.org/SebastianHellmann Research Group: http://aksw.org

Yes, I can ask. I am talking a lot with them as we are redeploying DBpedia live and also pushing the new DBpedia to them soon. I think, they also had a specific issue with how Wikidata does linked data, but I didn't get it, as it was mentioned too briefly. All the best, Sebastian On 10.06.19 22:46, Stas Malyshev wrote: -- All the best, Sebastian Hellmann Director of Knowledge Integration and Linked Data Technologies (KILT) Competence Center at the Institute for Applied Informatics (InfAI) at Leipzig University Executive Director of the DBpedia Association Projects: http://dbpedia.org, http://nlp2rdf.org, http://linguistics.okfn.org, https://www.w3.org/community/ld4lt <http://www.w3.org/community/ld4lt> Homepage: http://aksw.org/SebastianHellmann Research Group: http://aksw.org