Hi! Indeed, Wikidata grows and will continue growing. But I don't see clearly what the purpose of this thread is. Is it to propose possible technical and financial improvements? Regards, David On 5/3/19 14:24, Gerard Meijssen wrote: -- David Abián _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Gerard mentioned the PROBLEM in the 2nd sentence. I read it clearly.... is so bad that people are asked what kind of updates they are running because it makes a difference in the lag times there are. The response times are typically attributed to SPARQL queries from what I have seen, as well as applying multiple edits with scripts or mass operations. Although I recall there is a light queue mechanism inherent in the Blazegraph architecture that contributes to this, and I am fine with slower writes. What most users are not comfortable with is the slower reads in different areas of Wikidata. We need to identify those slow read areas or figure out a way to get consensus on what parts of Wikidata reading affect our users the most. So let's be constructive here: Gerard - did you have specific areas that affect your daily work, and what from of work is that (reading/writing , which areas) ? Thad https://www.linkedin.com/in/thadguidry/ _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Gerard, I like wikidata a lot, kudos to the community for keeping it going. But keep it real, there is no exponential growth here. We are looking at a slow and sustainable growth at the moment with possibly a plateauing of number of users and when it comes to total number of wikidata items. just take a look at the statistics. Date | Content pages | Page edits since Wikidata was set up | Registered users | Active users 4/2015 | 13,911,417 | 213,027,375 | 1,913,828 | 15,168 5/2016 | 17,432,789 | 328,781,525 | 2,688,788 | 16,833 7/2017 | 28,037,196 | 514,252,789 | 2,835,219 | 18,081 7/2018 | 49,081,962 | 701,319,718 | 2,970,150 | 18,578 4/2019 | 56,377,647 | 931,449,205 | 3,236,569 | 20,857 When you refer to "growing like a weed". What's that page views? queries per day? Mentions in the media? Best, Marco On Fri, May 3, 2019 at 3:36 PM Gerard Meijssen <gerard.meijssen(a)gmail.com> wrote: -- --- Marco Neumann KONA _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hoi, Wikidata grows like mad. This is something we all experience in the really bad response times we are suffering. It is so bad that people are asked what kind of updates they are running because it makes a difference in the lag times there are. Given that Wikidata is growing like a weed, it follows that there are two issues. Technical - what is the maximum that the current approach supports - how long will this last us. Fundamental - what funding is available to sustain Wikidata. For the financial guys, growth like Wikidata is experiencing is not something you can reliably forecast. As an organisation we have more money than we need to spend, so there is no credible reason to be stingy. For the technical guys, consider our growth and plan for at least one year. When the impression exists that the current architecture will not scale beyond two years, start a project to future proof Wikidata. It will grow and the situation will get worse before it gets better. Thanks, GerardM PS I know about phabricator tickets, they do not give the answers to the questions we need to address. _______________________________________________ Wikidata mailing list Wikidata@lists.wikimedia.org<mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata<https://lists.wiki…

really bad kind of there are. As I've delved deeper into Wikidata I get the feeling it is being developed with the assumptions of infinite resources, and no strong guidelines of exactly what the scope is (i.e. where you draw the line between what belongs in Wikidata and what does not). This (and concerns of it being open to data vandalism) has personally made me back-off a bit. I'd originally planned to have Wikidata be the primary data source, but I'm now leaning towards keeping data tables and graphs outside, with scheduled scripts to import into Wikidata, and export from Wikidata. year. The 37GB (json, bz2) data dump file (it was already 33GB, twice the size of the English wikipedia dump, when I grabbed it last November) is unwieldy. And, as there is no incremental changes being published, it is hard to create a mirror. Can that dump file be split up in some functional way, I wonder? Darren -- Darren Cook, Software Researcher/Developer _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Erik Paulson, 12/05/19 01:54: In fact that was one of the proposals in "The future of bibliographic data in Wikidata: 4 possible federation scenarios". https://www.wikidata.org/wiki/Wikidata:WikiCite/Roadmap Federico _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi! We may also want to consider if Wikidata is actually the best store for all kinds of data. Let's consider example: https://www.wikidata.org/w/index.php?title=Q57009452 This is an entity that is almost 2M in size, almost 3000 statements and each edit to it produces another 2M data structure. And its dump, albeit slightly smaller, still 780K and will need to be updated on each edit. Our database is obviously not optimized for such entities, and they won't perform very well. We have 21 million scientific articles in the DB, and if even 2% of them would be like this, it's almost a terabyte of data (multiplied by number of revisions) and billions of statements. While I am not against storing this as such, I do wonder if it's sustainable to keep such kind of data together with other Wikidata data in a single database. After all, each query that you run - even if not related to that 21 million in any way - will have to still run in within the same enormous database and be hosted on the same hardware. This is especially important for services like Wikidata Query Service where all data (at least currently) occupies a shared space and can not be easily separated. Any thoughts on this? -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi! True, WDQS does not have versions. But each time the edit is made, we now have to download and work through the whole 2M... It wasn't a problem when we were dealing with regular-sized entities, but current system certainly is not good for such giant ones. As for decentralizing, WDQS supports federation, but for obvious reasons federated queries are slower and less efficient. That said, if there were separate store for such kind of data, it might work as cross-querying against other Wikidata data wouldn't be very frequent. But this is something that Wikidata community needs to figure out how to do. -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi! We may also want to consider if Wikidata is actually the best store for all kinds of data. Let's consider example: https://www.wikidata.org/w/index.php?title=Q57009452 This is an entity that is almost 2M in size, almost 3000 statements and each edit to it produces another 2M data structure. And its dump, albeit slightly smaller, still 780K and will need to be updated on each edit. Our database is obviously not optimized for such entities, and they won't perform very well. We have 21 million scientific articles in the DB, and if even 2% of them would be like this, it's almost a terabyte of data (multiplied by number of revisions) and billions of statements. While I am not against storing this as such, I do wonder if it's sustainable to keep such kind of data together with other Wikidata data in a single database. After all, each query that you run - even if not related to that 21 million in any way - will have to still run in within the same enormous database and be hosted on the same hardware. This is especially important for services like Wikidata Query Service where all data (at least currently) occupies a shared space and can not be easily separated. Any thoughts on this?

Hi! We may also want to consider if Wikidata is actually the best store for all kinds of data. Let's consider example: https://www.wikidata.org/w/index.php?title=Q57009452 This is an entity that is almost 2M in size, almost 3000 statements and each edit to it produces another 2M data structure. And its dump, albeit slightly smaller, still 780K and will need to be updated on each edit. Our database is obviously not optimized for such entities, and they won't perform very well. We have 21 million scientific articles in the DB, and if even 2% of them would be like this, it's almost a terabyte of data (multiplied by number of revisions) and billions of statements. While I am not against storing this as such, I do wonder if it's sustainable to keep such kind of data together with other Wikidata data in a single database. After all, each query that you run - even if not related to that 21 million in any way - will have to still run in within the same enormous database and be hosted on the same hardware. This is especially important for services like Wikidata Query Service where all data (at least currently) occupies a shared space and can not be easily separated. Any thoughts on this? -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

We may also want to consider if Wikidata is actually the best store for all kinds of data. Let's consider example: https://www.wikidata.org/w/index.php?title=Q57009452 This is an entity that is almost 2M in size, almost 3000 statements ...

While I am not against storing this as such, I do wonder if it's sustainable to keep such kind of data together with other Wikidata data in a single database.

Indeed, these collaborations in high-energy physics are not static quantities, they change essentially every day (people getting hired and had their contract expired, and most likely every two papers have a slightly different author list. Cheers Yaroslav On Sun, May 5, 2019 at 5:58 PM Darren Cook &lt;darren(a)dcook.org <mailto:darren@dcook.org>> wrote: store for statements ... A paper with 2884 authors! arxiv.org <http://arxiv.org> deals with it by calling them the "Atlas Collaboration": https://arxiv.org/abs/1403.0489 The actual paper does the same (with the full list of names and affiliations in the Appendix). The nice thing about graph databases is we should be able to set author to point to an "Atlas Collaboration" node, and then have that node point to the 2884 individual author nodes (and each of those nodes point to their affiliation). What are the reasons to not re-organize it that way? My first thought was that who is in the collaboration changes over time? But does it change day to day, or only change each academic year? Either way, maybe we need to point the author field to something like "Atlas Collaboration 2014a", and clone-and-modify that node each time we come to a paper that describes a different membership? Or is it better to do each persons membership of such a group with a start and end date? (BTW, arxiv.org <http://arxiv.org> tells me there are 1059 results for ATLAS Collaboration; don't know if one "result" corresponds to one "paper", though.) Wikidata data It feels like it belongs in "core" Wikidata. Being able to ask "which papers has this researcher written?" seems like a good example of a Wikidata query. Similarly,  "which papers have The ATLAS Collaboration" worked on?" But, also, are queries like "Which authors of Physics papers went to a high school that had more than 1000 students?" part of the goal of Wikidata? If so, Wikidata needs optimizing in such a way that makes such queries both possible and tractable. Darren _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata