I was wondering if anyone knew if there was a limit on the number of Q-items (or L-items) one could create on a wikibase-cloud project?
In theory, no. There are no hardcoded limits or anything like that.
In practice, if your dataset becomes too large and complex you may run out of memory on your wiki’s Blazegraph instance, and as this is a free service it’s not exactly unlimited. That said, the largest Wikibase I am hosting on WBC has over 300,000 items and I haven’t run into any issues yet. How large is the dataset you have in mind?
Thank you, James Hare On Feb 6, 2024 at 7:23 PM -0800, Matthew Ong matthewcong@gmail.com, wrote:
I was wondering if anyone knew if there was a limit on the number of Q-items (or L-items) one could create on a wikibase-cloud project? _______________________________________________ Wikibase-cloud mailing list -- wikibase-cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/wikibase-cloud.lists.wikimedia.o...
It could be about the size of yours (300,000+). If I wanted to avoid that size I would need to rethink my project design a bit...
The other dimension is going to be the number of statements per item. Assuming you had 500,000 items, I think each item would need to have 40 or more statements before you ran into real performance issues. Then again, I haven’t actually stress tested this (nor am I inclined to; I think management would prefer if I didn’t).
Thank you, James Hare On Feb 7, 2024 at 9:56 AM -0800, Matthew Ong matthewcong@gmail.com, wrote:
It could be about the size of yours (300,000+). If I wanted to avoid that size I would need to rethink my project design a bit... _______________________________________________ Wikibase-cloud mailing list -- wikibase-cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/wikibase-cloud.lists.wikimedia.o...
My understanding is that it is the rate of edits that causes the most issues. Data at rest has less cost, and IIRC Blazegraph scales logarithmically (although the main Wikibase database is linear).
Try to avoid busy times, aim to run it overnight, assuming most Wikibase.cloud users are doing work in work hours - this is not the case for other sites, but it has a high proportion of research users. And if you get 400 or 500 errors, back off. (Graphs for Wikibase.cloud load might help.)
If loading a large number of items, try to do each item in one edit, because adding an item and then modifying it to add statements as some scripts do may multiply the work for the servers.
-- Laurence 'GreenReaper' Parry
________________________________ From: James Hare james@scatter.red Sent: 08 February 2024 22:45 To: wikibase-cloud@lists.wikimedia.org wikibase-cloud@lists.wikimedia.org; Matthew Ong matthewcong@gmail.com Subject: [Wikibase-cloud] Re: Question on number of Q-items
The other dimension is going to be the number of statements per item. Assuming you had 500,000 items, I think each item would need to have 40 or more statements before you ran into real performance issues. Then again, I haven’t actually stress tested this (nor am I inclined to; I think management would prefer if I didn’t).
Thank you, James Hare On Feb 7, 2024 at 9:56 AM -0800, Matthew Ong matthewcong@gmail.com, wrote: It could be about the size of yours (300,000+). If I wanted to avoid that size I would need to rethink my project design a bit...
wikibase-cloud@lists.wikimedia.org