Hi!
Today we are indexing in ElasticSearch almost all string properties (except a few) and select item properties (P31 and P279). We've been asked to extend this set and index more item properties (https://phabricator.wikimedia.org/T199884). We did not do it from the start because we did not want to add too much data to the index at once, and wanted to see how the index behaves. To evaluate what this change would mean, some statistics:
All usage of item properties in statements is about 231 million uses (according to sqid tool database). Of those, about 50M uses are "instance of" which we are already indexing. Another 98M uses belong to two properties - published in (P1433) and cites (P2860). Leaving about 86M for the rest of the properties.
So, if we index all the item properties except P2860 and P1433, we'll be a little more than doubling the amount of data we're storing for this field, which seems OK. But if we index those too, we'll be essentially quadrupling it - which may be OK too, but is bigger jump and one that may potentially cause some issues.
So, we have two questions: 1. Do we want to enable indexing for all item properties? Note that if you just want to find items with certain statement values, Wikidata Query Service matches this use case best. It's only in combination with actual fulltext search where on-wiki search is better.
2. Do we need to index P2860 and P1433 at all, and if so, would it be ok if we omit indexing for now?
Would be glad to hear thoughts on the matter.
Thanks,