Hi!

Today we are indexing in ElasticSearch almost all string properties
(except a few) and select item properties (P31 and P279). We've been
asked to extend this set and index more item properties
(https://phabricator.wikimedia.org/T199884). We did not do it from the
start because we did not want to add too much data to the index at once,
and wanted to see how the index behaves. To evaluate what this change
would mean, some statistics:

All usage of item properties in statements is about 231 million uses
(according to sqid tool database). Of those, about 50M uses are
"instance of" which we are already indexing. Another 98M uses belong to
two properties - published in (P1433) and cites (P2860). Leaving about
86M for the rest of the properties.

So, if we index all the item properties except P2860 and P1433, we'll be
a little more than doubling the amount of data we're storing for this
field, which seems OK. But if we index those too, we'll be essentially
quadrupling it - which may be OK too, but is bigger jump and one that
may potentially cause some issues.

So, we have two questions:
1. Do we want to enable indexing for all item properties? Note that if
you just want to find items with certain statement values, Wikidata
Query Service matches this use case best. It's only in combination with
actual fulltext search where on-wiki search is better.

2. Do we need to index P2860 and P1433 at all, and if so, would it be ok
if we omit indexing for now?

Would be glad to hear thoughts on the matter.

Thanks,
--
Stas Malyshev
smalyshev@wikimedia.org

_______________________________________________
discovery-private mailing list
discovery-private@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/discovery-private