Hi! Today we are indexing in ElasticSearch almost all string properties (except a few) and select item properties (P31 and P279). We've been asked to extend this set and index more item properties (https://phabricator.wikimedia.org/T199884). We did not do it from the start because we did not want to add too much data to the index at once, and wanted to see how the index behaves. To evaluate what this change would mean, some statistics: All usage of item properties in statements is about 231 million uses (according to sqid tool database). Of those, about 50M uses are "instance of" which we are already indexing. Another 98M uses belong to two properties - published in (P1433) and cites (P2860). Leaving about 86M for the rest of the properties. So, if we index all the item properties except P2860 and P1433, we'll be a little more than doubling the amount of data we're storing for this field, which seems OK. But if we index those too, we'll be essentially quadrupling it - which may be OK too, but is bigger jump and one that may potentially cause some issues. So, we have two questions: 1. Do we want to enable indexing for all item properties? Note that if you just want to find items with certain statement values, Wikidata Query Service matches this use case best. It's only in combination with actual fulltext search where on-wiki search is better. 2. Do we need to index P2860 and P1433 at all, and if so, would it be ok if we omit indexing for now? Would be glad to hear thoughts on the matter. Thanks, -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi, and thanks for working on this! My subjective view: * We don't need P2860/P1433 indexed, at least not at the moment * I would really like dates (mainly, born/died), especially if they work for "greater units", that is, I search for a year and get an item back, even though the statament is month- or day-precise Cheers, Magnus On Thu, Jul 26, 2018 at 10:48 PM Stas Malyshev <smalyshev(a)wikimedia.org> wrote: _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

+1 with Magnus on years of birth and death (but perhaps /only/ years of birth and death, or close surrogates eg years of baptism and burial, and inception or publication date for things, otherwise the search specificity would become useless with too many other 'significant event' dates)

What I'd try to avoid in general is indexing terms that have only doc since they are pretty useless.

I think we already index way more than P31 and P279.

So I think that the increase is smaller than what you anticipate. What I'd try to avoid in general is indexing terms that have only doc since they are pretty useless.

I think we should investigate what kind of data we may have here, and at least for statement_keywords I would not index data that contain random text (esp. natural language) since they are prone to be unique and impossible to search.

The top 1000 is: https://docs.google.com/spreadsheets/d/1E58W_t_o6vTNUAx_TG3ifW6-eZE4KJ2VGEa…

* I would really like dates (mainly, born/died), especially if they work for "greater units", that is, I search for a year and get an item back, even though the statament is month- or day-precise

So, we have two questions: 1. Do we want to enable indexing for all item properties? Note that if you just want to find items with certain statement values, Wikidata Query Service matches this use case best. It's only in combination with actual fulltext search where on-wiki search is better.

2. Do we need to index P2860 and P1433 at all, and if so, would it be ok if we omit indexing for now?

Thanks a lot for looking into this, Stas! On Thu, Jul 26, 2018 at 11:49 PM Stas Malyshev <smalyshev(a)wikimedia.org> wrote: I would say yes. Yes it should be perfectly fine to go without these for now - maybe always. They're mostly (only?) used on the large corpus of scientific papers. Cheers Lydia -- Lydia Pintscher - http://about.me/lydia.pintscher Product Manager for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hi! I don't think it's *very* poor anymore, but it certainly can be better. Err.... The problem is that what you see as a human and what search engine uses for lookups are very different things. While for text articles it is similar, for structured data it's quite different, and treating structured data the same way as text is not going to produce good results, partially because most search algorithms make assumptions that come from text world, partially because we'd be ignoring useful clues present in structured data. There are such fields, but it makes no sense to put references there, because there's no such thing as "Dutch reference". References do not change with language. That presents a problem. While you see "instance of": "human", the data is P31:Q5. We can, of course, put "instance of": "human" in the index. But what if label for Q5 changes? Now we have to re-index 10 million records. And while we're doing it, what if another label for such item changes again? We'd have to start another million-size reindex. In a week, we'd have a backlog of hopeless size, or will require processing power that we just don't have. Note also that ElasticSearch doesn't really do document updates - it just writes a new document. So frequent updates to the same document is not its optimal scenario, and we're talking about propagating each label edit to each item that is linked to that one. I'm afraid that would explode on us very quickly. The problem is not indexing labels, the problem is keeping them up-to-date on 50 million interlinked items. When displaying, it's easy - you don't need to worry until you show it, and most items are shown only rarely. Even then you see a label out of date now and then. But with search, you can't update label on use - when you want to use it (i.e. look up), it should already be up-to-date, otherwise it's useless. I don't know any such special format, do you? We of course have JSON updates to ElasticSearch, but as I noted before, updates are the problem there, not format. RDF of course also does not carry denormalized data, so we also update only entries that need updating, and fetch labels on use. We can not do it for search index. I don't think format here is the problem. Sure, but experience have shown that the strategy of "dump everything into one huge text" works very poorly in Wikidata. That's why we implemented specialized search that knows about how the structured data works. If the search sucks less now than it did before, that's the reason. Wikidata search tasks would be under "Wikidata" + "Discovery-Search". There are multiple tasks for it, but if you want to add any, please feel welcome to browse and add. -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

I don't know any such special format, do you?

Statements are indexed, but you have to use haswbstatement with specific property to look for them.

This is a bit tangential to the topic, but isn’t that basically what schema.org was developed for? (I’m not sure if that’s still its primary purpose, but as far as I know it was started by a group of search engines to develop a unified format websites could use to make their semantics more accessible to those search engines.)