On Thu, Oct 18, 2012 at 11:22:05AM -0700, Asher Feldman wrote:

I think Solr is the right direction for us to go in. Current efforts can pave the way for a complete refresh of WMF's article full text search as well as how our developers approach information retrieval. We just need to make sure that these efforts are unified, with commonality around the client api, configuration, indexing (preferably with updates asynchronously pushed to Solr in near real-time), and schema definition. This is important from an operational aspect as well, where it would be ideal to have a single distributed and redundant cluster.

I'm curious, has anyone evaluated ElasticSearch and whether it'd be more or less suitable for us than Solr? If so, I'd be very interested in the comparison results for our use cases.

Regards, Faidon

That is great to hear. Thanks for tying us together, Asher.

For Wikidata, we have not uploaded our Solr extension yet (mostly because we are waiting for the repository to be set up), but we will then upload it soon once it is there. I would be especially interested in sharing schema and config snippets for the many languages we support, but as far as I can tell this is not so much a requirement for Max and maybe not even for the TranslateMemory, not sure.

We also selected Solarium to connect to Solr. It is a bit worrysome that it is basically a one person project if I see this correctly, but the library seems small enough not to pose the risk of becoming too much of a maintenance legacy I'd say -- especially compared to the alternatives.

Any preferences for Solr 3 v 4? It seems that 4 is the smarter choice, but we are having trouble to get 4 run on labs. Solr 3 works pretty much out of the box, though.

Also, we should probably at some point consider how the different extensions and their dependencies should be handled. I'd prefer not to ship three different versions of Solarium with three extensions :)

Cheers, Denny

P.S.: Yuri, regarding GESIS' Solr implementation, they have done some great work for using Solr as a store for the structured data in SMW. Funny thing is, they actually do not use Solr for the search itself! This work is somewhat relevant for Wikidata phase 3, but unfortunately quite irrelevant for TranslationMemory or Geodata.

2012/10/18 Asher Feldman afeldman@wikimedia.org:

Hi all,

I'm excited to see that Max has made a lot of great progress in adding Solr support to the GeoData extension so that we don't have to use mysql for spatial search - https://gerrit.wikimedia.org/r/#/c/27610/

GeoData makes use of the Solarium php client, which is currently included as a part of the extension. GeoData will be our second use of Solar, after TranslationMemory extension which is already deployed - https://www.mediawiki.org/wiki/Help:Extension:Translate/Translation_memories and the Wikidata team is working on using Solr in their extensions as well.

TranslationMemory also uses Solarium, a copy of which is also bundled with and loaded from the extension. For a loading and config example - https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=bl...

I think Solr is the right direction for us to go in. Current efforts can pave the way for a complete refresh of WMF's article full text search as well as how our developers approach information retrieval. We just need to make sure that these efforts are unified, with commonality around the client api, configuration, indexing (preferably with updates asynchronously pushed to Solr in near real-time), and schema definition. This is important from an operational aspect as well, where it would be ideal to have a single distributed and redundant cluster.

It would be great to see the i18n, mobile tech, wikidata, and any other interested parties collaborate and agree on a path forward, with a quick sprint around common code that all can use.

-Asher