Forwarding to the Discovery mailing list, although it sounds like the OP hopes to have any possible discussion on Wikitech-l.

I wonder if there would be ways for WMF Discovery to leverage the work that's being done already on Commoncrawl and Commonsearch for use in Wikimedia internal search.

Pine

---------- Forwarded message ---------- From: Sylvain Zimmer sylvain@sylvainzimmer.com Date: Sun, Mar 6, 2016 at 11:46 AM Subject: [Wikitech-l] Using Wikipedia/Wikidata in a nonprofit search engine To: wikitech-l@lists.wikimedia.org

Hi,

Some of you may be familiar with http://commoncrawl.org ; they are doing an excellent job of making large crawls of the web accessible to everyone.

I've been working on an open search engine based on these crawls for a while, and I would love to have your feedbacks on the project: https://about.commonsearch.org/

Specifically, I would be curious to know what you would consider to be the best possible integration of Wikipedia & Wikidata in a general search engine?

As a first step, we have just started using the "official website" property from Wikidata and we are considering importing the Wikipedia abstracts next (https://github.com/commonsearch/cosr-back/issues/11).

I'm looking forward to your feedbacks... or contributions! :-)

Thanks in advance,

PS: A few wikimedians recommended me to post on wikitech-l to keep the focus on the technical aspects of the project and hopefully avoid linking this project in any way to the KE stuff, which it actually predates by far (https://news.ycombinator.com/item?id=6209088).

-- Sylvain Zimmer http://sylvinus.org

_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l