Request For Comments - Extension:SphinxSearch - Wikitech-l

List overview All Threads
Download

newer

Request For Comments - Extension:SphinxSearch

older

Re: [Wikitech-l] Request For...

Good thing we use PHP

Paul Grinberg

25 Sep 2007 25 Sep '07

2:58 a.m.

Hello, I just uploaded the first version of a new extension called SphinxSearch ( http://www.mediawiki.org/wiki/Extension:SphinxSearch ). This extension uses the Sphinx Search engine as the backend for full text searches of wiki content. See the MW page for full description of the extension. Since this is the first release of the extension, and since searching is such a fundamental requirement of a wiki-type webpage, I wanted to submit this extension for your scrutiny. I would like to get feedback on not only the extension's functionality, but also the clarity and completeness of the installation instructions. At the same time, I have a question to the community. Is there a good way to change the default behavior of the Search box on the left of every page (rendered with monobook that is)? In other words, my extension adds a new Special Page. I would like the search box to use my special page instead of the default Special:Search. The reason for this is because I do not want to completely get rid of the build-in search engine ... at least not yet. Many thanks in advance, paul.

Show replies by date

Simetrical

25 Sep 25 Sep

3:39 a.m.

You are aware of the LuceneSearch extension used by Wikipedia, right? Not that there's anything wrong with making a Sphinx plugin too, but your introductory preamble on the linked page and your apparent uncertainty as to how to proceed in replacing the default search suggest to me that you might not be aware of it to begin with. Incidentally, there's no mileage in having access to the default search. It's slow to even maintain the indexes. Anyone using a better search solution would be best off dropping the relevant indexes/tables or otherwise disabling it for performance reasons if nothing else.

Maury Markowitz

12:40 p.m.

On 9/24/07, Simetrical <Simetrical+wikilist(a)gmail.com> wrote:

...

Incidentally, there's no mileage in having access to the default search. It's slow to even maintain the indexes. Anyone using a better search solution would be best off dropping the relevant indexes/tables or otherwise disabling it for performance reasons if nothing else.

Out of curiosity, what is the reason we just don't use Google (or such)? The search capabilities in en.wiki are almost completely useless, IMHO, it can't even find articles with identical names if the capitalization is wrong, and it completely lacks anything like spell checking or reasonable relevance rankling. In order to find articles on en.wiki, I invariably open a second browser to search in, and this strikes me as rather sub-optimal. Is there anything I can do about this at "my end"? Maury

Magnus Manske

1:15 p.m.

On 9/25/07, Maury Markowitz <maury.markowitz(a)gmail.com> wrote:

...

On 9/24/07, Simetrical <Simetrical+wikilist(a)gmail.com> wrote:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar. Cheers, Magnus

Magnus Manske

1:18 p.m.

On 9/25/07, Magnus Manske <magnusmanske(a)googlemail.com> wrote:

...

On 9/25/07, Maury Markowitz <maury.markowitz(a)gmail.com> wrote:

On 9/24/07, Simetrical <Simetrical+wikilist(a)gmail.com> wrote:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar. Cheers, Magnus

D'oh! There's already one at [[Wikipedia:WikiProject User scripts/Scripts/Google link]] Oh well...

Paul Grinberg

2:36 p.m.

Magnus Manske <magnusmanske@...> writes:

...

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar. Cheers, Magnus

D'oh! There's already one at [[Wikipedia:WikiProject User scripts/Scripts/Google link]] Oh well...

I'm not able to view your googlesearch.js nor the wikipedia one. However, if I understand the premise of this code, it basically adds a search form to submint the requested query to google.com which will then search only your domain. This application only works for wiki's that can be seen from the outside world. In the case of my wiki, which is a corporate repository if information, it is behind a firewall, such an approach simply would not work. Hence the need for a local search engine like Sphinx. Paul.

Magnus Manske

3:10 p.m.

On 9/25/07, Paul Grinberg <gri6507(a)yahoo.com> wrote:

...

Magnus Manske <magnusmanske@...> writes:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar. Cheers, Magnus

D'oh! There's already one at [[Wikipedia:WikiProject User scripts/Scripts/Google link]] Oh well...

The google search has nothing to do with the lucene search. lucene works locally, like sphinx. Magnus

Schneelocke

1:23 p.m.

2007/9/25, Magnus Manske <magnusmanske(a)googlemail.com>om>:

...

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar.

That's neat. Is it possible to do something similar when you're using the Classic skin, too? -- schnee

Magnus Manske

2 p.m.

On 9/25/07, Schneelocke <schneelocke(a)gmail.com> wrote:

...

2007/9/25, Magnus Manske <magnusmanske(a)googlemail.com>om>:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar.

That's neat. Is it possible to do something similar when you're using the Classic skin, too?

I just tried, but there's lots'o'stuff missing there (e.g., "id=xxx"), so it will be hard to do. Possible, though. Magnus

Magnus Manske

2:06 p.m.

On 9/25/07, Magnus Manske <magnusmanske(a)googlemail.com> wrote:

...

On 9/25/07, Schneelocke <schneelocke(a)gmail.com> wrote:

2007/9/25, Magnus Manske <magnusmanske(a)googlemail.com>om>:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar.

That's neat. Is it possible to do something similar when you're using the Classic skin, too?

I just tried, but there's lots'o'stuff missing there (e.g., "id=xxx"), so it will be hard to do. Possible, though.

OK, got it running :-) Put importScript('User:Magnus Manske/googlesearch.js'); on your User:XXX/standard.js page Cheers, Magnus

Platonides

2:23 p.m.

Magnus Manske wrote:

...

On 9/25/07, Magnus Manske wrote:

On 9/25/07, Schneelocke wrote:

2007/9/25, Magnus Manske wrote:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar.

That's neat. Is it possible to do something similar when you're using the Classic skin, too?

I just tried, but there's lots'o'stuff missing there (e.g., "id=xxx"), so it will be hard to do. Possible, though.

OK, got it running :-) Put importScript('User:Magnus Manske/googlesearch.js'); on your User:XXX/standard.js page Cheers, Magnus

You're fast :) Instead of using if (skin == "standard"), why not using if (!<result of document.getElementById(<missingid>)>) ?

Schneelocke

2:40 p.m.

2007/9/25, Magnus Manske <magnusmanske(a)googlemail.com>om>:

...

OK, got it running :-) Put importScript('User:Magnus Manske/googlesearch.js'); on your User:XXX/standard.js page

Wonderful, thank you. :) -- schnee

Platonides

2:05 p.m.

Schneelocke wrote:

...

2007/9/25, Magnus Manske:

Freshly hacked: Add importScript('User:Magnus Manske/googlesearch.js'); to your monobook.js page. This will add a "google search" button in your sidebar.

That's neat. Is it possible to do something similar when you're using the Classic skin, too?

You'd usually put it on User:yourname/classic.js instead. The problem here is that classic skin misses several ids. Monobook search: <form action="/wiki/Special:Search" id="searchform"><div> <input id="searchInput" name="search" type="text" title="Search Wikipedia [f]" accesskey="f" value="" /> <input type='submit' name="go" class="searchButton" id="searchGoButton" value="Go" />  <input type='submit' name="fulltext" class="searchButton" id="mw-searchButton" value="Search" /> </div></form> Classic search: <form name="search" class="inline" method="post" action="/wiki/Special:Search"> <input type="text" name="search" size="19" value="" /> <input type="submit" name="go" value="Go" /> <input type="submit" name="fulltext" value="Search" /> </form>

Brion Vibber

2:30 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Maury Markowitz wrote:

...

Out of curiosity, what is the reason we just don't use Google (or such)?

Because being entirely dependent on the proprietary technology of a third-party company is unwise (lock-in dependency, privacy issues) and contrary to our organizational goals (maintain the site based on open technology). - -- brion vibber (brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG+RuQwRnhpk1wk44RAgHMAKCzF1JmBFXgi9thjEBQQRRzze4u0wCfdfmo MYS48hVvTXXk2f+uBwOGEsw= =5eJE -----END PGP SIGNATURE-----

Rob Church

3:10 p.m.

On 25/09/2007, Maury Markowitz <maury.markowitz(a)gmail.com> wrote:

...

I'm delighted to hear that all the excellent work that's been put into developing Lucene Search for Wikipedia as of late, as well as the ongoing work, is so easily characterised as pointless, or perhaps you haven't tried searching for anything lately - it's getting better. We don't "just use Google" because we'd like our users to remain within the same site when searching it, to avoid confusion due to an inconsistent user experience, and the only means Google would provide us with to avoid that are either too expensive to justify, or proprietary, something we're desperately committed to avoiding. Rob Church

Jim Wilson

4:45 p.m.

Regarding Sphinx, I haven't used it, but I'm excited that there's a feasible non-java alternative to Lucene. Don't get me wrong, I like Lucene in principle - however many hosts don't permit Java apps (barring significant cost increase). Ferret[1] may also work, though I don't know if anyone's attempted using it for MW search. Is there any work being done on a native PHP alternative? Something which could conceivably plug into the ArticleSaveComplete hook and keep a text index up-to-date? [1] http://ferret.davebalmain.com/trac/ -- Jim R. Wilson (jimbojw) On 9/25/07, Rob Church <robchur(a)gmail.com> wrote:

...

On 25/09/2007, Maury Markowitz <maury.markowitz(a)gmail.com> wrote:

Brion Vibber

5:29 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Wilson wrote:

...

Is there any work being done on a native PHP alternative? Something which could conceivably plug into the ArticleSaveComplete hook and keep a text index up-to-date?

There's a PHP version of Lucene in the Zend Framework stuff, though I've never tried it. No idea how performance compares to Lucene/Java or to the MySQL fulltext search. - -- brion vibber (brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG+UWEwRnhpk1wk44RAseIAKDJf/fc9XNd/MdSzCGhIT3uUHLAEQCgk0TV AysY/tWnJxh9C2XZx/T8my8= =kcuu -----END PGP SIGNATURE-----

Brion Vibber

2:32 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Paul Grinberg wrote:

...

At the same time, I have a question to the community. Is there a good way to change the default behavior of the Search box on the left of every page (rendered with monobook that is)? In other words, my extension adds a new Special Page. I would like the search box to use my special page instead of the default Special:Search. The reason for this is because I do not want to completely get rid of the build-in search engine ... at least not yet.

I'd recommend simply plugging in to the SearchEngine plugin. If you have to replace the entire Special:Search, then just do so as the old LuceneSearch plugin does. There's no reason to keep the mysql search around as well. - -- brion vibber (brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG+Rv2wRnhpk1wk44RAvqKAJ9qy/HLj+x85BbaDU3+CUuC3AaCYwCfSnDp S8QY+DmFMudYcfKFPGVubqw= =naug -----END PGP SIGNATURE-----

Joseph Hagerty

6:28 p.m.

Actually I have been experimenting with SphinxSearch and decided in the last week or so to use it on my sites, so thanks Paul!! Joe Hagerty -----Original Message----- From: wikitech-l-bounces(a)lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Paul Grinberg Sent: Monday, September 24, 2007 7:59 PM To: wikitech-l(a)lists.wikimedia.org Subject: [Wikitech-l] Request For Comments - Extension:SphinxSearch Hello, I just uploaded the first version of a new extension called SphinxSearch ( http://www.mediawiki.org/wiki/Extension:SphinxSearch ). This extension uses the Sphinx Search engine as the backend for full text searches of wiki content. See the MW page for full description of the extension. Since this is the first release of the extension, and since searching is such a fundamental requirement of a wiki-type webpage, I wanted to submit this extension for your scrutiny. I would like to get feedback on not only the extension's functionality, but also the clarity and completeness of the installation instructions. At the same time, I have a question to the community. Is there a good way to change the default behavior of the Search box on the left of every page (rendered with monobook that is)? In other words, my extension adds a new Special Page. I would like the search box to use my special page instead of the default Special:Search. The reason for this is because I do not want to completely get rid of the build-in search engine ... at least not yet. Many thanks in advance, paul. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

David Gerard

7:51 p.m.

On 25/09/2007, Joseph Hagerty <revjoe(a)revjoe.com> wrote:

...

Actually I have been experimenting with SphinxSearch and decided in the last week or so to use it on my sites, so thanks Paul!!

It'd be interesting to see a search comparison for a Wikipedia database between MW Lucene and Sphinx (and any other search backends) on a Wikipedia data set. Do we/can we collect search queries on en:wp? Say, 1 in 1000 searches, no details just the search text. That'll show what people are actually looking for. - d.

Andrew Garrett

27 Sep 27 Sep

5:50 a.m.

On 9/26/07, David Gerard <dgerard(a)gmail.com> wrote:

...

Do we/can we collect search queries on en:wp? Say, 1 in 1000 searches, no details just the search text. That'll show what people are actually looking for.

It would be ridiculously easy to do. Proof of concept is at http://vps.epstone.net/~andrew/wiki/index.php/MediaWiki:Common.js -- Andrew Garrett

David Gerard

11:38 a.m.

On 27/09/2007, Andrew Garrett <andrew(a)epstone.net> wrote:

...

On 9/26/07, David Gerard <dgerard(a)gmail.com> wrote:

...

> Do we/can we collect search queries on en:wp? Say, 1 in 1000 searches, > no details just the search text. That'll show what people are actually > looking for.

...

It would be ridiculously easy to do. Proof of concept is at http://vps.epstone.net/~andrew/wiki/index.php/MediaWiki:Common.js

Well yes :-) I can see no privacy implication, but I might have missed something. How many searches do we get a day? Is this useful data to those working on MediaWiki/Wikipedia search? (not to me, I must add - if I had concrete plans to apply it, I'd be pushing this hard) - d.

Matthew Britton

28 Sep 28 Sep

10:06 a.m.

David Gerard wrote:

...

On 27/09/2007, Andrew Garrett <andrew(a)epstone.net> wrote:

On 9/26/07, David Gerard <dgerard(a)gmail.com> wrote:

> Do we/can we collect search queries on en:wp? Say, 1 in 1000 searches, > no details just the search text. That'll show what people are actually > looking for.

It would be ridiculously easy to do. Proof of concept is at http://vps.epstone.net/~andrew/wiki/index.php/MediaWiki:Common.js

It would be useful data to those writing the encyclopedia; common search terms that are red links are likely to be good places to set up redirects or disambiguation pages. -Gurch

6078

days inactive

6081

days old

wikitech-l@lists.wikimedia.org

Manage subscription

22 comments

13 participants

tags (0)

participants (13)

Andrew Garrett
Brion Vibber
David Gerard
Jim Wilson
Joseph Hagerty
Magnus Manske
Matthew Britton
Maury Markowitz
Paul Grinberg
Platonides
Rob Church
Schneelocke
Simetrical