Stripping Question Marks From Wiki Searches
Do you ask questions on Wikipedia? Would you like better results?

Summary: Because the large majority of question marks are used to ask questions by users unfamiliar with bash-style wildcards, the default behavior for CirrusSearch will now be to ignore question marks (replacing them with a space). Escaping them with a backslash (\?) will preserve their wildcard meaning. Regular expressions in insource: will not be affected and should not be escaped. This option can be modified on a per-wiki basis if needed (see $wgCirrusSearchStripQuestionMarks).


When people ask how old is tom cruise? on Wikipedia they almost certainly don’t expect the question mark in cruise? to match an additional letter. They aren’t looking for the words cruised, cruiser,  or cruises—but that’s what they get, and it keeps them from finding the information they are really after.

Search on Wikipedia (and other Wikimedia projects) includes a lot of features that most users don’t know about. Most require special keywords, and some even require specialized knowledge, such as familiarity with regular expressions. It’s pretty difficult to invoke these special features by accident.

But search also supports two particular bash-style wildcards without any special syntax: * will match any number of characters, and ? will match exactly one. Asterisks do come up from time to time, but people use question marks all the time—because they like to ask questions!

A recent review of query-string features called out quotes and question marks as the two largest-impact predictors of unsuccessful queries on Wikipedia. In a follow-up survey of queries with question marks in six of the top ten Wikipedias (by search volume), most question marks are being used to ask questions (the other four of the top 10 were not reviewed). In all ten of the top ten, stripping final question marks dramatically decreased the number of ?-final queries that got either no results, or very few results (i.e., less than 3). The improvement was around 10-45% for ?-final queries, depending on the wiki. The overall impact is much more modest (less than 0.5%) because queries with question marks are not terribly common.

As a result of this analysis, we’ve implemented a change to search which will by default replace question marks with spaces (to preserve the word boundaries they intend in queries like how?why?). This setting can be changed on a per-wiki basis, and other options include (i) only stripping question marks at a clear word boundary (such as before a space), (ii) only stripping question marks at the end of the query, and (iii) leaving the question marks alone.

For the rarer users who do use question marks as a one-letter wildcard, when question mark stripping is enabled, question marks can be escaped with a backslash (e.g., wiki\?edia) to preserve their original wildcard meaning. Power searchers who use insource: won’t need to do anything special; queries withinsource: will not be modified.