On Thu, Oct 30, 2014 at 10:18 PM, MZMcBride z@mzmcbride.com wrote:
James Forrester wrote:
On 30 October 2014 09:53, Chad innocentkiller@gmail.com wrote:
New hardware is in place and we've got plenty of breathing room to wrap up the migration to the new search engine.
Excellent news!
Indeed! Thanks to all who made this possible. An independent search engine is an incredibly important piece of infrastructure that's now getting a more appropriate level of attention and love. This is great and I'm excited to see what we'll be able to (continue to) build on top of it.
I lost track of the discussion about the ability to run regular expressions across wikitext. I see a large amount of opportunity in being able to search through wikitext in real-time. There are plenty of findable issues and errors in our articles and search is a key component in improving the situation.
I have OK news and bad news on that front unfortunately.
We used to have brute force regex searches and they were pretty much garbage. It was just too easy to write a search that would take minutes to complete. That would cause the queue of other regex searches to get backed up. And it'd timeout on the varnish side so the user, even if they had the patience to wait for 5 minutes to get a result, wouldn't get one.
The OK news is that we cut over to using trigram accelerated regex searches about a week ago. Its better. Almost usable but not quite. The worst case run time is now 30 seconds. If you search for something that is rare it'll probably come back in a few seconds. There are issues with consistency when your request runs really long as well ( https://bugzilla.wikimedia.org/show_bug.cgi?id=72128). Far from perfect but serviceable.
The bad news. We've had two Cirrus outages in the past week that I believe are caused by the accelerator so its disabled and we're back to brute force for now. The first outage was on Monday and we didn't have a clue what caused it. We added logging to learn for the next time and it didn't happen again until this morning. The extra logging failed (:shakes fist:) but we were able to implicate this code in the process.
The silver lining is that I'll be working on it again and might be able to get some speed improvements while were there.
I'm not sure what the outage says about the schedule. I'll have to do some thinking about that. In the mean time it does say that we should keep the old search there as a backup. We've been able to fall back to it during the outages to minimize the suffering.
Nik