I'm trying to set up a more regular search index update for the Wikimedia sites.
To summarize how it's to work:
A process on maurus (the search build master) runs through the list of all
wikis, dumping their text and piping it to the search index builder program.
As each wiki completes, the newly built index is moved from the build directory
into the complete directory.
The lucene search servers currently restart themselves hourly as a precaution
against memory leaks; additionally before restart they will now do an rsync to
copy over any complete new indexes from the master. [There may be some
refinements to make on this, such as keeping 'live' and 'update' copies
and
swapping them out during the restart.]
This should keep search index updates happening within a day or two, rather than
the extremely long and irregular schedule of before.
Currently the build process is running on maurus since last night; it's
currently about 1/3 through enwiki and doesn't appear to have spewed any errors.
I'll check in on it again this evening and if things look ok I'll set up the
synchronization process.
-- brion vibber (brion @
pobox.com)