[Labs-l] Partial DNS outage 14/11

[Ryan Lane]

Last night I noticed that LDAP was somewhat slow and determined that
equality and substring indexes were needed for:

memberUid
associatedDomain
SOARecord

A substring index was added for member.

Unfortunately while adding the indexes I made an error on virt0 (labs-ns0)
for the associatedDomain index, creating an index for associateddomain
instead. This caused equality and substring searches for associatedDomain
records to fail on virt0 (labs-ns0), but succeed on virt1000 (labs-ns1)
thereby returning NXDOMAIN for some records on virt0 but correct results
from virt1000. I just deleted the index, created it correctly, and rebuilt
it. DNS searches are now working properly again.

- Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20131114/d443ed63/attachment.html>