[Labs-l] Recent performance issues and fixes

[Ryan Lane]

We've had a number of LDAP related performance issues lately. While
reviewing the logs I noticed a very large number of queries for groups with
relatively high gids from every instance. I also noticed than when running
commands like: "id laner" the lookup of groups would stall shortly halfway
through my group listing (I have about 80 groups).

This can often be a sign that the search limit is being reached. We had
increased the search limits a while back, but I've noticed recently that
we've started reaching them again. It's a bad idea to continue to raise the
search limits.

Thankfully, nslcd, the daemon we use for nss lookups, has a setting for
paged lookups. After setting a pagesize, lookups were quite a bit faster,
especially on subsequent requests.

I also added a setting which should reduce the number of lookups for some
instances that have added and removed system users: nss_min_uid 499. This
setting says: "If the user's uid is lower than 499, don't do an LDAP
lookup". Unfortunately, this setting is only available for our precise
instances.

nss_min_uid was temporarily pushed to lucid instances as well, so some
instances may have had failed LDAP lookups for a couple of minutes.

- Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20130121/6ee6e0d9/attachment.html>