[Labs-l] Custom nagios checks

[Petr Bena]

Hi,

I think it's a time to finally make it possible for users to create
own check for nagios (icinga).

I will try to document how current icinga is setup on
https://wikitech.wikimedia.org/wiki/Icinga/Labs

Currently there is a nagiosbuilder which is a python script made by
Damian which query the ldap and build nagios cfg files based on that.

My idea is to create configuration files / templates for this
nagiosbuilder so that it would apply different options for certain
hosts based on this configuration. Users would just

 * create own check, place it somewhere on the server which they want
to monitor the service at and insert it to nrpe
(/etc/nagios/nrpe.d/yourservice.cfg)
* use some interface (to be discussed) to insert this check for specific host

and nagiosbuilder would

* query that interface in order to generate configuration files
* based on this config would set up custom services for these hosts

For us (developers) it's most easy to use gerrit as this interface, so
that people would directly update these configuration files used by
nagiosbuilder, however that pretty much suck, so I think it would be
better to create a new interface into labsconsole, so that people can
define their nagios checks directly as a property of each node.

That of course would require more coding and ops assistance but I
think it's doable. Some opinions?