[Labs-l] Custom nagios checks

[Damian Zaremba]

On 03/02/14 18:32, Petr Bena wrote:
> Hi,
>
> I think it's a time to finally make it possible for users to create
> own check for nagios (icinga).
>
> I will try to document how current icinga is setup on
> https://wikitech.wikimedia.org/wiki/Icinga/Labs
>
> Currently there is a nagiosbuilder which is a python script made by
> Damian which query the ldap and build nagios cfg files based on that.
>
> My idea is to create configuration files / templates for this
> nagiosbuilder so that it would apply different options for certain
> hosts based on this configuration. Users would just
>
>   * create own check, place it somewhere on the server which they want
> to monitor the service at and insert it to nrpe
> (/etc/nagios/nrpe.d/yourservice.cfg)
> * use some interface (to be discussed) to insert this check for specific host
>
> and nagiosbuilder would
>
> * query that interface in order to generate configuration files
> * based on this config would set up custom services for these hosts
>
> For us (developers) it's most easy to use gerrit as this interface, so
> that people would directly update these configuration files used by
> nagiosbuilder, however that pretty much suck, so I think it would be
> better to create a new interface into labsconsole, so that people can
> define their nagios checks directly as a property of each node.
>
> That of course would require more coding and ops assistance but I
> think it's doable. Some opinions?
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
The initial idea behind the templating thing was to try and stick to 
roughly the same as prod ie all service checks are based on roles 
assigned to instances.

Any custom checks should be linked in the same way - to a puppet role 
and then apply the role. To get the check in the config you'll have to 
add a case for the role (this is duplicate effort from prod, but 
hopefully should be maintainable).

I'd rather not see a custom interface for this as it will split away 
from production a lot more and cause hell down the line.

- Damian