[Labs-l] displaying puppet status for each host

[Andrew Bogott]

On 1/31/14 3:58 AM, Tim Landscheidt wrote:
> Antoine Musso <hashar+wmf at free.fr> wrote:
>
>>> I assume Ryan meant
>>> https://wikitech.wikimedia.org/wiki/Special:NovaInstance
>>> which has a column "Puppet status" ("ok"/"failed"/"stale").
>> That is nice!
>> Would it be possible to get notifications / emails whenever the status
>> of puppet change on an instance?  Or maybe a daily report would be enough.
> I believe Puppet status was monitored by
> http://icinga.wmflabs.org/ along with basic stuff like disk
> usage & Co., but it has been unavailable for some time now.
<snip>

Sorry I'm late to this conversation.  Indeed we use icinga on production 
to monitor puppet failures, and this used to work on labs.  The 
implementation of this seems generally unpopular, so when I added the 
'puppet status' filed to wikitech I routed around icinga and used a 
puppet reporting mechanism that integrates with OpenStack.  There are a 
few incremental changes, but the basic framework is in 
https://gerrit.wikimedia.org/r/#/c/100221/.  The report and callback 
system is quite simple!  So, Petr, if you want a custom/limited solution 
you might look into just writing a custom reporter.

Regarding notifications or reporting... there are a few ways we could 
approach this.  Probably the best thing is to revive icinga reporting, 
possibly be extending labsstatus.rb.  For now, though, I'd advise you to 
just have a look at wikitech if you're curious.

-Andrew