Thought I'd mention something I worked on recently - I have a cron on deployment-cumin that runs puppet across everything (that openstack says is running) and emails me with a list of hosts with any problems. It has a little config allowing associating a host with a task. deployment-prep is looking better than I thought.

(deploy-01 got broken in the security updates and I'm planning to look into cache-text04 later - think this was a timeout of some sort, likely related to my certificate work there)

---------- Forwarded message ----------
From: <krenair@beta.wmflabs.org>
Date: 15 June 2018 at 19:05
Subject: Deployment-prep Puppet error hosts report
To: krenair@gmail.com

Hostname	Task?
deployment-cache-text04.deployment-prep.eqiad.wmflabs	None
deployment-deploy-01.deployment-prep.eqiad.wmflabs	T192561

Hosts configured with tasks but are not listing as broken anymore:

Hostname	Task
deployment-mx.deployment-prep.eqiad.wmflabs	T184244