<div dir="ltr">I don't believe clush is functioning on tools-puppetmaster-02 at the moment fyi.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 28, 2017 at 10:01 AM, Andrew Bogott <span dir="ltr"><<a href="mailto:abogott@wikimedia.org" target="_blank">abogott@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 6/27/17 10:46 PM, Andrew Bogott wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
tl;dr: Tools has a new puppetmaster, tools-puppetmaster-01. I haven't migrated clush over to that box but plan to tomorrow.<br>
</blockquote></span>
Update: We can't really build new clush masters right now due to the dead Mirantis openstack repo. For now, tools-puppetmaster-02 will remain as the clush master.<br>
<br>
I've created a task about this package repo issue... we could potentially work around things by backporting to python 2 or (possibly) rebuilding on Stretch.<br>
<br>
<a href="https://phabricator.wikimedia.org/T169099" rel="noreferrer" target="_blank">https://phabricator.wikimedia.<wbr>org/T169099</a><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
-<br>
<br>
<br>
In the beginning there was<br>
<br>
<a href="https://gerrit.wikimedia.org/r/#/c/361675/" rel="noreferrer" target="_blank">https://gerrit.wikimedia.org/r<wbr>/#/c/361675/</a><br>
<br>
Which was a no-op on the production puppetmasters but not a no-op on labs puppetmasters due to my over-pruning. So, I swiftly reverted with<br>
<br>
<a href="https://gerrit.wikimedia.org/r/#/c/361710/" rel="noreferrer" target="_blank">https://gerrit.wikimedia.org/r<wbr>/#/c/361710/</a><br>
<br>
Which would've fixed things. Except, by that time, the apache config for all self-hosted puppetmasters was broken. And, most of those puppetmasters are their /own/ puppetmasters which meant they couldn't fix themselves... So, I went through the list of involved hosts (via watroles) and pasted in the missing bits in order to kickstart things and everything should be fine now.<br>
<br>
except... when I fixed the tools puppetmaster it started saying<br>
<br>
"Warning: Could not intern from text/plain: nested asn1 error"<br>
<br>
I do not know what that is, and Google doesn't know what it is either. Google provided a slight hint, though, as someone at puppet labs responded to a bug report about that message with no explanation but a terse 'this is fixed in the next version'.<br>
<br>
So... since labs-puppetmaster-02 was running a 3.7-series package and we're using 3.8 packages elsewhere, I did an 'apt-get install puppetmaster' to move to 3.8. Except, there was some latent unpuppetized pinning in the apt config for that box (a different, also long story) which meant that instead of upgrading to 3.8 it upgraded to 4.something and after that I was well and truly doomed. No amount of cert regenerating or rebooting or de- and re-installing puppet packages would make that 'could not intern' error go away or anything run cleanly.<br>
<br>
So, I built a new puppetmaster for tools, tools-puppet-master-01, copied the custom private patches on -02 over to -01, clobbered all the certs on all the existing tools instances and switched everything over to the new puppetmaster. The dance of getting certs cleaned and then initial puppet runs going before an old cron'd puppet run starts up and creates a new but broken cert turned out to be intricate and frustrating and took hours and resulted in the many, many shinken emails that you all received.<br>
<br>
This was good practice for me since I'm going to have to migrate the rest of labs to a new puppetmaster soon anyway... but I apologize for all the racket.<br>
<br>
-Andrew<br>
<br>
<br>
</blockquote>
<br>
<br>
______________________________<wbr>_________________<br>
Labs-admin mailing list<br>
<a href="mailto:Labs-admin@lists.wikimedia.org" target="_blank">Labs-admin@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-admin" rel="noreferrer" target="_blank">https://lists.wikimedia.org/ma<wbr>ilman/listinfo/labs-admin</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Chase Pettet</div></div><div>chasemp on <a href="https://phabricator.wikimedia.org/p/chasemp/" target="_blank">phabricator</a> and IRC<br></div></div></div></div></div></div></div></div>
</div>