[Labs-l] [Ops] Slowing down puppet runs?

[Jeff Green]

On Fri, 31 Oct 2014, Andrew Otto wrote:

>> Seems like it would be reasonable to reduce the apt update frequency to 
>> once or twice per day, and maybe even to handle it by splayed cron job 
>> instead of as a puppet exec.
>
> This could be annoying for those times when you are applying a new 
> puppetization in production that installs a package that has been newly 
> added or upgraded in our apt.  But I suppose those are edge cases that 
> we could work around by running a manual apt-get update if/when we need 
> to do that.

I hadn't thought of that--it's not really an issue in fundraising.

The change I would make to puppet would be to add "refreshonly => true" to 
the apt-update exec, and then make sure apt config changes notify that 
exec so it runs just like it does now after an apt config change.

Then I would have puppet add a cron job using "minute => fqdn_rand(60)" to 
spread the updates over an hour. Timing could be overridden with a class a 
parameter.

It should be feasible to force a one-time sitewide run through puppet too. 
I can think of a couple ways of doing so off the top of my head, but 
they're too awful to put into words. It would be slick to make 
puppetmaster poll our repos and do it automatically when the metadata 
changes.

jg

> On Oct 31, 2014, at 12:35 PM, Jeff Green <jgreen at wikimedia.org> wrote:
>
>> On a related note, I've been looking at puppet agent run time in frack 
>> and I noticed that puppet executes apt-get update literally every run, 
>> and this accounts for a large chunk of the agent run time.
>>
>> Seems like it would be reasonable to reduce the apt update frequency to 
>> once or twice per day, and maybe even to handle it by splayed cron job 
>> instead of as a puppet exec.
>>
>> jg
>>
>> On Fri, 31 Oct 2014, Giuseppe Lavagetto wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Puppet is very resource intensive and runs every 20 minutes are a
>>> waste of resources even in prod, probably; it is surely taking a
>>> not-so-small toll on labs in term of cpu/iowait spikes.
>>>
>>> +1 to Antoine's proposal.
>>>
>>> Ciao
>>>
>>> Giuseppe
>>>
>>> On 31/10/14 15:17, Andrew Bogott wrote:
>>>> On 10/31/14 4:48 AM, Antoine Musso wrote:
>>>>> Hello,
>>>>> I noticed labs instance run puppet every 20 minutes just like
>>>>> Wikimedia production.  I thought on labs it could be run just
>>>>> once per hour which would slightly reduce load.
>>>> That wouldn't necessarily be bad, but can you explain what problem you're hoping to fix?  As far as I know, virt1000 (the labs puppetmaster) doesn't have any complaints about load.
>>>> -A
>>>> _______________________________________________ Labs-l mailing
>>>> list Labs-l at lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>> - -- Giuseppe Lavagetto
>>> Wikimedia Foundation - TechOps Team
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1
>>>
>>> iEYEARECAAYFAlRTtm0ACgkQTwZ0G8La7IBUwQCfXABRdPGawdP55OJewk6g1YcI
>>> ntMAoKH4M1AIuvPrwsB06lSPgmOJ9bpG
>>> =zuUx
>>> -----END PGP SIGNATURE-----
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>> _______________________________________________
>> Ops mailing list
>> Ops at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/ops
>