[Labs-l] [Ops] Slowing down puppet runs?

[Andrew Bogott]

On 10/31/14 11:54 AM, Andrew Otto wrote:
>> Seems like it would be reasonable to reduce the apt update frequency to once or twice per day, and maybe even to handle it by splayed cron job instead of as a puppet exec.
> This could be annoying for those times when you are applying a new puppetization in production that installs a package that has been newly added or upgraded in our apt.  But I suppose those are edge cases that we could work around by running a manual apt-get update if/when we need to do that.

There are cases where a single puppet run adds a new apt repo, does an 
apt-get update, and then installs a package; if the update is skipped in 
between then the wrong package can be installed (which then persists 
through subsequent runs).  So any change to the 'apt-get update' logic 
would have to be approached very cautiously.

But, in any case, I don't think this is a problem in need of a 
solution.  'apt-get update' is slow, but I can't think of any reason why 
it would be resource-intensive.  The thing that's expensive is compiling 
puppet catalogs on the puppetmaster -- delays in the client run 
shouldn't really make a difference to anyone who isn't actively 
debugging puppet and running things over and over by hand.

-A
>
>
>
> On Oct 31, 2014, at 12:35 PM, Jeff Green <jgreen at wikimedia.org> wrote:
>
>> On a related note, I've been looking at puppet agent run time in frack and I noticed that puppet executes apt-get update literally every run, and this accounts for a large chunk of the agent run time.
>>
>> Seems like it would be reasonable to reduce the apt update frequency to once or twice per day, and maybe even to handle it by splayed cron job instead of as a puppet exec.
>>
>> jg
>>
>> On Fri, 31 Oct 2014, Giuseppe Lavagetto wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Puppet is very resource intensive and runs every 20 minutes are a
>>> waste of resources even in prod, probably; it is surely taking a
>>> not-so-small toll on labs in term of cpu/iowait spikes.
>>>
>>> +1 to Antoine's proposal.
>>>
>>> Ciao
>>>
>>> Giuseppe
>>>
>>> On 31/10/14 15:17, Andrew Bogott wrote:
>>>> On 10/31/14 4:48 AM, Antoine Musso wrote:
>>>>> Hello,
>>>>> I noticed labs instance run puppet every 20 minutes just like
>>>>> Wikimedia production.  I thought on labs it could be run just
>>>>> once per hour which would slightly reduce load.
>>>> That wouldn't necessarily be bad, but can you explain what problem you're hoping to fix?  As far as I know, virt1000 (the labs puppetmaster) doesn't have any complaints about load.
>>>> -A
>>>> _______________________________________________ Labs-l mailing
>>>> list Labs-l at lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
>>> - -- Giuseppe Lavagetto
>>> Wikimedia Foundation - TechOps Team
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1
>>>
>>> iEYEARECAAYFAlRTtm0ACgkQTwZ0G8La7IBUwQCfXABRdPGawdP55OJewk6g1YcI
>>> ntMAoKH4M1AIuvPrwsB06lSPgmOJ9bpG
>>> =zuUx
>>> -----END PGP SIGNATURE-----
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>> _______________________________________________
>> Ops mailing list
>> Ops at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/ops
>
> _______________________________________________
> Ops mailing list
> Ops at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops