<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">
<div class="" id="magicdomid919"><span class="">The Labs team has
spent a lot of time discussing how to avoid future capacity
crunches. This will be a long term consideration involving
coordination with project admins.</span></div>
<div class="" id="magicdomid697"><br>
</div>
<div class="" id="magicdomid736"><span class="">Outline of our
progress:</span></div>
<div class="" id="magicdomid738"><br>
</div>
<div class="" id="magicdomid935"><span class="">- Self-service
instance creation is again enabled for Labs in Horizon (</span><span
class=""><a href="https://horizon.wikimedia.org%29">https://horizon.wikimedia.org)</a></span></div>
<div class="" id="magicdomid741"><br>
</div>
<div class="" id="magicdomid748"><span class="">- We've reduced
the permitted overcommit ratios for RAM and disk space
(T140119 and T140122)<br>
</span></div>
<div class="" id="magicdomid8"><br>
</div>
<div class="" id="magicdomid931"><span class="">- We've reduced
default quotas which affects new and some existing projects.
We believe this will have two advantages: 1) When growth
happens we'll know about it, and 2) There will be an
opportunity to consult about what growth is necessary and how
best to take advantage of capacity. </span></div>
<div class="" id="magicdomid10"><br>
</div>
<div class="" id="magicdomid816"><span class="">- We are
transitioning to a process for quota increase requests using
Phabricator (see T140904)</span></div>
<div class="" id="magicdomid818"><br>
</div>
<div class="" id="magicdomid928"><span class="">- Even with our
new nodes we are still tight on space without diligent clean
up. </span><span class=""><a
href="https://wikitech.wikimedia.org/wiki/Purge_2016">https://wikitech.wikimedia.org/wiki/Purge_2016</a></span></div>
<div class="" id="magicdomid17"><br>
</div>
<div class="" id="magicdomid13"><span class="">Let me know if you
have any questions or concerns!</span></div>
<div class="" id="magicdomid14"><br>
</div>
<div class="" id="magicdomid15"><span class="">-Andrew & the
Labs team</span></div>
<br>
<br>
On 7/5/16 5:05 PM, Andrew Bogott wrote:<br>
</div>
<blockquote
cite="mid:ee2f700f-f07a-40a8-3872-f38fc07e9eb0@gmail.com"
type="cite">We're running into serious capacity issues this week,
so I have temporarily disabled the creation of new labs
instances. We have new hardware sitting in the datacenter
already, so at worst we'll be back to normal sometime next week.
<br>
<br>
Sorry for the inconvenience. And, as usual, if you have instances
you aren't using, please delete them!
<br>
<br>
-Andrew
<br>
<br>
<br>
</blockquote>
<p><br>
</p>
</body>
</html>