[Labs-l] [Labs-announce] [action required] We are removing inactive projects.

[Bryan Davis]

On Fri, Jul 8, 2016 at 3:24 PM, Chase Pettet <cpettet at wikimedia.org> wrote:
> In case some of of the readers of this list are unaware, new instance
> creation is currently suspended due to capacity.  We have some new resources
> we are working diligently to use for expanding but they are still a limited
> solution.

The fact that the servers we have operational now are full is actually
great because it means that people are using Labs more than ever
before. It's not great that the timing of several things conspired to
keep additional resources from being deployed before we hit a critical
point of usage.

I think that historically the collective "we" have done a poor job of
recognizing the cost and savings associated with use of the Labs
environment. On the cost side, thus far budget for hardware in Labs
has been virtually unlimited. By that I mean that as far as I know any
time more hardware has been asked for the budget has been found to
supply it. Human resources however are not so plentiful. Full time
staffing has never been higher than 3 engineers and often only 2 in
practice. This is just a reality of the WMF budget process.

On the savings side, some really really rough napkin math using
calculator.s3.amazonaws.com shows that just the virtual machines in
active use by Labs projects would cost between $45k and $50k USD per
month. On top of that a real AWS user would have to pay for data
transfer and other networking services and probably a few support
hours. That $50k USD per month also doesn't include any cost for
things like actively managed real-time replicas of the Wikimedia
production databases or any of the other shared services available in
Labs. All told I would say that the Wikimedia movement is getting more
than fair return on the moneys invested in the Labs hardware and
staffing.

I personally think that line items for Labs and Tool Labs are some of
the best spent dollars in the Wikimedia Foundation budget due to the
incredibly diverse and productive projects that they enable volunteers
to create. But I get a bit crabby when people act as though AGF only
applies on wiki. When Chase, Andrew, Yuvi, or anyone else spending
their time and energy to keep Labs running asks for a change of the
status quo they are asking with good intent.

When you spin up a new VM or launch a grid job in Tool Labs there is
an opportunity cost to the Wikimedia movement that should be
respected. Every compute hour or gigabyte of storage your personal
project consumes is one that is not available for someone else.
Shutting down unused systems and cleaning up projects that have run
their course is a reasonable thing to ask.

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855