<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>FYI, here is a thing that reading/ores people are daydreaming
about. I've made it clear that bare-metal-in-labs is a
nonstarter; having special hardware with special VM types is not
necessarily all that much work. I bet a labvirt with a GPU will
be super expensive though, and I don't know a damn thing about how
GPU resource-contention would be handled.<br>
</p>
<p>-A</p>
<br>
<div class="moz-forward-container">-------- Forwarded Message
--------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Delivered-To:
</th>
<td><a class="moz-txt-link-abbreviated" href="mailto:andrewbogott@gmail.com">andrewbogott@gmail.com</a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>by 10.55.130.1 with SMTP id e1csp77787qkd; Wed, 22 Feb
2017 08:57:16 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Received:
</th>
<td>by 10.237.36.116 with SMTP id
s49mr18754207qtc.128.1487782636280; Wed, 22 Feb 2017
08:57:16 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Return-Path:
</th>
<td><a class="moz-txt-link-rfc2396E" href="mailto:abogott+caf_=andrewbogott=gmail.com@wikimedia.org"><abogott+caf_=andrewbogott=gmail.com@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>from mail-qk0-f173.google.com (mail-qk0-f173.google.com.
[209.85.220.173]) by mx.google.com with ESMTPS id
q53si660238qtf.337.2017.02.22.08.57.16 for
<a class="moz-txt-link-rfc2396E" href="mailto:andrewbogott@gmail.com"><andrewbogott@gmail.com></a> (version=TLS1_2
cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22
Feb 2017 08:57:16 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received-SPF:
</th>
<td>pass (google.com: domain of
<a class="moz-txt-link-abbreviated" href="mailto:abogott+caf_=andrewbogott=gmail.com@wikimedia.org">abogott+caf_=andrewbogott=gmail.com@wikimedia.org</a>
designates 209.85.220.173 as permitted sender)
client-ip=209.85.220.173;</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Authentication-Results:
</th>
<td>mx.google.com; dkim=pass <a class="moz-txt-link-abbreviated" href="mailto:header.i=@wikimedia.org">header.i=@wikimedia.org</a>;
spf=pass (google.com: domain of
<a class="moz-txt-link-abbreviated" href="mailto:abogott+caf_=andrewbogott=gmail.com@wikimedia.org">abogott+caf_=andrewbogott=gmail.com@wikimedia.org</a>
designates 209.85.220.173 as permitted sender)
<a class="moz-txt-link-abbreviated" href="mailto:smtp.mailfrom=abogott+caf_=andrewbogott=gmail.com@wikimedia.org">smtp.mailfrom=abogott+caf_=andrewbogott=gmail.com@wikimedia.org</a>;
dmarc=pass (p=NONE sp=NONE dis=NONE)
header.from=wikimedia.org</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>by mail-qk0-f173.google.com with SMTP id n127so300qkf.0
for <a class="moz-txt-link-rfc2396E" href="mailto:andrewbogott@gmail.com"><andrewbogott@gmail.com></a>; Wed, 22 Feb 2017
08:57:16 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Google-DKIM-Signature:
</th>
<td>v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net;
s=20161025;
h=x-gm-message-state:delivered-to:dkim-signature:mime-version
:in-reply-to:references:from:date:message-id:subject:to:cc;
bh=geU9E0JuTBp45fwea4nD9VRvS20wnOpfRwfENW1LjhU=;
b=JCBzeI8wEoxwlF8L9N02zAe4B6yM5bhHph//6v2nv99TlzVWmYMh+M5YysH03KFmox
Pv6DRB9RYw7g+3w6On4goiSqbRlOanasU9gEBuOqLaivY/4ictoMjr0mfCZynFxGwwcO
nfgiGna0f9LVgrElZZDd01HweqTJ0uBW7tPPJDxt74YLf7I+tysJReVuvW/e4GzGtMHG
B801Icgxf5WNdjQYUImCtXXHQFxlDcW+PydqDPQMH5Tyg8ssKWUQwFqCKe0PVdtQsCY9
HLYgur5owYC+yZr2D1CNR3iUMH2jcHM0IdsuLmiKL6kSCy+x6oslXO4kJf3Hj3O7sFEy
3ZFw==</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Gm-Message-State:
</th>
<td>AMke39mzi2oJ/S99gBkx9KxBafTj2nbLxmj5JK4xkqZ1ij7Mt9aq5oAT2u0MCGpmwqclQKf7e+yX+X2zHORrO8Cpp0HTeDQVCA==</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Received:
</th>
<td>by 10.55.94.6 with SMTP id
s6mr34227257qkb.166.1487782635865; Wed, 22 Feb 2017
08:57:15 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Forwarded-To:
</th>
<td><a class="moz-txt-link-abbreviated" href="mailto:andrewbogott@gmail.com">andrewbogott@gmail.com</a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Forwarded-For:
</th>
<td><a class="moz-txt-link-abbreviated" href="mailto:abogott@wikimedia.org">abogott@wikimedia.org</a> <a class="moz-txt-link-abbreviated" href="mailto:andrewbogott@gmail.com">andrewbogott@gmail.com</a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Delivered-To:
</th>
<td><a class="moz-txt-link-abbreviated" href="mailto:abogott@wikimedia.org">abogott@wikimedia.org</a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>by 10.140.99.34 with SMTP id p31csp954099qge; Wed, 22
Feb 2017 08:57:14 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Received:
</th>
<td>by 10.237.50.229 with SMTP id
z92mr30222596qtd.182.1487782634693; Wed, 22 Feb 2017
08:57:14 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Return-Path:
</th>
<td><a class="moz-txt-link-rfc2396E" href="mailto:abaso@wikimedia.org"><abaso@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>from mx1001.wikimedia.org (mx1001.wikimedia.org.
[208.80.154.76]) by mx.google.com with ESMTPS id
52si1315621qtw.95.2017.02.22.08.57.14 for
<a class="moz-txt-link-rfc2396E" href="mailto:abogott@wikimedia.org"><abogott@wikimedia.org></a> (version=TLS1_2
cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22
Feb 2017 08:57:14 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received-SPF:
</th>
<td>pass (google.com: domain of <a class="moz-txt-link-abbreviated" href="mailto:abaso@wikimedia.org">abaso@wikimedia.org</a>
designates 2a00:1450:400c:c0c::22a as permitted sender)
client-ip=2a00:1450:400c:c0c::22a;</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>from mail-wr0-x22a.google.com
([2a00:1450:400c:c0c::22a]:33057) by mx1001.wikimedia.org
with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.84_2) (envelope-from <a class="moz-txt-link-rfc2396E" href="mailto:abaso@wikimedia.org"><abaso@wikimedia.org></a>)
id 1cgaE4-0000fA-8M for <a class="moz-txt-link-abbreviated" href="mailto:abogott@wikimedia.org">abogott@wikimedia.org</a>; Wed, 22 Feb
2017 16:57:14 +0000</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>by mail-wr0-x22a.google.com with SMTP id
97so6296811wrb.0 for <a class="moz-txt-link-rfc2396E" href="mailto:abogott@wikimedia.org"><abogott@wikimedia.org></a>; Wed,
22 Feb 2017 08:57:12 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">DKIM-Signature:
</th>
<td>v=1; a=rsa-sha256; c=relaxed/relaxed; d=wikimedia.org;
s=google;
h=mime-version:in-reply-to:references:from:date:message-id:subject:to
:cc; bh=geU9E0JuTBp45fwea4nD9VRvS20wnOpfRwfENW1LjhU=;
b=WFFw+gGuZwu1fwbD30zuj8lNJi5A1g7aoIe2iVvXiXk2ePZe0PpxBVjGtiO52g9DmK
ivzQn2KiDrcgF1X3pHaYSUiINhiR5N/IFRcmPznm0HESdJyycHRUySCxgRVHM6Hlt2a7
YflNt6jxThtJE3Hygnodo53N7EPKxHWtj4NV0=</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">X-Received:
</th>
<td>by 10.223.172.66 with SMTP id
v60mr25236742wrc.77.1487782631305; Wed, 22 Feb 2017
08:57:11 -0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">MIME-Version:
</th>
<td>1.0</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Received:
</th>
<td>by 10.80.148.203 with HTTP; Wed, 22 Feb 2017 08:56:30
-0800 (PST)</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">In-Reply-To:
</th>
<td><a class="moz-txt-link-rfc2396E" href="mailto:3a1992ee-44b4-8f8c-f946-15b8f6bad81c@wikimedia.org"><3a1992ee-44b4-8f8c-f946-15b8f6bad81c@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">References:
</th>
<td><a class="moz-txt-link-rfc2396E" href="mailto:CAB74=NqwwtYKkgxEZWex5HxFso1xBZ+Y4swRzJzRzhwKGXrnVw@mail.gmail.com"><CAB74=NqwwtYKkgxEZWex5HxFso1xBZ+Y4swRzJzRzhwKGXrnVw@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CALuiOYYEp34ZTR76x2cAEkmpd03njx+SZUEAbWP5cmqkp4ERaQ@mail.gmail.com"><CALuiOYYEp34ZTR76x2cAEkmpd03njx+SZUEAbWP5cmqkp4ERaQ@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAB74=NpEkj9r9SdGfsn=VFz44DxRz+Ch5gKrrZJeoEhdDBz5xw@mail.gmail.com"><CAB74=NpEkj9r9SdGfsn=VFz44DxRz+Ch5gKrrZJeoEhdDBz5xw@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAO8=cz3j_A018QCU9iry+UjWEFQZ5AcV4AD942RhEiFkXvqS=Q@mail.gmail.com"><CAO8=cz3j_A018QCU9iry+UjWEFQZ5AcV4AD942RhEiFkXvqS=Q@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAKP=3Wz0R5DpsA8-b+PmJHuUNYggisyN5_4yUStN6cga7ftApA@mail.gmail.com"><CAKP=3Wz0R5DpsA8-b+PmJHuUNYggisyN5_4yUStN6cga7ftApA@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAH8ZkEggG2WvTeGrZVVyMVKyThe7rZe+TqQsWfOkboW5HtsY7w@mail.gmail.com"><CAH8ZkEggG2WvTeGrZVVyMVKyThe7rZe+TqQsWfOkboW5HtsY7w@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAKP=3WwvZ3wfL1XH1aeszPN5Ja3sua_brFifzzSS5SHD0UzpEA@mail.gmail.com"><CAKP=3WwvZ3wfL1XH1aeszPN5Ja3sua_brFifzzSS5SHD0UzpEA@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAB74=NoUBJG_f31nb0ACzKyv5JZhnnwqK7N0LVoVeHssozTaEg@mail.gmail.com"><CAB74=NoUBJG_f31nb0ACzKyv5JZhnnwqK7N0LVoVeHssozTaEg@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAB74=NrrdD46n8nhY3rBmXNNkwkWxyiBiAb9xYNb8XzkUuHBOw@mail.gmail.com"><CAB74=NrrdD46n8nhY3rBmXNNkwkWxyiBiAb9xYNb8XzkUuHBOw@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAKP=3WxiqnJZjc=2hPcCe5U7cdbxBMqkdzr91SAOdO7+ns9m0w@mail.gmail.com"><CAKP=3WxiqnJZjc=2hPcCe5U7cdbxBMqkdzr91SAOdO7+ns9m0w@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAB74=Nq09YL-PY_nbq-oir_7dReYjzLfVz-FgqvPYmD5U=Nqow@mail.gmail.com"><CAB74=Nq09YL-PY_nbq-oir_7dReYjzLfVz-FgqvPYmD5U=Nqow@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAKP=3Wxy4dncDCQJo4bgGwAta-4+Y_xJ0iQHeC1GuPfH6y=RpQ@mail.gmail.com"><CAKP=3Wxy4dncDCQJo4bgGwAta-4+Y_xJ0iQHeC1GuPfH6y=RpQ@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:40905b9f-e1e6-1ab2-5d9b-baa5a1dbd9ac@wikimedia.org"><40905b9f-e1e6-1ab2-5d9b-baa5a1dbd9ac@wikimedia.org></a>
<a class="moz-txt-link-rfc2396E" href="mailto:CAB74=Npe3UwzPZv+ditB1bmNASvNuXgK00ZanM65E7-XySX_pg@mail.gmail.com"><CAB74=Npe3UwzPZv+ditB1bmNASvNuXgK00ZanM65E7-XySX_pg@mail.gmail.com></a>
<a class="moz-txt-link-rfc2396E" href="mailto:3a1992ee-44b4-8f8c-f946-15b8f6bad81c@wikimedia.org"><3a1992ee-44b4-8f8c-f946-15b8f6bad81c@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">From: </th>
<td>Adam Baso <a class="moz-txt-link-rfc2396E" href="mailto:abaso@wikimedia.org"><abaso@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Date: </th>
<td>Wed, 22 Feb 2017 10:56:30 -0600</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Message-ID:
</th>
<td><a class="moz-txt-link-rfc2396E" href="mailto:CAB74=Nq+aSspOt2bp1Hno58xjcERf3P8S_c_HPeaBgFWiY_NFw@mail.gmail.com"><CAB74=Nq+aSspOt2bp1Hno58xjcERf3P8S_c_HPeaBgFWiY_NFw@mail.gmail.com></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Subject:
</th>
<td>Re: Servers with GPUs</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">To: </th>
<td>Andrew Bogott <a class="moz-txt-link-rfc2396E" href="mailto:abogott@wikimedia.org"><abogott@wikimedia.org></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Cc: </th>
<td>Aaron Halfaker <a class="moz-txt-link-rfc2396E" href="mailto:ahalfaker@wikimedia.org"><ahalfaker@wikimedia.org></a>, Dario
Taraborelli <a class="moz-txt-link-rfc2396E" href="mailto:dtaraborelli@wikimedia.org"><dtaraborelli@wikimedia.org></a>, Ellery
Wulczyn <a class="moz-txt-link-rfc2396E" href="mailto:ewulczyn@wikimedia.org"><ewulczyn@wikimedia.org></a>, Andrew Otto
<a class="moz-txt-link-rfc2396E" href="mailto:otto@wikimedia.org"><otto@wikimedia.org></a>, Corey Floyd
<a class="moz-txt-link-rfc2396E" href="mailto:cfloyd@wikimedia.org"><cfloyd@wikimedia.org></a>, Andrew Otto
<a class="moz-txt-link-rfc2396E" href="mailto:acotto@gmail.com"><acotto@gmail.com></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Content-Type:
</th>
<td>multipart/alternative;
boundary=f403045cf1523e23fe05492163db</td>
</tr>
</tbody>
</table>
<br>
<br>
<div dir="ltr">I feel comfortable saying we want one of these
options. I'm trying to gchat Ryan Lane to see if has insight on
the Nova support. I just realized when I checked action=history
he was one of the editors of <a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/HeterogeneousGpuAcceleratorSupport"
target="_blank" style="font-size:12.8px">HeterogeneousGpuAcceleratorSup<wbr>port</a>!
<div><br>
</div>
<div>-Adam<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 21, 2017 at 2:16 PM,
Andrew Bogott <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:abogott@wikimedia.org" target="_blank">abogott@wikimedia.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span>
<div
class="gmail-m_-5394071749873894282m_245415591453473483moz-cite-prefix">On
2/21/17 1:36 PM, Adam Baso wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I think either that, or if it's
easier, OpenStack-provisioned physical servers.
Do you think the latter is doable? </div>
</blockquote>
</span> It's possible, although the last time we
visited that issue we swiftly determined that even
though people were asking for it no one actually
wanted it. The conclusion from that process is at <a
moz-do-not-send="true"
class="gmail-m_-5394071749873894282m_245415591453473483moz-txt-link-freetext"
href="https://wikitech.wikimedia.org/wiki/Labs_labs_labs/Bare_Metal"
target="_blank">https://wikitech.wikimedia.org<wbr>/wiki/Labs_labs_labs/Bare_Meta<wbr>l</a><br>
<br>
If GPU instance support is in nova and actually
maintained, that might be worth a try.<span
class="gmail-m_-5394071749873894282HOEnZb"><font
color="#888888"><br>
<br>
-A</font></span>
<div>
<div class="gmail-m_-5394071749873894282h5"><br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">I gather the former would require
different, if not deeper analysis (cf. <a
moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/HeterogeneousGpuAcceleratorSupport"
target="_blank">HeterogeneousGpuAcceleratorSup<wbr>port</a>).
<div><br>
</div>
<div>-Adam<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 21,
2017 at 12:08 PM, Andrew Bogott <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:abogott@wikimedia.org"
target="_blank">abogott@wikimedia.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974moz-cite-prefix">Can
y'all tell me a bit more about how
this would relate to labs? Is the
idea that you want an option to
create VMs with virtualized GPU
hardware? Or... something else?
(I'm not immediately clear on how
that would work, but I'm largely
ignorant on the subject.)<span
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709HOEnZb"><font
color="#888888"><br>
<br>
-A</font></span>
<div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709h5"><br>
<br>
<br>
On 2/21/17 12:02 PM, Aaron
Halfaker wrote:<br>
</div>
</div>
</div>
<div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709h5">
<blockquote type="cite">
<div dir="auto">+9000 :D.
(also + Andrew bogott)
<div dir="auto"><br>
</div>
<div dir="auto">Would love
to have this kind of
resource in labs and
openly available. I'm
personally stoked to start
experimenting but not
ready to invest in
production GPUs yet. A
few of my external
collaborators have asked
about GPU resources in
labs too.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Feb 21, 2017 11:52, "Adam
Baso" <<a
moz-do-not-send="true"
href="mailto:abaso@wikimedia.org"
target="_blank">abaso@wikimedia.org</a>>
wrote:<br
type="attribution">
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Following
up on this here
current thread, what
do you all think about
doing the GPU
acceleration in Labs
first?
<div><br>
</div>
<div>I don't know if
it was halfak or
marktraceur who
suggested it first
(although Aaron's
brought it up a
couple times now),
but it's <i>probably</i>
less up front
architectural
overhead to start
out with, even if in
the future we'd have
a strict requirement
on HA (Q1 FY 18-19
at the very latest
under current
guess). As Aaron has
rightly noted doing
this in Labs also
lets us learn plus
give community
members greater
access to innovate
early, too. The
primary downside of
not getting this in
production with HA
up front is if
funding dries up for
FY 18-19 we're stuck
or pegged to certain
workflows later on.
But maybe it's not
worth worrying about
that too much.
<div><br>
</div>
<div>I think one
very much open
question, though,
would be if it's
possible to have a
machine with the
GPU card installed
and specifically
assigned in Labs.
Does anyone know
if that's actually
possible?</div>
<div><br>
</div>
<div>-Adam</div>
<div><br>
</div>
<div><br>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 16, 2017 at
4:56 PM, Aaron
Halfaker <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ahalfaker@wikimedia.org" target="_blank">ahalfaker@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px
0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">+1
for looping me
in on that
thread &
discussion.
I'd like to loop
in someone from
Labs (probably
andrebogott)
about purchasing
GPUs for Labs so
that we can (1)
run prediction
models in Beta
and (2) empower
our
volunteers/external
researchers to
experiment with
us. <span
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045HOEnZb"><font
color="#888888">
<div><br>
</div>
<div>-Aaron</div>
</font></span></div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 16,
2017 at 12:11
PM, Adam Baso
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:abaso@wikimedia.org" target="_blank">abaso@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">+Corey</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 16,
2017 at 11:15
AM, Adam Baso
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:abaso@wikimedia.org" target="_blank">abaso@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">We
have a need
for push
notification
servers
already, so
I've opened a
thread with
Mark and
Faidon about
getting those
servers and
putting in the
Nvidia-recommended cards for TensorFlow (ostensibly for machine vision),
for the sake
of simplifying
assumptions
about
hardware. I'm
awaiting their
feedback about
whether we
actually need
to split the
servers. If we
<i>do</i> need
to split the
servers for
separate
purposes, then
I think that
means we'd
push back the
online
computer
vision servers
and GPUs
purchase to
early Q4 FY
17-18 rather
than just
getting it
done Q1 FY
17-18 - which
is when we
need to be
moving
aggressively
on push
notification
so it would be
prudent to
just get it
done in one
fell swoop.
<div><br>
</div>
<div>Aaron, I
know last week
you had said
you'd be <i>open</i> to
collaborating
on this...and
I was quite
noncommittal!...But
I appreciate
your saying
you'd <i>like</i> to
collaborate
here. Would
you like if I
loop you on
that thread
with Mark and
Faidon? <b>Any
others who
should / would
like to join
that thread?</b>
Just be
prepared for
the thread to
be covering
two separate
use cases -
one on
cross-platform
push
notification
and one on
basic machine
vision.</div>
<span
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513HOEnZb"><font
color="#888888">
<div><br>
</div>
<div>-Adam</div>
</font></span></div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 16,
2017 at 10:57
AM, Aaron
Halfaker <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ahalfaker@wikimedia.org"
target="_blank">ahalfaker@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">OK
so I think
we'll want to
(1) get a GPU
in the stat
boxes ASAP and
(2) decide
whether we
want to plan
GPU resources
in Prod for
FY2018 or
FY2019.
<div><br>
</div>
<div>For (2),
I don't think
my team's
current plans
will bring us
to using the
GPU in
production in
the next year,
but I suspect
that Reading
may want to
push some work
re. image
processing in
that time. If
that's the
case, I want
my team to be
able to
collaborate
and support
getting that
deployed in
prod. To do
this well, I
want GPU
resources in
Wikimedia Labs
too. That
sounds like a
whole other
can of worms.
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Feb 15,
2017 at 4:40
PM, Ellery
Wulczyn <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ewulczyn@wikimedia.org"
target="_blank">ewulczyn@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Having
GPUs for
training
should be
sufficient for
now, although
if we end up
getting a ton
of use, using
GPUs could be
a lot faster
and probably
cheaper than
provisioning
the same
compute amount
of CPUs.</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 12:21
PM, Aaron
Halfaker <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ahalfaker@wikimedia.org"
target="_blank">ahalfaker@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">If
we only need
the GPU for
model
training,
it'll be OK to
use one stat
box. If we
need the GPU
for
scoring/predictions,
we'll need a
whole new
hardware
plan. </div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 1:06
PM, Andrew
Otto <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:otto@wikimedia.org" target="_blank">otto@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Hm, a good rule of thumb
is: If it can
be offline or
not running
and not affect
end users,
then it is
probably fine
to use a stat
box.</div>
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 1:56
PM, Adam Baso
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:abaso@wikimedia.org" target="_blank">abaso@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Got
it - I take
that to be the
case whether
it's batched
operation
(e.g., on
millions of
files) or it's
more of an
in-the-user-flow
sort of thing.
Is that right?</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 10:55
AM, Andrew
Otto <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:acotto@gmail.com" target="_blank">acotto@gmail.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I’d
say, if you
are going to
just to
analytics type
stuff, then
the single
stat machine
will do. If
you want to
depend on a
GPU for a
end-user
production
thing, then
you’ll have to
work with ops
to find
another place
to run it. :/
:)</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 11:40
AM, Aaron
Halfaker <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ahalfaker@wikimedia.org"
target="_blank">ahalfaker@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Ellery,
will we need
the GPUs in
order to use a
NN or will we
only need it
for training
models?</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 10:21
AM, Adam Baso
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:abaso@wikimedia.org" target="_blank">abaso@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I
envision two
primary uses:
<div><br>
</div>
<div>1) Large
scale batch
offline
processing of
existing media
assets so that
the material
is ready for
curatorial
flows.</div>
<div>2) As
part of an end
user flow
where multiple
concurrent
users are
uploading
media and
verifying and
adding
structured
data on the
fly as part of
production
use.</div>
<div><br>
</div>
<div>Can both
of these be
done on stats
machines?</div>
<div><br>
</div>
<div>Ought we
have GPU
acceleration
in two
machines
instead of one
machine?</div>
<span
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639HOEnZb"><font
color="#888888">
<div><br>
</div>
<div>-Adam</div>
<div><br>
</div>
<div><br>
</div>
</font></span></div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 8:15
AM, Andrew
Otto <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:otto@wikimedia.org" target="_blank">otto@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Oh ya! If you have a use
case for this
too, all the
better!</div>
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div
class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Do you need it for
analytics type
work? Or do
you need it to
process stuff
for a
production
feature?</div>
</div>
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639m_-4451679102871761861HOEnZb">
<div
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639m_-4451679102871761861h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Feb 2,
2017 at 9:10
AM, Aaron
Halfaker <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:ahalfaker@wikimedia.org"
target="_blank">ahalfaker@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi Adam,</div>
<div><br>
</div>
<div>+ a bunch
of CCs</div>
<div><br>
</div>
Last I heard,
Dario thought
we might be
able to cover
the cost with
Research
budget. Otto
thought that
we could get a
top of line
GPU and load
it into an
analytics
machine some
time in Q4 of
this year.
Ellery was
planning to
use 3rd party
GPU processing
services until
it was ready.
<div><br>
<div>
<div>See <a
moz-do-not-send="true"
href="https://phabricator.wikimedia.org/T148843" target="_blank">https://phabricator.wikime<wbr>dia.org/T148843</a></div>
</div>
</div>
<div><br>
</div>
<div>-Aaron</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Feb 1,
2017 at 6:01
PM, Adam Baso
<span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:abaso@wikimedia.org" target="_blank">abaso@wikimedia.org</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Aaron,
okay if I
schedule a 20
minute meeting
with you to
talk servers
with GPUs?
<div><br>
</div>
<div>Broadly,
I'm trying to
figure out
what server
CapEx I need
to ask of Mark
(e.g., for
TensorFlow
object
detection in
anticipation
of work later
in FY 17-18 /
earlier FY
18-19). I had
asked him the
other day
about when he
needs requests
for next FY,
and he
basically said
the sooner the
better.</div>
<span
class="gmail-m_-5394071749873894282m_245415591453473483gmail-m_-4240006389916125709m_-1941634467393186974m_-4826646701549151045m_7327860486425803260m_-7609473567412279513m_-7468477604771789240m_-6986456113631074273m_5713288028632980946m_4719900974687532620m_6637965112908006474m_-7130635102425802995m_6867535296650873161m_-1614115059232508639m_-4451679102871761861m_-659134508675495638m_4072067741347120972HOEnZb"><font
color="#888888">
<div><br>
</div>
<div>-Adam</div>
</font></span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>