<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">"Kubernetes" was the word I couldn't remember.<br><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">How can I get a project hosted in k8s? If possible, I don't want my project to be "moved" to k8s; rather, I want a new project to be made in k8s, so I can test it and figure out. If I decide to migrate, then I will ask my actual project to be moved to k8s. Is that possible?<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Nov 5, 2016 at 2:17 PM, Bryan Davis <span dir="ltr"><<a href="mailto:bd808@wikimedia.org" target="_blank">bd808@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Sat, Nov 5, 2016 at 8:37 AM, Huji Lee <<a href="mailto:huji.huji@gmail.com">huji.huji@gmail.com</a>> wrote:<br>
> Yeah I think it has to do with NFS too. The slow part is clearly the I/O<br>
> operations.<br>
><br>
> I heard there is a new container we are working on in Labs (whose name I<br>
> forgot) which is faster that NFS. Is that virtualenv?<br>
<br>
</span>Webservices running with the kubernetes backend [0] have demonstrated<br>
much faster NFS access for things like Python virtualenvs. This is<br>
because the kubernetes containers (called "pods" in the upstream<br>
documentation) are configured to use an NFS cache layer that is not<br>
enabled on the OGE grid nodes or the bastions.<br>
<br>
The reasons that the grid and bastions do not have this NFS cache<br>
enabled are a bit murky, but at some point in the past it was the<br>
cache was disabled as a potential fix for a problem with stale data<br>
being read by some tools/bots. The techops team is cautious about<br>
changing the settings to re-enable the cache due lack of understanding<br>
about what could break as a result. Since the kubernetes environment<br>
is opt-in and new, they were confident that enabling the cache system<br>
there would not cause unknown problems.<br>
<br>
The larger issue of NFS slowness is to some extent a side effect of<br>
the growth of Tool Labs usage. More users, tools, and bots reading and<br>
writing files on the NFS shares cause more contention for the finite<br>
resources the underlying NFS servers have. A project has been underway<br>
for several months to migrate the NFS services to a new set of<br>
servers. This migration is currently scheduled to complete on November<br>
14th [1]. There are no guarantees that NFS access will be<br>
significantly faster once the new servers are in use, but the setup<br>
will be better documented and more familiar to the Labs techops team.<br>
That should make it easier for them to troubleshoot issues and work<br>
towards providing a better experience for all of us.<br>
<br>
[0]: <a href="https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes" rel="noreferrer" target="_blank">https://wikitech.wikimedia.<wbr>org/wiki/Help:Tool_Labs/Web/<wbr>Kubernetes</a><br>
[1]: <a href="https://lists.wikimedia.org/pipermail/labs-announce/2016-November/000177.html" rel="noreferrer" target="_blank">https://lists.wikimedia.org/<wbr>pipermail/labs-announce/2016-<wbr>November/000177.html</a><br>
<span class="HOEnZb"><font color="#888888"><br>
Bryan<br>
--<br>
Bryan Davis Wikimedia Foundation <<a href="mailto:bd808@wikimedia.org">bd808@wikimedia.org</a>><br>
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA<br>
irc: bd808 v:<a href="tel:415.839.6885%20x6855" value="+14158396885">415.839.6885 x6855</a><br>
</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" rel="noreferrer" target="_blank">https://lists.wikimedia.org/<wbr>mailman/listinfo/labs-l</a><br>
</div></div></blockquote></div><br></div>