[Labs-l] Commands are so slow in Labs

[Bryan Davis]

On Sat, Nov 5, 2016 at 8:37 AM, Huji Lee <huji.huji at gmail.com> wrote:
> Yeah I think it has to do with NFS too. The slow part is clearly the I/O
> operations.
>
> I heard there is a new container we are working on in Labs (whose name I
> forgot) which is faster that NFS. Is that virtualenv?

Webservices running with the kubernetes backend [0] have demonstrated
much faster NFS access for things like Python virtualenvs. This is
because the kubernetes containers (called "pods" in the upstream
documentation) are configured to use an NFS cache layer that is not
enabled on the OGE grid nodes or the bastions.

The reasons that the grid and bastions do not have this NFS cache
enabled are a bit murky, but at some point in the past it was the
cache was disabled as a potential fix for a problem with stale data
being read by some tools/bots. The techops team is cautious about
changing the settings to re-enable the cache due lack of understanding
about what could break as a result. Since the kubernetes environment
is opt-in and new, they were confident that enabling the cache system
there would not cause unknown problems.

The larger issue of NFS slowness is to some extent a side effect of
the growth of Tool Labs usage. More users, tools, and bots reading and
writing files on the NFS shares cause more contention for the finite
resources the underlying NFS servers have. A project has been underway
for several months to migrate the NFS services to a new set of
servers. This migration is currently scheduled to complete on November
14th [1]. There are no guarantees that NFS access will be
significantly faster once the new servers are in use, but the setup
will be better documented and more familiar to the Labs techops team.
That should make it easier for them to troubleshoot issues and work
towards providing a better experience for all of us.

[0]: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes
[1]: https://lists.wikimedia.org/pipermail/labs-announce/2016-November/000177.html

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855