[Labs-l] Commands are so slow in Labs

[Martin Urbanec]

You can migrate yourself by running webservice --backend=gridengine stop &&
webservice --backend=kubernetes start from the default to kubernetes
and webservice --backend=kubernetes stop && webservice --backend=gridengine
start for switching from kubernetes to the default and then try it. You can
run it any time you want.

Martin

ne 6. 11. 2016 v 1:11 odesílatel Huji Lee <huji.huji at gmail.com> napsal:

> "Kubernetes" was the word I couldn't remember.
>
> How can I get a project hosted in k8s? If possible, I don't want my
> project to be "moved" to k8s; rather, I want a new project to be made in
> k8s, so I can test it and figure out. If I decide to migrate, then I will
> ask my actual project to be moved to k8s. Is that possible?
>
> On Sat, Nov 5, 2016 at 2:17 PM, Bryan Davis <bd808 at wikimedia.org> wrote:
>
> On Sat, Nov 5, 2016 at 8:37 AM, Huji Lee <huji.huji at gmail.com> wrote:
> > Yeah I think it has to do with NFS too. The slow part is clearly the I/O
> > operations.
> >
> > I heard there is a new container we are working on in Labs (whose name I
> > forgot) which is faster that NFS. Is that virtualenv?
>
> Webservices running with the kubernetes backend [0] have demonstrated
> much faster NFS access for things like Python virtualenvs. This is
> because the kubernetes containers (called "pods" in the upstream
> documentation) are configured to use an NFS cache layer that is not
> enabled on the OGE grid nodes or the bastions.
>
> The reasons that the grid and bastions do not have this NFS cache
> enabled are a bit murky, but at some point in the past it was the
> cache was disabled as a potential fix for a problem with stale data
> being read by some tools/bots. The techops team is cautious about
> changing the settings to re-enable the cache due lack of understanding
> about what could break as a result. Since the kubernetes environment
> is opt-in and new, they were confident that enabling the cache system
> there would not cause unknown problems.
>
> The larger issue of NFS slowness is to some extent a side effect of
> the growth of Tool Labs usage. More users, tools, and bots reading and
> writing files on the NFS shares cause more contention for the finite
> resources the underlying NFS servers have. A project has been underway
> for several months to migrate the NFS services to a new set of
> servers. This migration is currently scheduled to complete on November
> 14th [1]. There are no guarantees that NFS access will be
> significantly faster once the new servers are in use, but the setup
> will be better documented and more familiar to the Labs techops team.
> That should make it easier for them to troubleshoot issues and work
> towards providing a better experience for all of us.
>
> [0]: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes
> [1]:
> https://lists.wikimedia.org/pipermail/labs-announce/2016-November/000177.html
>
> Bryan
> --
> Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
> [[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
> irc: bd808                                        v:415.839.6885 x6855
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20161106/bd3ab7d4/attachment-0001.html>