[Labs-l] Commands are so slow in Labs

[Huji Lee]

What does "webservice" refer to here? My labs projects are not serving the
web (they are mostly bots); would that mean I won't benefit from k8s?

On Sun, Nov 6, 2016 at 1:07 AM, Martin Urbanec <martin.urbanec at wikimedia.cz>
wrote:

> You can migrate yourself by running webservice --backend=gridengine stop
> && webservice --backend=kubernetes start from the default to kubernetes
> and webservice --backend=kubernetes stop && webservice --backend=gridengine
> start for switching from kubernetes to the default and then try it. You can
> run it any time you want.
>
> Martin
>
> ne 6. 11. 2016 v 1:11 odesílatel Huji Lee <huji.huji at gmail.com> napsal:
>
>> "Kubernetes" was the word I couldn't remember.
>>
>> How can I get a project hosted in k8s? If possible, I don't want my
>> project to be "moved" to k8s; rather, I want a new project to be made in
>> k8s, so I can test it and figure out. If I decide to migrate, then I will
>> ask my actual project to be moved to k8s. Is that possible?
>>
>> On Sat, Nov 5, 2016 at 2:17 PM, Bryan Davis <bd808 at wikimedia.org> wrote:
>>
>> On Sat, Nov 5, 2016 at 8:37 AM, Huji Lee <huji.huji at gmail.com> wrote:
>> > Yeah I think it has to do with NFS too. The slow part is clearly the I/O
>> > operations.
>> >
>> > I heard there is a new container we are working on in Labs (whose name I
>> > forgot) which is faster that NFS. Is that virtualenv?
>>
>> Webservices running with the kubernetes backend [0] have demonstrated
>> much faster NFS access for things like Python virtualenvs. This is
>> because the kubernetes containers (called "pods" in the upstream
>> documentation) are configured to use an NFS cache layer that is not
>> enabled on the OGE grid nodes or the bastions.
>>
>> The reasons that the grid and bastions do not have this NFS cache
>> enabled are a bit murky, but at some point in the past it was the
>> cache was disabled as a potential fix for a problem with stale data
>> being read by some tools/bots. The techops team is cautious about
>> changing the settings to re-enable the cache due lack of understanding
>> about what could break as a result. Since the kubernetes environment
>> is opt-in and new, they were confident that enabling the cache system
>> there would not cause unknown problems.
>>
>> The larger issue of NFS slowness is to some extent a side effect of
>> the growth of Tool Labs usage. More users, tools, and bots reading and
>> writing files on the NFS shares cause more contention for the finite
>> resources the underlying NFS servers have. A project has been underway
>> for several months to migrate the NFS services to a new set of
>> servers. This migration is currently scheduled to complete on November
>> 14th [1]. There are no guarantees that NFS access will be
>> significantly faster once the new servers are in use, but the setup
>> will be better documented and more familiar to the Labs techops team.
>> That should make it easier for them to troubleshoot issues and work
>> towards providing a better experience for all of us.
>>
>> [0]: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes
>> [1]: https://lists.wikimedia.org/pipermail/labs-announce/2016-
>> November/000177.html
>>
>> Bryan
>> --
>> Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
>> [[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
>> irc: bd808                                        v:415.839.6885 x6855
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20161106/d9e8e20e/attachment.html>