[Labs-l] Failure in submitting python job to grid engine after upgrade

[Anthony Di Franco]

Not sure if this is a labs-specific issue or just a stumper for me on how
the infrastructure works, but I'm having trouble after the upgrade of
tools-login submitting jobs from it to the grid engine in a way that worked
pre-upgrade.

I have a python script running from a virtualenv (I had to rebuild the
virtualenv for the new python version,) which, when run from tools-login
directly, runs fine, but when submitted via the grid engine, is unable to
import the SSL module correctly (symptom: httplib2 does not initialize
properly in a way that indicates SSL module did not load).

I was originally running the virtualenv-installed python executable via its
absolute path. I switched to sourcing c-env from the script and running the
python so implied in hopes that might give me a better chance of the right
environment; no luck.

I have looked at specifying PYTHONPATH and examined sys.path from within
the script to confirm it is the same in both cases.

Does anyone have any thoughts on what might be going wrong? Could a
system-wide SSL module that subtly doesn't work, or is absent, be
implicated? Differences in what the module itself, if it is the same, is
doing under the two conditions?

On Sun, Mar 22, 2015 at 2:39 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:

> Hello!
>
> If you try to ssh to tools-login now, you might get a scary looking
> SSH warning. Do not fear! Things are all good. The new fingerprint is:
>
> f9:c1:53:72:f0:57:85:dd:6f:4d:cb:f6:d0:ce:84:aa
>
> And you can remove the entry line for tools-login from your
> ~/.ssh/known_hosts and proceed to log in.
>
> Why was this happening? tools-login the old instance is a bit messed
> up, and the fastest way to restore service is to just create a new
> instance and point the old IP at it. This also allowed me to upgrade
> them to trusty, which would give you new versions of a lot of software
> :D We have preserved the old instance for forensics, and will figure
> out what had happened over the next few days.
>
> Thank you for your understanding, and sorry about the scare :)
>
> --
> Yuvi Panda T
> http://yuvi.in/blog
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150325/b3fc7d33/attachment.html>