[Labs-l] second attempt to request alternative login server

[Petr Bena]

On Wed, Mar 6, 2013 at 6:01 PM, Jeremy Baron <jeremy at tuxmachine.com> wrote:
> On Wed, Mar 6, 2013 at 4:54 PM, Petr Bena <benapetr at gmail.com> wrote:
>> okay this is third time when we have same outage... bastion2 and 3
>> were accessible for short time after bastion1's gluster died, then
>> they died as well. public keys weren't accessible on any of them so
>> basically labs were inaccessible for anyone.
>
> citation needed. I was just able to log in to both of
> bastion[23].wmflabs.org on the first try.

yes - I said they were working, now they don't. When bastion1 died I
was still there, so I could check that /public/keys wasn't readable at
all (some IO error) I don't have root on bastion so I couldn't read
logs though, no idea why they aren't readable for everyone

>
> [removed garbage about password auth being wonderful...]
>

HOW DARE YOU

>> Set up a cron script that sync a local folder on bastion with
>> /public/keys so that when gluster is down or that folder isn't working
>> login to bastion's still works.
>
> That might be feasible. But really the solution is don't let people
> kill the bastion. idk how we do that. and idk why the past social
> restrictions aren't sufficient. maybe we need ulimit or cgroups or
> something. :-(
>

it weren't people who kill them it was gluster or something like that
- we need reliable storage for keys if it's only way to login

> -Jeremy
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l