[QA] WMF Continuous Integration currently offline

[aude]

On Wed, Jun 8, 2016 at 4:48 PM, Antoine Musso <hashar+wmf at free.fr> wrote:

> On 08/06/16 18:47, Antoine Musso wrote:
>
>> Le 08/06/2016 à 15:02, Antoine Musso a écrit :
>>
>>>
>>> The operation team has worked hard this European morning to backup
>>> files, investigate the raid issue and setup a new host.
>>>
>>> We are in the process of reinstalling everything on the new host and
>>> bring back Jenkins and Zuul on it.
>>>
>>> No ETA yet, since a 5 years old boxes must have hidden issues which
>>> makes it hard to estimate how long it would need to fully recover.
>>>
>>
>> A status update:
>>
>> Ops (Jaime, Faidon, Mark, Chris) had a disk replaced and the raid array
>> is rebuilding right now.  Should take roughly an hour from now.  If the
>> disk and raid are confirmed to be fine, we would bring back Jenkins and
>> Zuul.
>>
>> A new server has been installed contint1001. Jenkins data are being
>> copied there.  We would need to adjust a few network rules and update IP
>> address in configuration files then attempt to switch to that new setup.
>>
>> Main task is:
>> https://phabricator.wikimedia.org/T137265
>>
>
> The CI service is back since 19:00 UTC after a disk got replaced and the
> RAID array rebuild successfully.
>

Thanks hashar and everyone who helped out.

Cheers,
Katie


>
> The issue might well occurs again and we would move the various services
> out of the server (gallium).
>
> --
> Antoine Musso
>
>
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>



-- 
@wikidata
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20160608/0a10b0fd/attachment.html>