Hive and Oozie unavailable for a brief hardware maintenance on Sept 7th - Analytics

Luca Toscano

7 Sep 7 Sep

11:57 a.m.

New subject: Hive and Oozie unavailable for a brief hardware maintenance on Sept 7th

Hi again, the maintenance didn't happen because we currently have important Spark jobs that need to finish, so if nobody opposes I'd like to move it to Monday (Sept 10th, EU morning time). As always please follow up with me in https://phabricator.wikimedia.org/T203165 or IRC if this affects your work :) Thanks and sorry for the late notice! Luca Il giorno gio 6 set 2018 alle ore 17:23 Luca Toscano <ltoscano(a)wikimedia.org> ha scritto:

...

Hi everybody, just wanted to let you know that tomorrow morning EU time I'll stop Hive and Oozie on analytics1003 (Hadoop Cluster) to allow the host to be rebooted. I don't plan to kill any job but to wait for their completion. While the maintenance happens some errors might happen to your jobs. The reboot should take some minutes to complete if everything goes as planned. Tracking task: https://phabricator.wikimedia.org/T203165 Please let me know on IRC or in the task if this affects you :) Luca (on behalf of the Analytics team)

Reply

Luca Toscano

1:08 p.m.

New subject: Hive and Oozie unavailable for a brief hardware maintenance on Sept 7th

Hi Goran, sorry for the delay but this morning we didn't manage to drain the cluster and reboot, I am going to do it now. It should take a maximum of 2 hours from now. Apologies for the inconvenience. Luca Il giorno lun 10 set 2018 alle ore 13:08 Goran Milovanovic < goran.milovanovic_ext(a)wikimedia.de> ha scritto:

...

Hi, I am sorry if it turns out that you have already informed us about this: please, is the cluster reboot completed? I have a lot of Hive jobs waiting. Thanks, Goran On Fri, Sep 7, 2018, 13:57 Luca Toscano <ltoscano(a)wikimedia.org> wrote:

Hi again, the maintenance didn't happen because we currently have important Spark jobs that need to finish, so if nobody opposes I'd like to move it to Monday (Sept 10th, EU morning time). As always please follow up with me in https://phabricator.wikimedia.org/T203165 or IRC if this affects your work :) Thanks and sorry for the late notice! Luca Il giorno gio 6 set 2018 alle ore 17:23 Luca Toscano < ltoscano(a)wikimedia.org> ha scritto:

Hi everybody, just wanted to let you know that tomorrow morning EU time I'll stop Hive and Oozie on analytics1003 (Hadoop Cluster) to allow the host to be rebooted. I don't plan to kill any job but to wait for their completion. While the maintenance happens some errors might happen to your jobs. The reboot should take some minutes to complete if everything goes as planned. Tracking task: https://phabricator.wikimedia.org/T203165 Please let me know on IRC or in the task if this affects you :) Luca (on behalf of the Analytics team)

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Reply

Luca Toscano

3:32 p.m.

New subject: Hive and Oozie unavailable for a brief hardware maintenance on Sept 7th

Hi again, maintenance completed! Luca Il giorno lun 10 set 2018 alle ore 15:10 Goran Milovanovic < goran.milovanovic_ext(a)wikimedia.de> ha scritto:

...

Hi Luca, thanks for feedback, no problem at all, and good luck with your operation. Best, Goran Goran S. Milovanović, PhD Data Scientist, Software Department Wikimedia Deutschland ------------------------------------------------ "It's not the size of the dog in the fight, it's the size of the fight in the dog." - Mark Twain ------------------------------------------------ On Mon, Sep 10, 2018 at 3:09 PM Luca Toscano <ltoscano(a)wikimedia.org> wrote:

Hi Goran, sorry for the delay but this morning we didn't manage to drain the cluster and reboot, I am going to do it now. It should take a maximum of 2 hours from now. Apologies for the inconvenience. Luca Il giorno lun 10 set 2018 alle ore 13:08 Goran Milovanovic < goran.milovanovic_ext(a)wikimedia.de> ha scritto:

Hi, I am sorry if it turns out that you have already informed us about this: please, is the cluster reboot completed? I have a lot of Hive jobs waiting. Thanks, Goran On Fri, Sep 7, 2018, 13:57 Luca Toscano <ltoscano(a)wikimedia.org> wrote:

Hi again, the maintenance didn't happen because we currently have important Spark jobs that need to finish, so if nobody opposes I'd like to move it to Monday (Sept 10th, EU morning time). As always please follow up with me in https://phabricator.wikimedia.org/T203165 or IRC if this affects your work :) Thanks and sorry for the late notice! Luca Il giorno gio 6 set 2018 alle ore 17:23 Luca Toscano < ltoscano(a)wikimedia.org> ha scritto: > Hi everybody, > > just wanted to let you know that tomorrow morning EU time I'll stop > Hive and Oozie on analytics1003 (Hadoop Cluster) to allow the host to be > rebooted. I don't plan to kill any job but to wait for their completion. > While the maintenance happens some errors might happen to your jobs. The > reboot should take some minutes to complete if everything goes as planned. > > Tracking task: https://phabricator.wikimedia.org/T203165 > > Please let me know on IRC or in the task if this affects you :) > > Luca (on behalf of the Analytics team) > _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Reply