Hi everybody,
We are going to change the Yarn scheduler in a bit, moving from Fair to Capacity. More info in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop#Queues
It shouldn't impact ongoing jobs, but there will probably be some tuning/adjustment to do during the next few days. If you see any weird behavior please ping us or contact us in T277062.
A gift for your patience: this change will allow users to select hadoop nodes with GPUs when launching jobs :) (we have 6, running the same GPUs as stat100[5,8]).
Luca
The capacity scheduler has been deployed, we'll add the GPU labels tomorrow as a second step (to avoid monitoring too many changes at once).
Luca
On Mon, Apr 26, 2021 at 8:50 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
We are going to change the Yarn scheduler in a bit, moving from Fair to Capacity. More info in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop#Queues
It shouldn't impact ongoing jobs, but there will probably be some tuning/adjustment to do during the next few days. If you see any weird behavior please ping us or contact us in T277062.
A gift for your patience: this change will allow users to select hadoop nodes with GPUs when launching jobs :) (we have 6, running the same GPUs as stat100[5,8]).
Luca
Hi everybody,
GPU label added, please check https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop#Queues for more info about how to use it!
Luca
On Mon, Apr 26, 2021 at 10:53 AM Luca Toscano ltoscano@wikimedia.org wrote:
The capacity scheduler has been deployed, we'll add the GPU labels tomorrow as a second step (to avoid monitoring too many changes at once).
Luca
On Mon, Apr 26, 2021 at 8:50 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
We are going to change the Yarn scheduler in a bit, moving from Fair to Capacity. More info in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop#Queues
It shouldn't impact ongoing jobs, but there will probably be some tuning/adjustment to do during the next few days. If you see any weird behavior please ping us or contact us in T277062.
A gift for your patience: this change will allow users to select hadoop nodes with GPUs when launching jobs :) (we have 6, running the same GPUs as stat100[5,8]).
Luca
analytics-announce@lists.wikimedia.org