I am finishing the upgrade of elasticsearch to 1.7.5 for codfw (eqiad still to do). For this, I used a small script [1], heavily inspired (copied / stolen / ...) from bd808. The script is ugly, but it does the job. It runs the deployment over a list of hosts, pausing for manual steps / validations along the way. The script runs locally on my workstation, so it is subject to loss of connectivity, local crashes, hard to handover, ...
Do we have a central place for those kind of scripts? I'd like to version it in a more obvious place than my personal Github repo. Do we have examples of similar scripts? A specific tool for this? Rundeck [2] comes to mind. Note that I'm not a huge fan of Rundeck as it brings far too much complexity for simple tasks, but the concept of having a central place of re usable operational components is appealing.
[1] https://github.com/gehel/elasticsearch-utility-scripts [2] http://rundeck.org/
On Tue, Mar 1, 2016 at 6:02 PM, Guillaume Lederrey glederrey@wikimedia.org wrote:
I am finishing the upgrade of elasticsearch to 1.7.5 for codfw (eqiad still to do). For this, I used a small script [1], heavily inspired (copied / stolen / ...) from bd808. The script is ugly, but it does the job. It runs the deployment over a list of hosts, pausing for manual steps / validations along the way. The script runs locally on my workstation, so it is subject to loss of connectivity, local crashes, hard to handover, ...
Do we have a central place for those kind of scripts? I'd like to version it in a more obvious place than my personal Github repo. Do we have examples of similar scripts? A specific tool for this? Rundeck [2] comes to mind. Note that I'm not a huge fan of Rundeck as it brings far too much complexity for simple tasks, but the concept of having a central place of re usable operational components is appealing.
[1] https://github.com/gehel/elasticsearch-utility-scripts [2] http://rundeck.org/
We already have es-tool[0] in puppet which is installed on all Elastic nodes. You can easily iterate over the list of nodes and issue a `sudo es-tool restart-fast` (which waits for recovery, btw)
-Chad
[0] https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/e...
On Tue, Mar 1, 2016 at 7:30 PM, Chad Horohoe chorohoe@wikimedia.org wrote:
On Tue, Mar 1, 2016 at 6:02 PM, Guillaume Lederrey glederrey@wikimedia.org wrote:
I am finishing the upgrade of elasticsearch to 1.7.5 for codfw (eqiad still to do). For this, I used a small script [1], heavily inspired (copied / stolen / ...) from bd808. The script is ugly, but it does the job. It runs the deployment over a list of hosts, pausing for manual steps / validations along the way. The script runs locally on my workstation, so it is subject to loss of connectivity, local crashes, hard to handover, ...
Do we have a central place for those kind of scripts? I'd like to version it in a more obvious place than my personal Github repo. Do we have examples of similar scripts? A specific tool for this? Rundeck [2] comes to mind. Note that I'm not a huge fan of Rundeck as it brings far too much complexity for simple tasks, but the concept of having a central place of re usable operational components is appealing.
[1] https://github.com/gehel/elasticsearch-utility-scripts [2] http://rundeck.org/
We already have es-tool[0] in puppet which is installed on all Elastic nodes. You can easily iterate over the list of nodes and issue a `sudo es-tool restart-fast` (which waits for recovery, btw)
Yep, my script actually uses es-tool. It is mainly some glue around it to iterate on a list of servers. I'm interested in that glue. How do we orchestrate deployment across multiple servers? The locally running script that I have does not feel like a solution...
-Chad
[0] https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/e...
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery
Guillaume Lederrey, 01/03/2016 19:02:
Do we have a central place for those kind of scripts?
More than one. :P https://git.wikimedia.org/summary/?r=operations/software.git https://git.wikimedia.org/summary/?r=mediawiki/tools/code-utils.git https://git.wikimedia.org/summary/?r=mediawiki/tools/release.git
Nemo
Thanks! Looks like I need to add my scripts there... I will have a look...
On Tue, Mar 1, 2016 at 7:39 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Guillaume Lederrey, 01/03/2016 19:02:
Do we have a central place for those kind of scripts?
More than one. :P https://git.wikimedia.org/summary/?r=operations/software.git https://git.wikimedia.org/summary/?r=mediawiki/tools/code-utils.git https://git.wikimedia.org/summary/?r=mediawiki/tools/release.git
Nemo
discovery mailing list discovery@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/discovery