Hi,
If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?
I'm asking about this because soon we'll see requests about removing more and more Jessie support from the Puppet codebase [0] and there's no exit strategy from Jessie for Shinken (it's not available in Stretch unless we want to package things ourselves, which I tried and failed).
0 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/491460
Thanks,
Hi!
On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni gtirloni@wikimedia.org wrote:
Hi,
If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?
I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?
I'm asking about this because soon we'll see requests about removing more and more Jessie support from the Puppet codebase [0] and there's no exit strategy from Jessie for Shinken (it's not available in Stretch unless we want to package things ourselves, which I tried and failed).
Ouch! I didn't realize there's no shinken in stretch :( though it seems like a great occasion to get rid of it!
HTH, Filippo
Shinken also seems like an abandoned project. It would be great to move to icinga2 so we’d be on a live project, and that’s in Debian stable. It would need very similar management to shinken if we went that route since it would be a cloud-internal service (possibly improved over what we do now?). I don’t believe that opening the prod firewall to nrpe interactions happening with Cloud VPS is wise.
Brooke Storm Operations Engineer Wikimedia Cloud Services bstorm@wikimedia.org mailto:bstorm@wikimedia.org IRC: bstorm_
On Feb 25, 2019, at 11:06 AM, Filippo Giunchedi fgiunchedi@wikimedia.org wrote:
Hi!
On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni <gtirloni@wikimedia.org mailto:gtirloni@wikimedia.org> wrote: Hi,
If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?
I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?
I'm asking about this because soon we'll see requests about removing more and more Jessie support from the Puppet codebase [0] and there's no exit strategy from Jessie for Shinken (it's not available in Stretch unless we want to package things ourselves, which I tried and failed).
Ouch! I didn't realize there's no shinken in stretch :( though it seems like a great occasion to get rid of it!
HTH, Filippo _______________________________________________ Cloud-admin mailing list Cloud-admin@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/cloud-admin
On 2/25/19 12:06 PM, Filippo Giunchedi wrote:
Hi!
On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni <gtirloni@wikimedia.org mailto:gtirloni@wikimedia.org> wrote:
Hi, If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?
One of the reasons we haven't done this already is that WMF production monitoring seems to be in constant flux -- it's disappointing to adopt a monitoring tool only to have it deprecated a month later in production. Is the current version of icinga that's running in prod actually your desired final state there, or is something else up and coming?
-Andrew
I'm asking about this because soon we'll see requests about removing more and more Jessie support from the Puppet codebase [0] and there's no exit strategy from Jessie for Shinken (it's not available in Stretch unless we want to package things ourselves, which I tried and failed).Ouch! I didn't realize there's no shinken in stretch :( though it seems like a great occasion to get rid of it!
HTH, Filippo
Cloud-admin mailing list Cloud-admin@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/cloud-admin
Hi all,
On Mon, Feb 25, 2019 at 7:33 PM Andrew Bogott abogott@wikimedia.org wrote:
On 2/25/19 12:06 PM, Filippo Giunchedi wrote:
Hi!
On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni gtirloni@wikimedia.org wrote:
Hi,
If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?
I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?
One of the reasons we haven't done this already is that WMF production monitoring seems to be in constant flux -- it's disappointing to adopt a monitoring tool only to have it deprecated a month later in production. Is the current version of icinga that's running in prod actually your desired final state there, or is something else up and coming?
Yes I can confirm that the current icinga+stretch combo in production is what we'll stay on. I think there's consensus that icinga is showing its age, however the project of replacing icinga (or even deciding not to!) AFAIK isn't formally on the radar, IOW production icinga deprecation (if any) is many quarters out.
HTH, Filippo
On 2/26/19 2:40 AM, Filippo Giunchedi wrote:
Hi all,
On Mon, Feb 25, 2019 at 7:33 PM Andrew Bogott <abogott@wikimedia.org mailto:abogott@wikimedia.org> wrote:
On 2/25/19 12:06 PM, Filippo Giunchedi wrote:Hi! On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni <gtirloni@wikimedia.org <mailto:gtirloni@wikimedia.org>> wrote: Hi, If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route? I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?One of the reasons we haven't done this already is that WMF production monitoring seems to be in constant flux -- it's disappointing to adopt a monitoring tool only to have it deprecated a month later in production. Is the current version of icinga that's running in prod actually your desired final state there, or is something else up and coming?Yes I can confirm that the current icinga+stretch combo in production is what we'll stay on. I think there's consensus that icinga is showing its age, however the project of replacing icinga (or even deciding not to!) AFAIK isn't formally on the radar, IOW production icinga deprecation (if any) is many quarters out.
Ok! So, in that case, icinga-on-a-VM seems like the obvious move to me. Giovanni, does that suit you OK?
HTH, Filippo
On 2/26/19 11:24 AM, Andrew Bogott wrote:
On 2/26/19 2:40 AM, Filippo Giunchedi wrote:
Hi all,
On Mon, Feb 25, 2019 at 7:33 PM Andrew Bogott <abogott@wikimedia.org mailto:abogott@wikimedia.org> wrote:
On 2/25/19 12:06 PM, Filippo Giunchedi wrote:Hi! On Mon, Feb 25, 2019 at 6:40 PM Giovanni Tirloni <gtirloni@wikimedia.org <mailto:gtirloni@wikimedia.org>> wrote: Hi, If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route? I'm not sure, what checks does shinken do ATM that we'd need to port over to icinga? Right off the bat it seems simpler to me to fire up a icinga stretch VM and move over the checks and their puppetization to that VM instead, what do you think?One of the reasons we haven't done this already is that WMF production monitoring seems to be in constant flux -- it's disappointing to adopt a monitoring tool only to have it deprecated a month later in production. Is the current version of icinga that's running in prod actually your desired final state there, or is something else up and coming?Yes I can confirm that the current icinga+stretch combo in production is what we'll stay on. I think there's consensus that icinga is showing its age, however the project of replacing icinga (or even deciding not to!) AFAIK isn't formally on the radar, IOW production icinga deprecation (if any) is many quarters out.
Ok! So, in that case, icinga-on-a-VM seems like the obvious move to me. Giovanni, does that suit you OK?
Sounds good to me. Re-used the Puppet code we have like Fillipo said and get going :)
I'll run a few tests and create a phab task for this. Thanks all for the input!
Hi Giovanni,
It doesn't feel right from a security standpoint to merge the production and cloud services concerns. I would also be concerned about the additional load placed on the production Icinga instance (check latency comes to mind).
I would recommend we spin up cloud services-specific copy of what we have in production until we know what our next monitoring solution looks like.
If you would like a hand, I could make some time to assist.
CW
On Mon, Feb 25, 2019 at 10:40 AM Giovanni Tirloni gtirloni@wikimedia.org wrote:
Hi,
If we poke roles in the firewall so Icinga can reach the VMs and we define the monitoring::service stuff in Puppet, is that all we need to shutdown Shinken? Do you think there would be any concerns with going that route?
I'm asking about this because soon we'll see requests about removing more and more Jessie support from the Puppet codebase [0] and there's no exit strategy from Jessie for Shinken (it's not available in Stretch unless we want to package things ourselves, which I tried and failed).
0 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/491460
Thanks,
-- Giovanni Tirloni Operations Engineer Wikimedia Foundation
cloud-admin@lists.wikimedia.org