Recently one of our Cloud VPS servers ran out of disk space. Is this something that you can get alerts for from Grafana or something else (e.g. by e-mail)? I looked around a bit in the Wikitech documentation and found Alertmanager#Grafana alerts https://wikitech.wikimedia.org/wiki/Alertmanager#Grafana_alerts, but it looks like that's not for users of the Cloud VPS, rather the maintainers.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
I'm taking the lack of response as a "no"🙂
We have a Grafana account that we use for other servers. Is there any reason, technical or otherwise, why it would be a bad idea to use that for the Cloud VPS server?
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
On Tue, 30 May 2023 at 14:08, Sebastian Berlin < sebastian.berlin@wikimedia.se> wrote:
Recently one of our Cloud VPS servers ran out of disk space. Is this something that you can get alerts for from Grafana or something else (e.g. by e-mail)? I looked around a bit in the Wikitech documentation and found Alertmanager#Grafana alerts https://wikitech.wikimedia.org/wiki/Alertmanager#Grafana_alerts, but it looks like that's not for users of the Cloud VPS, rather the maintainers.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
Hi, sorry for the delayed response.
First of all, the Wikitech section you found is about the wikiprod Grafana instance at https://grafana.wikimedia.org. The WMCS Grafana instance (https://grafana.wmcloud.org) only queries data from various Prometheus instances, so for those we use Prometheus and prometheus-alertmanager directly instead of adding yet another component to the alerting stack.
So, yes, it is possible to configure e-mail or IRC alerts from the Prometheus metrics we collect from all Cloud VPS instances.[0] The bad news is that there's no self-service user interface for it yet, so you would need to ask a Cloud VPS admin to do any changes manually. It's not really documented anywhere yet, except this tiny section[1] which is aimed at admins who have direct access to the project.
I'm happy to do the changes if you have a Prometheus query to alert on written already, just create a task in the #Cloud-VPS project and tag me (@taavi) on it.
[0]: https://prometheus.wmcloud.org/ [1]: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monito...
Taavi
Thanks for the reply.
I looked a bit at the links, but it's a lot of things that I don't really know how they work. When I set up Grafana to monitor our other servers there were some default monitoring that included alerts and I was hoping there was something similar here.
This isn't critical at the moment so I'll leave it for now. At some point we probably will need alerts so I may get back to you then.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
On Mon, 12 Jun 2023 at 11:06, Taavi Väänänen hi@taavi.wtf wrote:
Hi, sorry for the delayed response.
First of all, the Wikitech section you found is about the wikiprod Grafana instance at https://grafana.wikimedia.org. The WMCS Grafana instance (https://grafana.wmcloud.org) only queries data from various Prometheus instances, so for those we use Prometheus and prometheus-alertmanager directly instead of adding yet another component to the alerting stack.
So, yes, it is possible to configure e-mail or IRC alerts from the Prometheus metrics we collect from all Cloud VPS instances.[0] The bad news is that there's no self-service user interface for it yet, so you would need to ask a Cloud VPS admin to do any changes manually. It's not really documented anywhere yet, except this tiny section[1] which is aimed at admins who have direct access to the project.
I'm happy to do the changes if you have a Prometheus query to alert on written already, just create a task in the #Cloud-VPS project and tag me (@taavi) on it.
[1]:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monito...
Taavi _______________________________________________ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/