Hello!
We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262 that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.
What we need from you:
1. A quick review to see if we're missing anything. 2. A list of Kibana dashboards and saved searches that need translation to use the new format.
Thanks in advance! And as always, questions are welcome!
On 5/12/21 3:50 PM, Cole White wrote:
Hello!
We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262 that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.
This is great! I don't think anyone is super attached to the current format of the logs, so anything you want to do to rework them is great. The only thing moderately interesting about openstack logs is that they include a request ID which is passed around and consistent across the different services; it's very useful for tracking particular issues so it should maintain pride of place in whatever post-processed messages we wind up with.
What we need from you:
- A quick review to see if we're missing anything.
Done, I've listed three more services in the patch: glance, trove, barbican
- A list of Kibana dashboards and saved searches that need translation to use the new format.
There are a lot -- basically one per service x one per deployment. You can find the list by searching for 'OpenStack' and then disregarding the Horizon and haproxy things.
That said, the most important dashboards are:
- OpenStack codfw1dev
- OpenStack eqiad1
- OpenStack Nova codfw1dev Control Services
- OpenStack Nova eqiad1 Control Services
- OpenStack Nova codfw1dev Compute
- OpenStack Nova eqiad1 Compute
If you run out of gas at the end of that list (or even part way through) feel free to drop the cleanup on me and I can decide if the other dashboards are worth updating or can be done away with.
Thanks in advance! And as always, questions are welcome!
Thanks again!
-Andrew
-- Cole White Wikimedia Foundation
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org To unsubscribe send an email to cloud-admin-leave@lists.wikimedia.org
Hey Andrew, thanks for the help!
Those dashboards you listed now have ECS equivalents. Please have a look when you have a chance. Note that the ECS-formatted logs are being duplicated and these duplicated logs go back no more than two days.
If those dashboards look good to you, let me know and we'll turn off the duplication. Doing that will fully transition these logs to ECS and lift the two-day retention limit.
Thanks!
On Thu, May 13, 2021 at 1:00 PM Andrew Bogott abogott@wikimedia.org wrote:
On 5/12/21 3:50 PM, Cole White wrote:
Hello!
We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262 that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.
This is great! I don't think anyone is super attached to the current format of the logs, so anything you want to do to rework them is great. The only thing moderately interesting about openstack logs is that they include a request ID which is passed around and consistent across the different services; it's very useful for tracking particular issues so it should maintain pride of place in whatever post-processed messages we wind up with.
What we need from you:
- A quick review to see if we're missing anything.
Done, I've listed three more services in the patch: glance, trove, barbican
- A list of Kibana dashboards and saved searches that need
translation to use the new format.
There are a lot -- basically one per service x one per deployment. You can find the list by searching for 'OpenStack' and then disregarding the Horizon and haproxy things.
That said, the most important dashboards are:
OpenStack codfw1dev
OpenStack eqiad1
OpenStack Nova codfw1dev Control Services
OpenStack Nova eqiad1 Control Services
OpenStack Nova codfw1dev Compute
OpenStack Nova eqiad1 Compute
If you run out of gas at the end of that list (or even part way through) feel free to drop the cleanup on me and I can decide if the other dashboards are worth updating or can be done away with.
Thanks in advance! And as always, questions are welcome!
Thanks again!
-Andrew
-- Cole White Wikimedia Foundation
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org To unsubscribe send an email to cloud-admin-leave@lists.wikimedia.org
Hey all,
It has been a couple weeks since the identified dashboards were transitioned to ECS. At this point, we're ready to fully transition Openstack logs and turn off duplication to the legacy indexes.
To accomplish this, we intend to fully transition OpenStack logs on Thursday, June 24th provided there are no concerns. A note will be added to the legacy dashboards indicating the transition as well as the cutoff date so that prior logs can still be accessed.
Please let us know if you have any questions or concerns.
Thanks! CW
On Tue, Jun 8, 2021 at 5:00 PM Cole White cwhite@wikimedia.org wrote:
Hey Andrew, thanks for the help!
Those dashboards you listed now have ECS equivalents. Please have a look when you have a chance. Note that the ECS-formatted logs are being duplicated and these duplicated logs go back no more than two days.
If those dashboards look good to you, let me know and we'll turn off the duplication. Doing that will fully transition these logs to ECS and lift the two-day retention limit.
Thanks!
On Thu, May 13, 2021 at 1:00 PM Andrew Bogott abogott@wikimedia.org wrote:
On 5/12/21 3:50 PM, Cole White wrote:
Hello!
We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262 that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.
This is great! I don't think anyone is super attached to the current format of the logs, so anything you want to do to rework them is great. The only thing moderately interesting about openstack logs is that they include a request ID which is passed around and consistent across the different services; it's very useful for tracking particular issues so it should maintain pride of place in whatever post-processed messages we wind up with.
What we need from you:
- A quick review to see if we're missing anything.
Done, I've listed three more services in the patch: glance, trove, barbican
- A list of Kibana dashboards and saved searches that need
translation to use the new format.
There are a lot -- basically one per service x one per deployment. You can find the list by searching for 'OpenStack' and then disregarding the Horizon and haproxy things.
That said, the most important dashboards are:
OpenStack codfw1dev
OpenStack eqiad1
OpenStack Nova codfw1dev Control Services
OpenStack Nova eqiad1 Control Services
OpenStack Nova codfw1dev Compute
OpenStack Nova eqiad1 Compute
If you run out of gas at the end of that list (or even part way through) feel free to drop the cleanup on me and I can decide if the other dashboards are worth updating or can be done away with.
Thanks in advance! And as always, questions are welcome!
Thanks again!
-Andrew
-- Cole White Wikimedia Foundation
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org To unsubscribe send an email to cloud-admin-leave@lists.wikimedia.org
-- Cole White Wikimedia Foundation
This sounds just fine to me. My current workflows and expectations regarding openstack logs are still only half-formed so this is a perfectly good time to change them and/or break them.
-Andrew
On 6/21/21 11:34 AM, Cole White wrote:
Hey all,
It has been a couple weeks since the identified dashboards were transitioned to ECS. At this point, we're ready to fully transition Openstack logs and turn off duplication to the legacy indexes.
To accomplish this, we intend to fully transition OpenStack logs on Thursday, June 24th provided there are no concerns. A note will be added to the legacy dashboards indicating the transition as well as the cutoff date so that prior logs can still be accessed.
Please let us know if you have any questions or concerns.
Thanks! CW
On Tue, Jun 8, 2021 at 5:00 PM Cole White <cwhite@wikimedia.org mailto:cwhite@wikimedia.org> wrote:
Hey Andrew, thanks for the help! Those dashboards you listed now have ECS equivalents. Please have a look when you have a chance. Note that the ECS-formatted logs are being duplicated and these duplicated logs go back no more than two days. If those dashboards look good to you, let me know and we'll turn off the duplication. Doing that will fully transition these logs to ECS and lift the two-day retention limit. Thanks! On Thu, May 13, 2021 at 1:00 PM Andrew Bogott <abogott@wikimedia.org <mailto:abogott@wikimedia.org>> wrote: On 5/12/21 3:50 PM, Cole White wrote:Hello! We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch <https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262> that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.This is great! I don't think anyone is super attached to the current format of the logs, so anything you want to do to rework them is great. The only thing moderately interesting about openstack logs is that they include a request ID which is passed around and consistent across the different services; it's very useful for tracking particular issues so it should maintain pride of place in whatever post-processed messages we wind up with.What we need from you: 1. A quick review to see if we're missing anything.Done, I've listed three more services in the patch: glance, trove, barbican1. A list of Kibana dashboards and saved searches that need translation to use the new format.There are a lot -- basically one per service x one per deployment. You can find the list by searching for 'OpenStack' and then disregarding the Horizon and haproxy things. That said, the most important dashboards are: - OpenStack codfw1dev - OpenStack eqiad1 - OpenStack Nova codfw1dev Control Services - OpenStack Nova eqiad1 Control Services - OpenStack Nova codfw1dev Compute - OpenStack Nova eqiad1 Compute If you run out of gas at the end of that list (or even part way through) feel free to drop the cleanup on me and I can decide if the other dashboards are worth updating or can be done away with.Thanks in advance! And as always, questions are welcome!Thanks again! -Andrew-- Cole White Wikimedia Foundation _______________________________________________ Cloud-admin mailing list --cloud-admin@lists.wikimedia.org <mailto:cloud-admin@lists.wikimedia.org> To unsubscribe send an email tocloud-admin-leave@lists.wikimedia.org <mailto:cloud-admin-leave@lists.wikimedia.org>-- Cole White Wikimedia Foundation-- Cole White Wikimedia Foundation
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
Reminder: this change https://gerrit.wikimedia.org/r/c/operations/puppet/+/699039 is expected to roll out today.
Please let me know if you have any questions or concerns.
On Mon, Jun 21, 2021 at 10:34 AM Cole White cwhite@wikimedia.org wrote:
Hey all,
It has been a couple weeks since the identified dashboards were transitioned to ECS. At this point, we're ready to fully transition Openstack logs and turn off duplication to the legacy indexes.
To accomplish this, we intend to fully transition OpenStack logs on Thursday, June 24th provided there are no concerns. A note will be added to the legacy dashboards indicating the transition as well as the cutoff date so that prior logs can still be accessed.
Please let us know if you have any questions or concerns.
Thanks! CW
On Tue, Jun 8, 2021 at 5:00 PM Cole White cwhite@wikimedia.org wrote:
Hey Andrew, thanks for the help!
Those dashboards you listed now have ECS equivalents. Please have a look when you have a chance. Note that the ECS-formatted logs are being duplicated and these duplicated logs go back no more than two days.
If those dashboards look good to you, let me know and we'll turn off the duplication. Doing that will fully transition these logs to ECS and lift the two-day retention limit.
Thanks!
On Thu, May 13, 2021 at 1:00 PM Andrew Bogott abogott@wikimedia.org wrote:
On 5/12/21 3:50 PM, Cole White wrote:
Hello!
We (Observability) have identified a few OpenStack syslog producers as prime candidates for migration to the Common Logging Schema. We've prepared a patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/689262 that will duplicate the currently produced OpenStack logs to an ECS-compatible form to demo and prepare ECS-compatible Kibana dashboards.
This is great! I don't think anyone is super attached to the current format of the logs, so anything you want to do to rework them is great. The only thing moderately interesting about openstack logs is that they include a request ID which is passed around and consistent across the different services; it's very useful for tracking particular issues so it should maintain pride of place in whatever post-processed messages we wind up with.
What we need from you:
- A quick review to see if we're missing anything.
Done, I've listed three more services in the patch: glance, trove, barbican
- A list of Kibana dashboards and saved searches that need
translation to use the new format.
There are a lot -- basically one per service x one per deployment. You can find the list by searching for 'OpenStack' and then disregarding the Horizon and haproxy things.
That said, the most important dashboards are:
OpenStack codfw1dev
OpenStack eqiad1
OpenStack Nova codfw1dev Control Services
OpenStack Nova eqiad1 Control Services
OpenStack Nova codfw1dev Compute
OpenStack Nova eqiad1 Compute
If you run out of gas at the end of that list (or even part way through) feel free to drop the cleanup on me and I can decide if the other dashboards are worth updating or can be done away with.
Thanks in advance! And as always, questions are welcome!
Thanks again!
-Andrew
-- Cole White Wikimedia Foundation
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org To unsubscribe send an email to cloud-admin-leave@lists.wikimedia.org
-- Cole White Wikimedia Foundation
-- Cole White Wikimedia Foundation
cloud-admin@lists.wikimedia.org