FYI, the production log interface Kibana (aka logstash) is switching from
version 3 to 4, which are unfortunately incompatible. That means that you
will need to recreate them. The forwarded mail provides more detail.
Cheers,
Marko
---------- Forwarded message ----------
From: Bryan Davis <bd808(a)wikimedia.org>
Date: 29 June 2016 at 01:28
Subject: [Ops] Saving kibana dashboards before the Kibana4 upgrade
To: ops-l <ops(a)lists.wikimedia.org>
Erik B has been putting in a lot of work to prepare for upgrading the
Elasticsearch and Kibana services in the Logstash cluster. Things are
looking pretty good now for an upgrade sometime next week (2016-07-05
- 2016-07-08) [0].
As announced previously, saved Kibana dashboards are not portable from
our existing Kibana3 to the new Kibana4. The current plan is to copy
over to production all dashboards that have been created at
https://kibana4.wmflabs.org (beta cluster test server). I'll be doing
some work over the next couple of days to make sure the dashboards I
can't live without have been recreated there. I would suggest that
others do the same or ping me to see if I have time to port their
dashboard for them.
Creating a dashboard in Kibana4 is similar to Kibana3, but there is at
least one big difference. The "panels" from Kibana3 have been replaced
with "visualizations" in Kibana4. A visualization exists outside of a
dashboard, may or may not be associated with an existing saved query,
and must be made before it can be added to a dashboard. This takes a
bit of poking at to get comfortable with but all in all it's not a
horrible process. There's a video on elastic.co [1] that may or may
not be helpful in understanding how to port things over.
[0]: https://phabricator.wikimedia.org/T136001
[1]: https://www.elastic.co/blog/recreating-kibana-3-dashboards-in-kibana-4
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA
irc: bd808 v:415.839.6885 x6855
_______________________________________________
Ops mailing list
Ops(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ops
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation
Hello,
There was an outage yesterday that took out the whole SCB cluster~[1]. Due
to the number of services hosted there, this was a user-facing event that
lasted approximately 20 minutes. More notes and thoughts are welcomed!
Cheers,
Marko
[1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20160608-SCB
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation
Hello,
As of recently, there are two new commands that can be used by service
owners on their target nodes to inspect their service.
$ check-<service-name>
Issuing this command will trigger the nagios check which plays your
service's monitoring script.
$ tail-<service-name>
This command gives you the production logs in a human-readable format. I
added these commands to the appropriate documentation section on
Wikitech~[1].
Cheers,
Marko
[1]
https://wikitech.wikimedia.org/wiki/Services/Deployment#Dealing_with_Proble…
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation