I am the operator of i-0000040c.pmtpa.wmflabs, and i-0000040b.pmtpa.wmflabs.<div><br></div><div>I personally am not sure why they are showing as offline as they are not.</div><div><br></div><div>~Jason<br><br><div class="gmail_quote">
On Sun, Sep 30, 2012 at 2:44 PM, 208.97.132.231 <span dir="ltr"><<a href="mailto:damian@damianzaremba.co.uk" target="_blank">damian@damianzaremba.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Complaining bit<br>
==========<br>
Once again I'm trying to clear up monitoring so we can improve it.
The following instances are currently reporting as down (some have
been for quite a while);<br>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:center;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;font-size:16px;white-space:normal;font-family:arial,serif;word-spacing:0px"><br>
* test5 - i-00000026.eqiad.wmflabs<br>
** Ryan - ping/nrpe is restricted from pmtpa to eqiad, is this
intended? Do we want 1 Nagios instance per Region or centralized
monitoring. Not an issue currently but needs deciding/sorting
before bringing the region online.<br>
* wlm-mysql-master - i-0000040c.pmtpa.wmflabs<br>
* wep - i-000000c2.pmtpa.wmflabs<br>
* analytics - i-000000e2.pmtpa.wmflabs<br>
* deployment-backup - i-000000f8.pmtpa.wmflabs<br>
* deployment-feed - i-00000118.pmtpa.wmflabs<br>
* configtest-main - i-000002dd.pmtpa.wmflabs<br>
* deployment-cache-bits02 - i-0000031c.pmtpa.wmflabs<br>
* puppet-abogott - i-00000389.pmtpa.wmflabs<br>
* mobile-wlm2 - i-0000038e.pmtpa.wmflabs<br>
* conventionextension-test - i-000003c0.pmtpa.wmflabs<br>
* lynwood - i-000003e5.pmtpa.wmflabs<br>
* wlm-apache1 - i-0000040b.pmtpa.wmflabs<br>
</span><br>
If any of these are yours could you either;<br>
a) Reply, if it's still pending file recovery from the block storage
migration (these should all be done).<br>
b) Delete, if it's not used and has no plan for being used.<br>
c) Start, if its purpose is to be used and is just stopped from the
last outage (block storage migration).<br>
d) Reply, if it needs to be down for some reason (I'll mark it as
such in monitoring, so it doesn't spam the channel)<br>
e) Reply, if it's online and functioning as expected (there might be
a security group etc issue)<br>
<br>
Current problems with monitoring<br>
=====================<br>
While monitoring everything based on puppet classes makes perfect
sense for production, currently because most things are not
packaged/puppetized and we're half dev and half 'semi-production'
monitoring rather sucks.<br>
<br>
Due to the current state of labs I suggest that we add an attribute
to instance entries in LDAP that allows monitoring to be enabled and
default to not monitoring.<br>
<br>
Now while that may seem silly, currently we can't really enable the
relay bot without flooding the channel with nonsense which makes
monitoring redundant.<br>
<br>
Actually limiting spam to things we care about (public http
instances, mysql servers etc) we can easily see when things are
actually breaking.<br>
<br>
Downsides<br>
---------<br>
We loose a general overview of instances, which causes a more
reactive approach - however we're no so proactive currently.<br>
<br>
Implementation choices<br>
===============<br>
a) Based on puppet classes (current usage)<br>
Pros;<br>
* Monitoring is standard<br>
* Monitoring is automatic<br>
<br>
Cons;<br>
* We're suppose to be developing, not standardizing (at this point)<br>
* Important services get masked by dev instances<br>
* We're not really monitoring services (they're not puppetized, yet)<br>
<br>
b) Based on user input (possibly stored in ldap as an entry under
the host)<br>
Pros;<br>
* People can test/develop monitoring<br>
* We can monitor things not yet puppetized<br>
* We can ignore unimportant things<br>
<br>
Cons;<br>
* Monitoring isn't standard<br>
* Monitoring isn't automatic<br>
* We're breaking from production in style<br>
<br>
While I'd love to spend my time convincing people using puppet is
the way forward, quite frankly the current state of the repo is a
mess. It's partly not usable in labs AFAIK (due to the way
parameters are handled and is general a mix of bad/confusing code
that's whitespace hell.<br>
<br>
As we move over to role classes with parametrized classes in modules
it should be easier and quicker to get changes in.<br>
<br>
Until there is a push monitoring is either mostly redundant or we
can work on improving it. As we have semi-production stuff I think
we should improve it.<br>
<br>
The issues become around if we want to enable user based monitoring
and treat nagios as a dev environment along side puppet classes,
keep puppet classes exclusively, use user input exclusively or split
the usage into 2 and have puppet based for 'production' services and
user based for dev.<br>
<br>
It would be 'easy' to allow 'extra' monitoring data to be specified
on an instances subpage, or even bang it in LDAP - however this
could encourage a path that we don't want.<br>
<br>
Features I'd like to see<br>
==============<br>
* User access to the web interface (ldap authenticated, based on
project membership)<br>
* More extensive monitoring of services (think about the beta
project and how crappy the monitoring for it is currently)<br>
* Optional subscription of alerts on a per project bases (think
about semi-production stuff where it would be nice to get an email
saying it's borked)<br>
* Puppetization of the setup<br>
* Expansion of the groups/templates to include everything in puppet
that's monitored in production (currently it's a very small common
list).<br>
* Grouping based on region<br>
* Grouping based on host (this is currently exposed via labsconsole,
we could scrape it for info or talk to nova directly I guess. Harder
than the above)<br>
* A bot that doesn't die randomly<br>
* A way to shard monitoring (per region) for when we get so many
instances it's not possible to have a single crappy box<br>
<br>
Features I'd be interested in exploring<br>
=======================<br>
* Using saltstack to grab monitoring info (for example puppet last
run time, this can be calculated from a state file and pushed back
to the monitoring instance or polled using minion to minion salt
access). SNMP traps kinda suck, rely on people updating their puppet
clones etc. Without adding sudo access to nrpe and writing a script
for it there's no other way to get root level access to grab the
file data. Extending saltstack (if we do end up using it widely) and
creating a 'feedback loop' would be nice<br>
* Being able to monitor misc data/servers (think labsconsole -
currently things like controllers are monitored on production
Nagios, this data isn't however relayed to #wikimedia-labs or widly
open to the labs community). While monitoring infrastructure from
within its self isn't a good idea generally from a centralized
community point it might be nice.<br>
* Adding other software (Graphite) to the 'common use' 'monitoring
stack'. For example in bots it would be nice to a) monitoring the
processes/random data in nagios but also b) push metrics out and
have historical graphs. Downside is graphite isn't currently
packaged for ubuntu in public repos, it is somewhere for prod
though. Also would need some form of proxy to determine project name
prefix for data coming in.<br>
* Adding a real api to labsconsole to expose the data we have in
there as well as allowing the creation/configuration and deletion of
instances. JSON output of SMW searches rather sucks a little due to
the filtering etc.<br>
* Exposing current status/uptime stats per project and instance on
labsconsole (not sure how easy it would be to transclude this/images
from ganglia). The instance pages are mostly useless and
uninteresting to look at. For example on the beta project it would
be interesting to be able to say 'it's been up 99.98% this month
with a response time of xxxms'. With data we can at least have an
idea when things are going crappy rather than 'it's broken', 'now
it's not'.<br>
<br>
TL;DR<br>
Our monitoring currently sucks, we need to get to a place where
rolling out a cluster based on puppet classes gets auto monitored
but also allow development without masking useful alerts.<br>
<br>
I'm not too sure on the perfect solution right now, however I'd love
some feedback/ideas from everyone else and to publicise what
monitoring we do have generally.<br>
<br>
Damian<br>
</div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><p><i>Thank you for contacting Jason Spriggs.</i></p>
</div>