This post is a revisit of a topic I have already blogged about, Who monitors the monitor?
Meta monitoring -> frequently compare an inventory with your monitoring configuration.
In my terminology I call this meta monitoring since it is not actively monitoring a business function or the functionality of an infrastructure item. By using meta monitoring I am making myself aware of the completeness of my monitoring. Meta monitoring should give an answer to the question: Is there is something I am missing?
Well, as most of you will say; we always miss something. I agree. But with meta monitoring, we will aim to limit the unknown to a bare minimum. If you don’t do it, your configuration will be hopelessly out of date within days.
There are plenty of tools on the market that will help you make inventories of more or less every aspect of your infrastructure. They are usually very expensive. And, honestly, to do this yourself is not even hard.
- Get a list of items from your current infrastructure (may it be vCenter or Amazon Cloud)
- Remove items that you know should not be monitored
- Compare this list with your monitoring system.
In an OP5 environment, you can even do this in a “one-liner”, for example:
root@op5-system:~# echo mysql-v001fry magnus monitor synology03 | sed -e 's/ /n/g' | grep -wv "$(printf "GET hostsnColumns:namen" | unixcat /opt/monitor/var/rw/live)" | xargs -I"{}" echo "Host "{}" is not configured in OP5"
Host "magnus" is not configured in OP5
Host "synology03" is not configured in OP5
Now, this one-liner is listing all configured hosts in your monitoring environment and using that list to filter away known (monitored) hosts from the list you echo, so it is probably not the most effective way to do it, but it works. See it as an example of how easy it can be.
Now, when it comes to gathering the complete (or partial) inventory from your infrastructure is also not that hard. In it’s simplest form you just copy/paste from your favorite excel sheet, or you request it from your infrastructure through an API. Amazon EC2 has a very powerful API. Just create a read-only user with access to your environment, and use a simple ruby script to get the names from EC2. Note that you need to point out which region you would like to list, and optionally add your proxy URI to the script below.
Example:
#!/usr/bin/ruby
%w[ rubygems aws-sdk ].each { |f| require f }
aws_api = AWS::EC2.new(:access_key_id => 'YOUR_ACCESS_KEY', :secret_access_key => 'YOUR_SECRET_KEY', :region=>'us-west-2' , :proxy_uri => '')
aws_api.client.describe_instances[:reservation_set].each do | instance |
instance[:instances_set][0][:tag_set].each do | tag |
puts tag[:value] if tag[:key] == 'Name'
end
end
Running this script will give you a list of your instances in Amazon EC2. I called this script “listEC2Instances.minimal.rb” and put it together with my one-liner:
root@op5-system:/opt/kmg/ec2/bin# ./listEC2Instances.minimal.rb
vpn-v001ec2
kmg-test002ec2
root@op5-system:/opt/kmg/ec2/bin# ./listEC2Instances.minimal.rb | sed -e 's/ /n/g' | grep -wv "$(printf "GET hostsnColumns:namen" | unixcat /opt/monitor/var/rw/live)" | xargs -I"{}" echo "Host "{}" is not configured in OP5"
Host "vpn-v001ec2" is not configured in OP5
Now, you know which hosts in your Amazon Cloud that are not monitored. Do something about it! =)