Graphite

Installing graphite on a CentOS server.

Graphite is a scalable real-time metrics graphing system, that is sometimes used in lieu of other systems like cacti, munin, collectd, or zabbix.

Getting the software installed on CentOS is fairly straightforward with RPM packages available in the EPEL repository.

Install graphite software and MySQL for backend

yum install graphite-web graphite-web-selinux mysql mysql-server MySQL-python

Start MySQL, lock it down, and create a Graphite database and user.

/etc/init.d/mysqld start
mysql_secure_installation
# Create graphite db, username and password
mysql -e "CREATE DATABASE graphite;" -u root -p
mysql -e "GRANT ALL PRIVILEGES ON graphite.* TO 'graphite'@'localhost' IDENTIFIED BY 'graphitePW01Vxzsigavms';" -u root -p
mysql -e 'FLUSH PRIVILEGES;' -u root -p

Configure Graphite to talk to this DB

The default install of graphite uses sqlite as a backend database, which is fine for testing purposes, but we’re going to use the MySQL database setup earlier.

Next, edit the local_settings.py file for graphite.

vi /etc/graphite-web/local_settings.py

Change the database configuration to look like this:

DATABASES = {
'default': {
'NAME': 'graphite',
'ENGINE': 'django.db.backends.mysql',
'USER': 'graphite',
'PASSWORD': 'graphitePW01Vxzsigavms',
}
}

Run the manage script for graphite to create the MySQL tables

/usr/lib/python2.6/site-packages/graphite/manage.py syncdb

It will prompt you to create a superuser, you can go ahead and do so.

Start apache and then you can browse to it

/etc/init.d/httpd start

Browse to http://IP address or hostname

That’s it, you’ve now got graphite installed and running. Next step is to start gathering statistical data, with carbon and sending it to graphite, and write it to disk with whisper.

Install Carbon and Whisper

yum install python-carbon python-whisper

Start Carbon Cache Service

/etc/init.d/carbon-cache start

We will need to configure the storage-schemas.conf in /etc/carbon/. We’ll start with a simple server load configuration taken from the graphite docs.

To summarize what’s there, you need to pick a good naming scheme, and configure retention policies.

vi /etc/carbon/storage-schemas.conf

Now we’ll add the following for server load.

[server_load]
priority = 100
pattern = ^servers\.
retentions = 60:43200,900:350400

After getting the storage schemas setup, we’ll need to write a client that will actually populate graphite with data. An example client is included with graphite web to reference.

/usr/share/doc/graphite-web-0.9.12/example-client.py

If we simply run it, we should see some output similar to this, and you will see a new folder on the graphite web interface for system load.

python /usr/share/doc/graphite-web-0.9.12/example-client.py
sending message

--------------------------------------------------------------------------------
system.loadavg_1min 0.00 1399300851
system.loadavg_5min 0.00 1399300851
system.loadavg_15min 0.00 1399300851

This should get your started installing and configuring Graphite and the related components on CentOS. For further configuration information, check out the graphite documents.

If you want to dissect the script a little bit, open it up and read it. The platform.system() command will return “Linux” in CentOS so it will get the data by running the open(‘/proc/loadavg’) command. Lets manually create this loop in a python terminal and watch it, then generate load on the server to watch the load values fluctuate.

From the terminal just type “python” to bring up the python shell, and then type the following while loop.

python
>>> import time
>>> while True:
... open('/proc/loadavg').read().strip().split()[:3]
... time.sleep(10)
...
['0.23', '0.11', '0.04']
['0.19', '0.11', '0.03']
['0.16', '0.10', '0.03']
['0.14', '0.10', '0.03']
['0.11', '0.10', '0.03']
['0.10', '0.09', '0.03']
['0.16', '0.10', '0.03']
['0.14', '0.10', '0.03']
['0.11', '0.10', '0.03']
['0.18', '0.11', '0.04']

You will see output similar to above, however your system is probably idle if it’s new, and you’ll just se 0.00 across the board. So lets open a second terminal or connection to the server, and generate an ssh key then delete it in a loop to put a little load on the server.

while `true`; do ssh-keygen -b 4096 -N '' -f /root/id_rsa -t rsa -q ; rm -f /root/id_rsa; sleep 3; done

While running this, you should see the values in your python script start to increase. There are some additional tools that can be used to gather data and send it to graphite which will be covered at a later time.

If you’re running into problems, the logs can be useful so check them out.

/var/log/carbon/*.log
/var/log/graphite-web/*.log

Bucky

While it’s good to know how to create a custom script for gathering data and sending it to graphite, there are some canned tools for common things like system statistics. Python bucky acts as a layer between collectd and graphite.

Install Bucky

yum -y install python-bucky

Collectd 5 Writes directly to Graphite

Although not yet in the official CentOS repos, the latest version of collectd (version 5 and above) can write directly to graphite without the need for bucky in between. It can be compiled from source, and I have packaged an RPM for it that installs to /opt/collectd5 which I will make available at some point.

(Comments)

Comments