Carbon Whisper Files (.wsp)

By default, graphite only displays 24 hours worth of data points. Trying to graph metrics older than that does not show up correctly.

The whisper files from carbon (*.wsp) are located in /var/lib/carbon/whisper by default on CentOS when installed via yum. The default settings in /etc/carbon/storage-schemas.conf is a catchall that defaults to a data point every 60 seconds and retains that for 1 day.

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

We will want to override this, say we have collectd metrics being sent to graphite and we want to maintain a data point every 60 seconds but retain it for 30 days. We would add the following ABOVE the aforementioned conf.

[collectd]
pattern = ^collectd.*
retentions = 60s:30d

Now when new whisper database files are created for anything listed in your tree under Graphite > collectd they will properly retain for 30 days. You can see the details on these files in the creates.log.

cat /var/log/carbon/creates.log

Lets say you are just discovering this issue, and did a google search and stumbled on this page. You’ve updated your storage-schemas.conf and still you only see 24 hours available. That’s because these *.wsp files were created with the default settings, and just changing the conf won’t update pre-existing wsp files, only new ones. You can use the whisper-info command to get the details on the wsp files.

whisper-info /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp
 
maxRetention: 86400
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 17308
 
Archive 0
retention: 86400
secondsPerPoint: 60
points: 1440
size: 17280
offset: 28

You can change the retention of this database file with whisper-resize.

whisper-resize /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp 60:30d
Created: /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp.tmp (518428 bytes)
Migrating data without aggregation...
Renaming old database to: /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp.bak
Renaming new database to: /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp
Retrieving all data from the archives

At this point, the db files have been replaced with the new retention, and the previous one is still available with a .bak extension. Let’s use whisper-info to examine the current and previous files.

whisper-info /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp
maxRetention: 2592000
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 518428
 
Archive 0
retention: 2592000
secondsPerPoint: 60
points: 43200
size: 518400
offset: 28
 
whisper-info /var/lib/carbon/whisper/collectd/salt_centoshowtos_org/memory/memory-free.wsp.bak
maxRetention: 86400
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 17308
 
Archive 0
retention: 86400
secondsPerPoint: 60
points: 1440
size: 17280
offset: 28

Here we can see retention has changed from 86400 to 2592000. To adjust all wsp files with whisper-resize, you can put it in a fore loop.

for WSP in `find /var/lib/carbon/whisper/collectd -name *.wsp -type f`; do whisper-resize $WSP 60:30d; done

If you run this as root, be sure to chown carbon:carbon the whisper files or else you’ll see a permission denied error in /var/log/carbon/console.log.

for WSP in `find /var/lib/carbon/whisper/collectd -name *.wsp -type f`; do chown carbon:carbon $WSP; done

Once you confirm everything looks good, you can delete the original files that are saved as *.wsp.bak.

for WSPBAK in `find /var/lib/carbon/whisper/collectd -name *.wsp.bak -type f`; do rm -f $WSPBAK; done
(Comments)

Comments