Differences between revisions 5 and 6
Revision 5 as of 2009-09-21 02:17:38
Size: 4953
Editor: EricEvans
Comment: really fix inline attachment
Revision 6 as of 2013-11-14 23:35:00
Size: 5014
Editor: 107
Comment: statcounter
Deletions are marked like this. Additions are marked like this.
Line 134: Line 134:


Cluster Monitoring and Management w/ OpenNMS


We can break cluster/network management down into a couple of areas:

  1. Real-time monitoring of service availability.
  2. Collection and trending of data to better understand cluster performance.

For (1), we're looking at things like "Are my nodes up? Do they respond to an ICMP ping?", or "Is the thrift service listening? Is it capable of responding to RPC requests?".

With (2) we're interested in collecting and reporting on data that will help us answer questions like, "What is the rate of storage consumption? When will I need to add capacity?" or "At what point does load start to adversely effect read/write latency?"

OpenNMS is a Free Software (GPL) network management platform written in Java. This page will document configuration and best practices when using OpenNMS for monitoring, data-collection, and management of Cassandra clusters.

Note: It is beyond the scope of this document to detail anything already covered in the actual docs.

Disclaimer: This page is a very early draft. No claims are made with respect to accuracy or completeness. Reading this might very well make you dumber. You have been warned.

Service Polling


Data Collection

Capability Detection

If you are using the Cassandra default of 8080 for JMX, then you'll need to comment out the definition for HTTP-8080 (it conflicts).

File: capsd-configuration.xml

<protocol-plugin protocol="JSR160-8080" scan="on" user-defined="false"
    <property key="port" value="8080"/>
    <property key="type" value="default"/>


File: jmx-datacollection-config.xml

<jmx-collection name="JSR160-8080" maxVarsPerPdu = "50">
    <rrd step = "300">

        <mbean name="cf.keyspace1.standard1"
            <attrib alias="ReadLatency" type="gauge" name="ReadLatency"/>
            <attrib alias="WriteLatency" type="gauge" name="WriteLatency"/>
            <attrib alias="PendingTasks" type="gauge" name="PendingTasks"/>
            <attrib alias="ReadCount" type="gauge" name="ReadCount"/>
            <attrib alias="WriteCount" type="gauge" name="WriteCount"/>
            <attrib alias="MemtableSwitchCount" type="gauge"
            <attrib alias="MemtableColumnCount" type="gauge"
            <attrib alias="MemtableDataSize" type="gauge"

File: collectd-configuration.xml

<service name="JSR160-8080" interval="300000" user-defined="false"
    <parameter key="port" value="8080"/>
    <parameter key="protocol" value="rmi"/>
    <parameter key="urlPath" value="/jmxrmi"/>
    <parameter key="collection" value="JSR160-8080"/>
    <parameter key="friendly-name" value="JSR160-8080"/>

<collector service="JSR160-8080"



File: snmp-graph.properties

report.cassandra.cf.latency.name=Keyspace1.Standard1 Latency
report.cassandra.cf.latency.command=--title="Read/write Latency" \
 DEF:readlatency={rrd1}:ReadLatency:AVERAGE \
 DEF:minReadlatency={rrd1}:ReadLatency:MIN \
 DEF:maxReadlatency={rrd1}:ReadLatency:MAX \
 DEF:writelatency={rrd2}:WriteLatency:AVERAGE \
 DEF:minWritelatency={rrd2}:WriteLatency:MIN \
 DEF:maxWritelatency={rrd2}:WriteLatency:MAX \
 LINE2:readlatency#0000ff:"Read latency" \
 GPRINT:readlatency:AVERAGE:"  Avg  \\: %5.2lf %s" \
 GPRINT:minReadlatency:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:maxReadlatency:MAX:"Max  \\: %5.2lf %s\\n" \
 LINE2:writelatency#00ff00:"Write latency" \
 GPRINT:writelatency:AVERAGE:" Avg  \\: %5.2lf %s" \
 GPRINT:minWritelatency:MIN:"Min  \\: %5.2lf %s" \
 GPRINT:maxWritelatency:MAX:"Max  \\: %5.2lf %s\\n"

reports=mib2.HCbits, mib2.bits, mib2.percentdiscards, mib2.percenterrors, \
mib2.discards, mib2.errors, mib2.packets, \
xmp.procs,xmp.filesys,xmp.xmpdstats,xmp.diskstats,xmp.diskkb, \

A sample report.



OpenNMS (last edited 2013-11-14 23:35:00 by 107)