If you start it using the standard startup script, Cassandra will listen for connections on port 8080 (port 7199 starting in 0.8.0-beta1) to view and tweak variables which it exposes via JMX. This may be helpful for debugging and monitoring. See also JmxGotchas.

The MemtableThresholds page describes how to use Jconsole as a client for this.

org.apache.cassandra.concurrent

All org.apache.cassandra.concurrent Types are thread pool executors which have Attributes ActiveCount, CompletedTasks and PendingTasks. They are viewable from the command line with 'nodeprobe tpstats'.

They are generally defined as follows :

org.apache.cassandra.concurrent.<THREADPOOL>.Attributes.ActiveCount

org.apache.cassandra.concurrent.<THREADPOOL>.Attributes.CompletedTasks

org.apache.cassandra.concurrent.<THREADPOOL>.Attributes.PendingTasks

org.apache.cassandra.concurrent.AE-SERVICE-STAGE

This thread pool is single threaded.

Per src/java/org/apache/cassandra/service/AntiEntropyService.java,

AntiEntropyService encapsulates "validating" (hashing) individual column families, exchanging MerkleTrees with remote nodes via a TreeRequest/Response conversation, and then triggering repairs for disagreeing ranges.

Tree comparison and repair triggering occur in the single threaded AE_SERVICE_STAGE.

org.apache.cassandra.concurrent.CONSISTENCY-MANAGER

This pool has 4 threads, which is hardcoded per src/java/org/apache/cassandra/service/StorageService.java,

    private static int consistencyThreads_ = 4; // not configurable

The CONSISTENCY-MANAGER executor is responsible for passive read repair, for example after a read with ConsistencyLevel.ONE.

Per src/java/org/apache/cassandra/service/StorageService.java,

/* This thread pool does consistency checks when the client doesn't care about consistency */

org.apache.cassandra.concurrent.FILEUTILS-DELETE-POOL

This thread pool is single threaded. Code is defined in src/java/org/apache/cassandra/io/DeletionService.java. This executor is responsible for deleting files, such as obsolete sstables and obsolete commitlog segments.

org.apache.cassandra.concurrent.FLUSH-SORTER-POOL

This thread pool is multi-threaded, with between 1 and Runtime.getRuntime().availableProcessors() threads in the pool.

See Also : FLUSH-WRITER-POOL and MEMTABLE-POST-FLUSHER. Per src/java/org/apache/cassandra/db/ColumnFamilyStore.java,

     * submitFlush first puts [Binary]Memtable.getSortedContents on the flushSorter executor,
     * which then puts the sorted results on the writer executor.  This is because sorting is CPU-bound,
     * and writing is disk-bound; we want to be able to do both at once.  When the write is complete,
     * we turn the writer into an SSTableReader and add it to ssTables_ where it is available for reads.
     *
     * For BinaryMemtable that's about all that happens.  For live Memtables there are two other things
     * that switchMemtable does (which should be the only caller of submitFlush in this case).
     * First, it puts the Memtable into memtablesPendingFlush, where it stays until the flush is complete
     * and it's been added as an SSTableReader to ssTables_.  Second, it adds an entry to commitLogUpdater
     * that waits for the flush to complete, then calls onMemtableFlush.  This allows multiple flushes
     * to happen simultaneously on multicore systems, while still calling onMF in the correct order,
     * which is necessary for replay in case of a restart since CommitLog assumes that when onMF is
     * called, all data up to the given context has been persisted to SSTables.

org.apache.cassandra.concurrent.FLUSH-WRITER-POOL

This thread pool is multi-threaded, with between 1 and Runtime.getRuntime().availableProcessors() threads in the pool.

See Also : FLUSH-SORTER-POOL and MEMTABLE-POST-FLUSHER. Per src/java/org/apache/cassandra/db/ColumnFamilyStore.java,

     * submitFlush first puts [Binary]Memtable.getSortedContents on the flushSorter executor,
     * which then puts the sorted results on the writer executor.  This is because sorting is CPU-bound,
     * and writing is disk-bound; we want to be able to do both at once.  When the write is complete,
     * we turn the writer into an SSTableReader and add it to ssTables_ where it is available for reads.
     *
     * For BinaryMemtable that's about all that happens.  For live Memtables there are two other things
     * that switchMemtable does (which should be the only caller of submitFlush in this case).
     * First, it puts the Memtable into memtablesPendingFlush, where it stays until the flush is complete
     * and it's been added as an SSTableReader to ssTables_.  Second, it adds an entry to commitLogUpdater
     * that waits for the flush to complete, then calls onMemtableFlush.  This allows multiple flushes
     * to happen simultaneously on multicore systems, while still calling onMF in the correct order,
     * which is necessary for replay in case of a restart since CommitLog assumes that when onMF is
     * called, all data up to the given context has been persisted to SSTables.

org.apache.cassandra.concurrent.GMFD

This thread pool is single threaded. Executor for the Gossiper.

org.apache.cassandra.concurrent.HINTED-HANDOFF-POOL

This thread pool is single threaded.

Per src/java/org/apache/cassandra/db/HintedHandOffManager.java,

 * For each table (keyspace), there is a row in the system hints CF.
 * SuperColumns in that row are keys for which we have hinted data.
 * Subcolumns names within that supercolumn are host IPs. Subcolumn values are always empty.
 * Instead, we store the row data "normally" in the application table it belongs in.
 *
 * So when we deliver hints we look up endpoints that need data delivered
 * on a per-key basis, then read that entire row out and send it over.

org.apache.cassandra.concurrent.LB-OPERATIONS

This thread pool is single threaded.

See Also : LB-TARGET. Per src/java/org/apache/cassandra/service/StorageLoadBalancer.java,

/* This thread pool is used for initiating load balancing operations */

org.apache.cassandra.concurrent.LB-TARGET

This thread pool is single threaded.

See Also : LB-OPERATIONS. Per src/java/org/apache/cassandra/service/StorageLoadBalancer.java,

/* This thread pool is used by target node to leave the ring. */

org.apache.cassandra.concurrent.LOAD-BALANCER-STAGE

This thread pool is single threaded.

org.apache.cassandra.concurrent.MEMTABLE-POST-FLUSHER

This thread pool is single threaded.

See Also : FLUSH-SORTER-POOL and FLUSH-WRITER-POOL. Per src/java/org/apache/cassandra/db/ColumnFamilyStore.java,

     * submitFlush first puts [Binary]Memtable.getSortedContents on the flushSorter executor,
     * which then puts the sorted results on the writer executor.  This is because sorting is CPU-bound,
     * and writing is disk-bound; we want to be able to do both at once.  When the write is complete,
     * we turn the writer into an SSTableReader and add it to ssTables_ where it is available for reads.
     *
     * For BinaryMemtable that's about all that happens.  For live Memtables there are two other things
     * that switchMemtable does (which should be the only caller of submitFlush in this case).
     * First, it puts the Memtable into memtablesPendingFlush, where it stays until the flush is complete
     * and it's been added as an SSTableReader to ssTables_.  Second, it adds an entry to commitLogUpdater
     * that waits for the flush to complete, then calls onMemtableFlush.  This allows multiple flushes
     * to happen simultaneously on multicore systems, while still calling onMF in the correct order,
     * which is necessary for replay in case of a restart since CommitLog assumes that when onMF is
     * called, all data up to the given context has been persisted to SSTables.

org.apache.cassandra.concurrent.MESSAGE-DESERIALIZER-POOL

This thread pool is multi-threaded, with between 1 and Runtime.getRuntime().availableProcessors() threads in the pool.

Per src/java/org/apache/cassandra/net/MessagingService.java,

// read executor puts messages to deserialize on this.

org.apache.cassandra.concurrent.MESSAGE-STREAMING-POOL

This thread pool is single threaded.

org.apache.cassandra.concurrent.RESPONSE-STAGE

This thread pool is mutli-threaded and has between 1 and Runtime.getRuntime().availableProcessors() threads in the pool.

org.apache.cassandra.concurrent.ROW-MUTATION-STAGE

This thread pool is multi-threaded and has between 1 and concurrectWriters threads in the pool.

Per src/java/org/apache/cassandra/concurrent/StageManager.java,

    private static int concurrentWriters_ = 32;

org.apache.cassandra.concurrent.ROW-READ-STAGE

This thread pool is multi-threaded and has between 1 and concurrectReaders threads in the pool.

Per src/java/org/apache/cassandra/concurrent/StageManager.java,

    private static int concurrentReaders_ = 8;

org.apache.cassandra.concurrent.STREAM-STAGE

This stage is single threaded.

org.apache.cassandra.db

org.apache.cassandra.db.CommitLog

org.apache.cassandra.db.CommitLog.Attributes.ActiveCount

org.apache.cassandra.db.CommitLog.Attributes.CompletedTasks

org.apache.cassandra.db.CommitLog.Attributes.PendingTasks

org.apache.cassandra.db.CompactionManager

org.apache.cassandra.db.CompactionManager.Attributes.BytesCompacted

org.apache.cassandra.db.CompactionManager.Attributes.BytesTotalInProgress

org.apache.cassandra.db.CompactionManager.Attributes.ColumnFamilyInProgress

org.apache.cassandra.db.CompactionManager.Attributes.MaximumCompactionThreshold

org.apache.cassandra.db.CompactionManager.Attributes.MinimumCompactionThreshold

org.apache.cassandra.db.CompactionManager.Attributes.PendingTasks

org.apache.cassandra.gms

org.apache.cassandra.gms.FailureDetector

org.apache.cassandra.gms.FailureDetector.Operations.dumpInterArrivalTimes

Dump endpoint arrival windows to a file in /var/tmp, per src/java/org/apache/cassandra/gms/FailureDetector.java:

FileOutputStream fos = new FileOutputStream("/var/tmp/output-" + System.currentTimeMillis() + ".dat", true);

org.apache.cassandra.service

org.apache.cassandra.service.StorageService

org.apache.cassandra.service.StorageService.Attributes.CurrentGenerationNumber

org.apache.cassandra.service.StorageService.Attributes.LiveNodes

org.apache.cassandra.service.StorageService.Attributes.LoadMap

org.apache.cassandra.service.StorageService.Attributes.LoadString

org.apache.cassandra.service.StorageService.Attributes.OperationMode

org.apache.cassandra.service.StorageService.Attributes.Token

org.apache.cassandra.service.StorageService.Attributes.UnreachableNodes

org.apache.cassandra.service.StorageService.Operations.clearSnapshot

Per src/java/org/apache/cassandra/db/Table.java :

            String snapshotPath = dataDirPath + File.separator + name + File.separator + SNAPSHOT_SUBDIR_NAME;
...
                FileUtils.deleteDir(snapshotDir);

org.apache.cassandra.service.StorageService.Operations.decommission

This operation may fail if :

• The node is not a member of the token ring yet.
• There are no other normal nodes in the ring.
• Data is currently moving to the node.

org.apache.cassandra.service.StorageService.Operations.forceTableCleanup

org.apache.cassandra.service.StorageService.Operations.forceTableCompaction

org.apache.cassandra.service.StorageService.Operations.forceTableFlush

org.apache.cassandra.service.StorageService.Operations.forceTableRepair

org.apache.cassandra.service.StorageService.Operations.getLiveNaturalEndpoints

org.apache.cassandra.service.StorageService.Operations.getNaturalEndpoints

org.apache.cassandra.service.StorageService.Operations.getRangeToEndPointMap

org.apache.cassandra.service.StorageService.Operations.loadBalance

org.apache.cassandra.service.StorageService.Operations.move

This operation may fail if :

• The target token is already owned by another node.
• Data is currently moving to this node.

org.apache.cassandra.service.StorageService.Operations.removeToken

This operation may fail if :

• There is a live node which owns this token.

org.apache.cassandra.service.StorageService.Operations.setLog4jLevel

org.apache.cassandra.service.StorageService.Operations.takeAllSnapshot

org.apache.cassandra.service.StorageService.Operations.takeSnapshot

org.apache.cassandra.service.StorageProxy

org.apache.cassandra.service.StorageProxy.Attributes.RangeOperations

org.apache.cassandra.service.StorageProxy.Attributes.ReadOperations

org.apache.cassandra.service.StorageProxy.Attributes.RecentRangeLatencyMicros

See Also : RecentReadLatencyMicros, RecentWriteLatencyMicros.

Per src/java/org/apache/cassandra/utils/LatencyTracker.java/getRecentLatencyMicros :

        long ops = opCount.get();
        long n = totalLatency.get();        
        return ((double)n - lastLatency) / (ops - lastOpCount);
        ...
        lastLatency = n;
        lastOpCount = ops;

org.apache.cassandra.service.StorageProxy.Attributes.RecentReadLatencyMicros

See Also RecentRangeLatencyMicros, RecentWriteLatencyMicros.

org.apache.cassandra.service.StorageProxy.Attributes.RecentWriteLatencyMicros

See Also : RecentRangeLatencyMicros, RecentReadLatencyMicros.

org.apache.cassandra.service.StorageProxy.Attributes.TotalRangeLatencyMicros

org.apache.cassandra.service.StorageProxy.Attributes.TotalReadLatencyMicros

org.apache.cassandra.service.StorageProxy.Attributes.TotalWriteLatencyMicros

org.apache.cassandra.service.StorageProxy.Attributes.WriteOperations

org.apache.cassandra.streaming

org.apache.cassandra.streaming.StreamingService

org.apache.cassandra.streaming.StreamingService.Attributes.Status

org.apache.cassandra.streaming.StreamingService.Attributes.StreamDestinations

org.apache.cassandra.streaming.StreamingService.Attributes.StreamSources

org.apache.cassandra.streaming.StreamingService.Operations.getIncomingFiles

See Also : getOutgoingFiles.

Per src/java/org/apache/cassandra/streaming/StreamingService.java :

            files.add(String.format("%s %d/%d", f.getFilename(), f.getPtr(), f.getExpectedBytes()));

org.apache.cassandra.streaming.StreamingService.Operations.getOutgoingFiles

See Also : getIncomingFiles.