Chukwa Test Plan

Unit Test

• Chunk Implementation Test
• Chunk Builder Test
• HDFS Usage Plugin Test
• Test Log Rotate
• Test File Tailing Adaptor
• Test File Expiration Policy
• Test File Tailing Adaptor Big Record
• Test UTF8 File Tailing Adaptor
• Test Exec Adaptor
• Chukwa Test Adaptor
• Test Client Acknowledgement
• Test Failed Collector
• Test Retry List Of Collectors
• Collector Test
• Test Chart Object
• Test Database Insertion Speed
• Test Database Query Speed
• Test Trapazoid calculator
• Test Database Driver Manager
• Test XSS Filter
• Test Demux
• Test PS Demux Output
• Test Hadoop Log Demux Output
  

Automated Integration Tests

• Agent To Collector
  • HTTP POST to unknown server
    
• Collector To Data Processors
  • HDFS file sink to unknown server
    
• Data Processors To Database
  • Database Loader with unknown sequence file
  • Database Loader with unknown hdfs namenode address
    
• Database To HICC
  • Database query for 1 hour data in the past 3 months.
  • Database query for the past 3 hours.
  • Database query for cross partition union.
    
• Check HICC graphs
  • Cluster System Metrics
    • CPU idle percentage
    • CPU nice percentage
    • CPU system percentage
    • CPU user percentage
    • CPU hardware interrupt percentage
    • CPU software interrupt percentage
    • Iowait percentage
    • Memory buffer percentage
    • Memory used percentage
    • Network card 1 busy percentage
    • Network card 2 busy percentage
    • Disk 1 busy percentage
    • Disk 2 busy percentage
    • Disk 3 busy percentage
    • Disk 4 busy percentage
    • Swap used percentage
    • Number of reporting hosts
    • System load 15 minutes average
    • System load 5 minutes average
    • System load 1 minutes average
    • Task total
    • Task running
    • Task sleep
    • Task stopped
    • Task zombie
    • Memory total
    • Memory buffers
    • Memory cached
    • Memory free
    • Network card 1 received error per second
    • Network card 1 received bytes per second
    • Network card 1 received dropped per second
    • Network card 1 received packets per second
    • Network card 2 received error per second
    • Network card 2 received bytes per second
    • Network card 2 received dropped per second
    • Network card 2 received packets per second
    • Disk 1 read kilobytes per second
    • Disk 1 read kilobytes per second
    • Disk 2 read kilobytes per second
    • Disk 2 read kilobytes per second
    • Disk 3 read kilobytes per second
    • Disk 3 read kilobytes per second
    • Disk 4 read kilobytes per second
    • Disk 4 read kilobytes per second
      
  • System Metrics
    • CPU idle percentage
    • CPU nice percentage
    • CPU system percentage
    • CPU user percentage
    • CPU hardware interrupt percentage
    • CPU software interrupt percentage
    • Iowait percentage
    • Memory buffer percentage
    • Memory used percentage
    • Network card 1 busy percentage
    • Network card 2 busy percentage
    • Disk 1 busy percentage
    • Disk 2 busy percentage
    • Disk 3 busy percentage
    • Disk 4 busy percentage
    • Swap used percentage
    • Number of reporting hosts
    • System load 15 minutes average
    • System load 5 minutes average
    • System load 1 minutes average
    • Task total
    • Task running
    • Task sleep
    • Task stopped
    • Task zombie
    • Memory total
    • Memory buffers
    • Memory cached
    • Memory free
    • Network card 1 received error per second
    • Network card 1 received bytes per second
    • Network card 1 received dropped per second
    • Network card 1 received packets per second
    • Network card 2 received error per second
    • Network card 2 received bytes per second
    • Network card 2 received dropped per second
    • Network card 2 received packets per second
    • Disk 1 read kilobytes per second
    • Disk 1 read kilobytes per second
    • Disk 2 read kilobytes per second
    • Disk 2 read kilobytes per second
    • Disk 3 read kilobytes per second
    • Disk 3 read kilobytes per second
    • Disk 4 read kilobytes per second
    • Disk 4 read kilobytes per second
      
  • HDFS FS Name System Metrics
    • Blocks total
    • Capacity remaining gigabytes
    • Capacity used gigabytes
    • Files total
    • Pending replication blocks
    • Scheduled replication blocks
    • Total load
    • Under replication blocks
      
  • HDFS Throughput Metrics
    • Reporting hosts
    • Number of operation for block reports per minute
    • Block verification failures
    • Blocks read per minute
    • Blocks removed per minute
    • Blocks replicated per minute
    • Blocks verified per minute
    • Blocks written per minute
    • Bytes read per minute
    • Bytes written per minute
    • Number of copy block operation per minute
    • Number of heartbeat operation
    • Number of read block operation
    • Number of read meta data operation
    • Number of reads from local client
    • Number of reads from remote client
    • Number of replace block operation
    • Number of write block operation
    • Writes from local client
    • Writes from remote client
    • Block reports average time
    • Copy block operation average time
    • Heart beats average time
    • Read block operation average time
    • Read meta data operation average time
    • Replace block operation average time
    • Write block operation average time
      
  • HDFS Name Node Metrics
    • Number of add block operation
    • Number of create file operation
    • Number of delete file operation
    • Files created
    • Files renamed
    • Files deleted
    • Get block locations
    • Get listing operation
    • Number of syncs operation
    • Number of transactions
    • Number of block report
    • Blocks corrupted
    • Safe mode time
    • Syncs average time
    • Transaction average time
    • Block report average time
    • File system image load time
      
  • HDFS Data Node Metrics
    • Number of operation for block reports
    • Number of block verification failures
    • Blocks read
    • Blocks removed
    • Blocks replicated
    • Blocks verified
    • Blocks written
    • Bytes read
    • Bytes written
    • Number of copy block operation
    • Number of heart beats
    • Number of read block operation
    • Number of read meta data operations
    • Reads from local client
    • Reads from remote client
    • Number of replace block operation
    • Number of write block operation
    • Writes from local client
    • Writes from remote client
      
  • Map Reduce Job Metrics
    • Jobs completed
    • Jobs submitted
    • Maps completed
    • Maps launched
    • Reduce completed
    • Reduce launched
      
  • JVM Metrics
    • NameNode
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
    • DataNode
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
    • Job Tracker
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
    • Task Tracker
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
    • Map
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
    • Shuffle
      • Memory heap committed
      • Memory heap used
      • Memory non heap committed
      • Memory non heap used
      • threads blocked
      • threads new
      • threads runnable
      • threads terminated
      • threads timed waiting
      • threads waiting
        
  • Usage Report
    • HDFS usage by queue
    • Map reduce slot hour by queue
    • HDFS usage by user
    • Map reduce slot hour by user
      

Manual Test

• Upgrade, service restart during upgrade
• Restart HDFS cluster
• Restart Map/Reduce cluster
• Restart Database