How to push new information to Chukwa

Add a new dataSource (Source Input)

Using Log4J

Chukwa comes with a Log4J Appender. Here the steps that you need to follow in order to use it:

1. Create a log4j.properties file that contains the following information:
   log4j.rootLogger=INFO, chukwa
   log4j.appender.chukwa=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
   log4j.appender.chukwa.File=${CHUKWA_HOME}/logs/${RECORD_TYPE}.log
   log4j.appender.chukwa.DatePattern='.'yyyy-MM-dd
   log4j.appender.chukwa.recordType=${RECORD_TYPE}
   log4j.appender.chukwa.layout=org.apache.log4j.PatternLayout
   log4j.appender.chukwa.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
   
2. Add these parameters to your java command line:
   • -DCHUKWA_HOME=${CHUKWA_HOME} -DRECORD_TYPE=<YourRecordType_Here> -Dlog4j.configuration=log4j.properties
   • -DRECORD_TYPE=<YourRecordType_Here> is the most important parameter.
   • You can only store one record type per file, so if you need to split your logs into different record types just create one appender per data type (%T% see hadoop logs4j configuration file)
     
3. Start your program. Now all your log statements should be written in ${CHUKWA_HOME}/logs/<YourRecordType_Here>.log
   

Static file like /var/log/messages

1. Edit ${CHUKWA_HOME}/conf/initial_adaptors
2. Add a line similar to this one:
   • add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine SysLog 0 /var/log/messages 0
     

This line will automatically register the "CharFileTailingAdaptorUTF8NewLine" adaptor for /var/log/messages

Register a file from another application/language

1. Open a socket from your application to the ChukwaLocalAgent
2. Write this line to the socket
   • Add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine <RecordType> <StartOffset> <fileName> <StartOffset>
   • Where <RecordType> is the data type that will identify your data
   • Where <StartOffset> is the start offset
   • Where <fileName> is the local path on your machine
3. Close the socket