How to push new information to Chukwa

Add a new dataSource (Source Input)

Using Log4J

Chukwa comes with a Log4J Appender. Here the steps that you need to follow in order to use it:

  1. Create a log4j.properties file that contains the following information:
    log4j.rootLogger=INFO, chukwa
    log4j.appender.chukwa=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
    log4j.appender.chukwa.File=${CHUKWA_HOME}/logs/${RECORD_TYPE}.log
    log4j.appender.chukwa.DatePattern='.'yyyy-MM-dd
    log4j.appender.chukwa.recordType=${RECORD_TYPE}
    log4j.appender.chukwa.layout=org.apache.log4j.PatternLayout
    log4j.appender.chukwa.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
  2. Add these parameters to your java command line:
    • -DCHUKWA_HOME=${CHUKWA_HOME} -DRECORD_TYPE=<YourRecordType_Here> -Dlog4j.configuration=log4j.properties
    • -DRECORD_TYPE=<YourRecordType_Here> is the most important parameter.
    • You can only store one record type per file, so if you need to split your logs into different record types just create one appender per data type (%T% see hadoop logs4j configuration file)
  3. Start your program. Now all your log statements should be written in ${CHUKWA_HOME}/logs/<YourRecordType_Here>.log

Static file like /var/log/messages

  1. Edit ${CHUKWA_HOME}/conf/initial_adaptors
  2. Add a line similar to this one:
    • add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine SysLog 0 /var/log/messages 0

This line will automatically register the "CharFileTailingAdaptorUTF8NewLine" adaptor for /var/log/messages

Register a file from another application/language

  1. Open a socket from your application to the ChukwaLocalAgent
  2. Write this line to the socket
    • Add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine <RecordType> <StartOffset> <fileName> <StartOffset>
    • Where <RecordType> is the data type that will identify your data
    • Where <StartOffset> is the start offset
    • Where <fileName> is the local path on your machine
  3. Close the socket
  • No labels