Add a new dataSource (Source Input)
Using Log4J
Chukwa comes with a Log4J Appender. Here the steps that you need to follow in order to use it:
- Create a log4j.properties file that contains the following information:
- log4j.rootLogger=INFO, chukwa
log4j.appender.chukwa=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppenderlog4j.appender.chukwa.File=${CHUKWA_HOME}/logs/${RECORD_TYPE}.log log4j.appender.chukwa.DatePattern='.'yyyy-MM-dd log4j.appender.chukwa.recordType=${RECORD_TYPE} log4j.appender.chukwa.layout=org.apache.log4j.PatternLayoutlog4j.appender.chukwa.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
- log4j.rootLogger=INFO, chukwa
- Add these parameters to your java command line:
-DCHUKWA_HOME=${CHUKWA_HOME} -DRECORD_TYPE=<YourRecordType_Here> -Dlog4j.configuration=log4j.properties
-DRECORD_TYPE=<YourRecordType_Here> is the most important parameter.
- You can only store one record type per file, so if you need to split your logs into different record types just create one appender per data type (%T% see hadoop logs4j configuration file)
Start your program. Now all your log statements should be written in ${CHUKWA_HOME}/logs/<YourRecordType_Here>.log
Static file like /var/log/messages
- Edit ${CHUKWA_HOME}/conf/initial_adaptors
- Add a line similar to this one:
add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine SysLog 0 /var/log/messages 0
This line will automatically register the "CharFileTailingAdaptorUTF8NewLine" adaptor for /var/log/messages
Register a file from another application/language
Open a socket from your application to the ChukwaLocalAgent
- Write this line to the socket
Add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine <RecordType> <StartOffset> <fileName> <StartOffset>
Where <RecordType> is the data type that will identify your data
Where <StartOffset> is the start offset
Where <fileName> is the local path on your machine
- Close the socket