Differences between revisions 1 and 2
Revision 1 as of 2010-02-02 20:29:00
Size: 2776
Editor: cn-sfo1-natout
Revision 2 as of 2010-02-02 20:36:40
Size: 2815
Editor: cn-sfo1-natout
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:

== Overview ==
Line 3: Line 6:
'''HDFS File System Structure''' == HDFS File System Structure ==
Line 20: Line 23:
'''Raw Log Collection and Aggregation Workflow''' == Raw Log Collection and Aggregation Workflow ==
Line 53: Line 56:
'''Log Directories Requiring Cleanup''' == Log Directories Requiring Cleanup ==


This document describes how Chukwa data is stored in HDFS and the processes that act on it.

HDFS File System Structure

The general layout of the Chukwa filesystem is as follows.


Raw Log Collection and Aggregation Workflow

What data is stored where is best described by stepping through the Chukwa workflow.

  1. Collectors write chunks to logs/*.chukwa files until a 64MB chunk size is reached or a given time interval is reached.

    • to: logs/*.chukwa

  2. Collectors close chunks and rename them to *.done

    • from: logs/*.chukwa

    • to: logs/*.done

  3. DemuxManager wakes up every 20 seconds, runs M/R to merges *.done files and moves them.

    • from: logs/*.done

    • to: demuxProcessing/mrInput

    • to: demuxProcessing/mrOutput

    • to: {{{dataSinkArchives/[yyyyMMdd]/*/*.done

  4. PostProcessManager wakes up every few minutes and aggregates, orders and de-dups record files.

    • from: postProcess/demuxOutputDir_*/[clusterName]/[dataType]/[dataType]_[yyyyMMdd]_[HH].R.evt}}}
    • to: repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[mm]/[dataType]_[yyyyMMdd]_[HH]_[N].[N].evt

  5. HourlyChukwaRecordRolling runs M/R jobs at 16 past the hour to group 5 minute logs to hourly.

    • from: repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[mm]/[dataType]_[yyyyMMdd]_[mm].[N].evt

    • to: temp/hourlyRolling/[clusterName]/[dataType]/[yyyyMMdd]

    • to: repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[dataType]_HourlyDone_[yyyyMMdd]_[HH].[N].evt

    • leaves: repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/rotateDone/

  6. DailyChukwaRecordRolling runs M/R jobs at 1:30AM to group hourly logs to daily.

    • from: repos/[clusterName]/[dataType]/[yyyyMMdd]/[HH]/[dataType]_[yyyyMMdd]_[HH].[N].evt

    • to: temp/dailyRolling/[clusterName]/[dataType]/[yyyyMMdd]

    • to: repos/[clusterName]/[dataType]/[yyyyMMdd]/[dataType]_DailyDone_[yyyyMMdd].[N].evt

    • leaves: repos/[clusterName]/[dataType]/[yyyyMMdd]/rotateDone/

  7. ChukwaArchiveManager every half hour or so aggregates and removes dataSinkArchives data using M/R.

    • from: dataSinkArchives/[yyyyMMdd]/*/*.done

    • to: archivesProcessing/mrInput

    • to: archivesProcessing/mrOutput

    • to: finalArchives/[yyyyMMdd]/*/chukwaArchive-part-*

Log Directories Requiring Cleanup

The following directories will grow over time and will need to be periodically pruned:

  • finalArchives/[yyyyMMdd]/*

  • repos/[clusterName]/[dataType]/[yyyyMMdd]/*.evt

Chukwa_Processes_and_Data_Flow (last edited 2010-12-29 05:13:15 by www-900)