Differences between revisions 3 and 4
Revision 3 as of 2008-02-14 12:38:16
Size: 900
Editor: socks1
Revision 4 as of 2009-09-20 23:38:37
Size: 900
Editor: localhost
Comment: converted to 1.6 markup
No differences found!

Pipeline Implementation

The following is a list of items that need to be realized for the new pipeline design(PigExecutionModel):

  • Implementation of all the Logical Operators
  • Implementation of the new Logical Plan generation with types
  • Implementation of all the Physical Operators
  • Implementation for the translation of Logical Plan into Physical plan
  • Implementation of the Stagification Process which converts the Physical Plan into multiple Map Reduce stages and submits each one as a separate job to the cluster

The idea for the translation of Logical Plan to Physical Plan is to put a generic infrastructure in place without any ties to the implementation of the operators. Once the infrastructure is in place, we can develop the operators and the stagification in a parallel fashion.

Timelines for each of these items need to include time for writing unit tests.

PipelineImpl (last edited 2009-09-20 23:38:37 by localhost)