Pipeline Implementation
The following is a list of items that need to be realized for the new pipeline design(PigExecutionModel):
Implementation of all the Logical Operators
Implementation of the new Logical Plan generation with types
Implementation of all the Physical Operators
Implementation for the translation of Logical Plan into Physical plan
Implementation of the Stagification Process which converts the Physical Plan into multiple Map Reduce stages and submits each one as a separate job to the cluster
The idea for the translation of Logical Plan to Physical Plan is to put a generic infrastructure in place without any ties to the implementation of the operators. Once the infrastructure is in place, we can develop the operators and the stagification in a parallel fashion.
Timelines for each of these items need to include time for writing unit tests.