Pipeline Implementation
The following is a list of items that need to be realized for the new pipeline design(PigExecutionModel):
- Implementation of all the Logical Operators
- Implementation of the new Logical Plan generation with types
- Implementation of all the Physical Operators
- Implementation for the translation of Logical Plan into Physical plan
- Implementation of the Stagification Process which converts the Physical Plan into multiple Map Reduce stages and submits each one as a separate job to the cluster
The idea for the translation of Logical Plan to Physical Plan is to put a generic infrastructure in place without any ties to the implementation of the operators. Once the infrastructure is in place, we can develop the operators and the stagification in a parallel fashion.
Timelines for each of these items need to include time for writing unit tests.