Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
The CompiledPlan feature, introduced in FLIP-190: Support Version Upgrades for Table API & SQL Programs, supports only streaming execution mode. Batch ExecNodes were explicitly excluded from the JSON plan (aka CompiledPlan).
The current proposal aims to achieve uniformity of representation for Batch and Streaming execution mode, and to allow serialization (and deserialization) for Batch ExecNodes in CompiledPlan.
Why should Flink support this?
- Uniformity of approach to all exec nodes allows to keep “middle-tier” parts, that are on the path of query serving, simpler.
- Engineers, tools and components would be able to follow the same processes for both streaming and batch queries.
- If some component needs to distinguish batch v.s. streaming in the CompiledPlan, a separate flag could be passed around, or this could be derived from the CompiledPlan itself.
- In production deployments, query processing front-end (parser, planner) could live in dedicated services. CompiledPlan allows to decouple tight version coupling between these services and the actual Flink cluster that would be executing a job prepared earlier. This simplifies deployment management.
- Enhanced maintainability and support. Future improvements to the infra and components that are dealing with CompiledPlan would be beneficial for both execution modes.
Public Interfaces
No changes to the public interfaces
Proposed Changes
The proposal expects to piggy-back on the general compiled plan infra introduced in the FLIP-190 and thereafter enhanced.
The changes are:
- Allow compiled plan operations in the BatchPlanner:
- compilePlan
- translatePlan
- loadPlan
- Annotations @ExecNodeMetadata, @JsonCreator, and @JsonProperty are added to BatchExec* classes.
- Once all BatchExec* classes are supported, a pre-submit validation is added in place to enforce CompiledPlan support for all newly added BatchExec* classes.
Compatibility, Deprecation, and Migration Plan
No plans to deprecate.
Test Plan
CompiledPlan testing infra to be extended to include corresponding tests for all BatchExec* operators.
Similar to *RestoreTest.java test in here:
Rejected Alternatives
No comparable alternatives.