This page describes the Flink Improvement Proposal (FLIP) process for proposing a major change to Flink.
Purpose
The purpose of FLIPs is to have a central place to collect and document planned major enhancements to Apache Flink. While JIRA is still the tool to track tasks, bugs, and progress, the FLIPs give an accessible high level overview of the result of design discussions and proposals. Think of FLIPs as collections of major design documents for user-relevant changes.
We want to make Flink a core architectural component for users. We also support a large number of integrations with other tools, systems, and clients. Keeping this kind of usage healthy requires a high level of compatibility between releases — core architectural elements can't break compatibility or shift functionality from release to release. As a result each new major feature or public API has to be done in a way that we can stick with it going forward.
This means when making this kind of change we need to think through what we are doing as best we can prior to release. And as we go forward we need to stick to our decisions as much as possible. All technical decisions have pros and cons so it is important we capture the thought process that lead to a decision or design to avoid flip-flopping needlessly.
Hopefully we can make these proportional in effort to their magnitude — small changes should just need a couple brief paragraphs, whereas large changes need detailed design discussions.
This process also isn't meant to discourage incompatible changes — proposing an incompatible change is totally legitimate. Sometimes we will have made a mistake and the best path forward is a clean break that cleans things up and gives us a good foundation going forward. Rather this is intended to avoid accidentally introducing half thought-out interfaces and protocols that cause needless heartburn when changed. Likewise the definition of "compatible" is itself squishy: small details like which errors are thrown when are clearly part of the contract but may need to change in some circumstances, likewise performance isn't part of the public contract but dramatic changes may break use cases. So we just need to use good judgement about how big the impact of an incompatibility will be and how big the payoff is.
What is considered a "major change" that needs a FLIP?
Any of the following should be considered a major change:
- Any major new feature, subsystem, or piece of functionality
- Any change that impacts the public interfaces of the project
What are the "public interfaces" of the project?
All of the following are public interfaces that people build around:
- DataStream, DataSet, SQL and Table API, including classes related to that, such as StreamExecutionEnvironment
- Classes marked with the @Public annotation
- On-disk binary formats, such as checkpoints/savepoints
- User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts, Kubernetes scripts
- Configuration settings
- Exposed monitoring information
Not all compatibility commitments are the same. We need to spend significantly more time on public APIs as these can break code for users. They cause people to rebuild code and lead to compatibility issues in large multi-dependency projects (which end up requiring multiple incompatible versions). Configuration, monitoring, and command line tools can be faster and looser — changes here will break monitoring dashboards and require a bit of care during upgrades but aren't a huge burden.
For the most part monitoring, command line tool changes, and configs are added with new features so these can be done with a single FLIP.
What should be included in a FLIP?
A FLIP should contain the following sections:
- Motivation: describe the problem to be solved
- Proposed Change: describe the new thing you want to do. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences, depending on the scope of the change.
- New or Changed Public Interfaces: impact to any of the "compatibility commitments" described above. We want to call these out in particular so everyone thinks about them.
- Migration Plan and Compatibility: if this feature requires additional support for a no-downtime upgrade describe how that will work
- Rejected Alternatives: What are the other alternatives you considered and why are they worse? The goal of this section is to help people understand why this is the best solution now, and also to prevent churn in the future when old alternatives are reconsidered.
Who should initiate the FLIP?
Anyone can initiate a FLIP but you shouldn't do it unless you have an intention of getting the work done to implement it (otherwise it is silly).
Create your Own FLIP
- If you are an Apache Flink committer, create a page which is a child of this one. You can do that by either clicking on "Create" in the header and choose "FLIP-Template" (and not "Blank page") to create your own FLIP. Take the next available FLIP number (see under "FLIP round-up" and give your proposal a descriptive heading. e.g. "FLIP 42: Enable Flink Streaming Jobs to stop gracefully".
- If you don't have the necessary permissions for creating a new page, please create a Google Doc and make that view-only. As a FLIP number, please use FLIP-XXX.
- Post that Google Doc to the mailing list for a discussion thread. When the discussions have been resolved, the contributor ask on the Dev mailing list to a committer/PMC to copy the contents from the Google Doc, and create a FLIP number for them. The contributor can then use that FLIP
to actually have a VOTE thread.
- Post that Google Doc to the mailing list for a discussion thread. When the discussions have been resolved, the contributor ask on the Dev mailing list to a committer/PMC to copy the contents from the Google Doc, and create a FLIP number for them. The contributor can then use that FLIP
Process
Here is the process for making a FLIP:
- Follow the instructions at "Create your Own FLIP".
- Fill in the sections as described above
- Start a [DISCUSS] thread on the Apache mailing list. Please ensure that the subject of the thread is of the format [DISCUSS] FLIP-{your FLIP number} {your FLIP heading} The discussion should happen on the mailing list not on the wiki since the wiki comment system doesn't work well for larger discussions. In the process of the discussion you may update the proposal. You should let people know the changes you are making. You either include the link to the FLIP page on Confluence, or you link to the view-only Google Doc.
- Once the proposal is finalized and there are no more open discussions
- If your FLIP is already in Confluence, proceed to step 5.
- If your FLIP is a Google Doc, please ask on the Dev mailing list to copy the contents from your Google Doc to a FLIP page, and create a FLIP number for you before proceeding to step 5.
- call a [VOTE] to have the proposal adopted. These proposals are more serious than code changes and more serious even than release votes. The criteria for acceptance is consensus.
- Please update the FLIP wiki page to reflect the current stage of the FLIP after a vote. This acts as the permanent record indicating the result of the FLIP (e.g., Accepted or Rejected). Also report the result of the FLIP vote to the voting thread on the mailing list so the conclusion is clear.
It's not unusual for a FLIP proposal to take long discussions to be finalized. Below are some general suggestions on driving FLIPs towards consensus. Notice that these are hints rather than rules. Contributors should make pragmatic decisions in accordance with individual situations.
- The progress of a FLIP should not be long blocked on an unresponsive reviewer. A reviewer who blocks a FLIP with dissenting opinions should try to respond to the subsequent replies timely, or at least provide a reasonable estimated time to respond.
- A typical reasonable time to wait for responses is 1 week, but be pragmatic about it. Also, it would be considerate to wait longer during holiday seasons (e.g., Christmas, Chinese New Year, etc.).
- We encourage FLIP proposers to actively reach out to the interested parties (e.g., previous contributors of the relevant part) early. It helps expose and address the potential dissenting opinions early, and also leaves more time for other parties to respond while the proposer works on the FLIP.
- Committers should use their veto rights with care. According to the ASF policy, vetos must be provided with a technical justification showing why the change is bad. They should not be used for simply blocking the process so the voter has more time to catch up.
FLIP round-up
Next FLIP Number: 455
Use this number as the identifier for your FLIP and increment this value.
Under discussion
Accepted
Title Release JIRA FLIP-36: Support Interactive Programming in Flink FLINK-19343
-
FLIP-36: Support Interactive Programming in Flink
Open
FLIP-37: Rework of the Table API Type System FLINK-12251
-
Rework the Table API & SQL type system
Open
FLIP-42: Rework Flink Documentation 1.3 FLINK-12639
-
FLIP-42: Rework Documentation
In Progress
FLIP-50: Spill-able Heap Keyed State Backend FLINK-12692
-
Support disk spilling in HeapKeyedStateBackend
Open
FLIP-51: Rework of the Expression Design FLINK-13773
-
Rework of the Expression Design
Closed
FLIP-65: New type inference for Table API UDFs FLINK-13191
-
Add a simplified UDF type extraction
Open
FLIP-79: Function DDL Support 1.10 FLINK-7151
-
Support function DDL
Closed
FLIP-83: Flink End-to-end Performance Testing Framework FLINK-14917
-
Add end to end performance testing framework
Open
FLIP-91: Support SQL Gateway 1.16 FLINK-15472
-
[FLIP-91] Support SQL Gateway
Closed
FLIP-93: JDBC catalog and Postgres catalog FLINK-15350
-
develop JDBC catalogs to connect to relational databases
Open
FLIP-95: New TableSource and TableSink interfaces FLINK-16987
-
FLIP-95: Add new table source and sink interfaces
Open
FLIP-100: Add Attempt Information FLINK-16050
-
Add Attempt Information
Open
FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API) FLINK-19153
-
FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)
Open
FLIP-151: Incremental snapshots for heap-based state backend FLINK-21648
-
FLIP-151: Incremental snapshots for heap-based state backend
Open
FLIP-168: Speculative Execution for Batch Job 1.16 FLINK-28131
-
FLIP-168: Speculative Execution for Batch Job
Closed
FLIP-190: Support Version Upgrades for Table API & SQL Programs FLINK-25217
-
FLIP-190: Support Version Upgrades for Table API & SQL Programs
Open
FLIP-196: Source API stability guarantees FLINK-25345
-
FLIP-196: Source API stability guarantees
Open
FLIP-197: API stability graduation process 1.15 FLINK-25346
-
FLIP-197: API stability graduation process
Open
FLIP-204: Introduce Hash Lookup Join FLINK-23687
-
Introduce partitioned lookup join to enforce input of LookupJoin to hash shuffle by lookup keys
In Progress
FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records FLINK-25509
-
FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
Open
FLIP-211: Kerberos delegation token framework 1.17 FLINK-21232
-
Kerberos delegation token framework
Closed
FLIP-214: Support Advanced Function DDL 1.16 FLINK-14055
-
Add advanced function DDL syntax "USING JAR"
Closed
FLIP-216: Introduce pluggable dialect and plan for migrating Hive dialect 1.18 FLINK-26603
-
[Umbrella] Decouple Hive with Flink planner
Resolved
FLIP-217: Support watermark alignment of source splits 1.17 FLINK-28853
-
FLIP-217 Support watermark alignment of source splits
Resolved
FLIP-218: Support SELECT clause in CREATE TABLE(CTAS) 1.17 FLINK-26942
-
Support SELECT clause in CREATE TABLE(CTAS)
Open
FLIP-221: Abstraction for lookup source cache and metric 1.16 FLINK-28415
-
FLIP-221: Support Partial and Full Caching in Lookup Table Source
Resolved
FLIP-222: Support full job lifecycle statements in SQL client FLINK-27344
-
FLIP-222: Support full job lifecycle statements in SQL client
In Progress
FLIP-223: Support HiveServer2 Endpoint 1.16 FLINK-28149
-
[FLIP-223] Support HiveServer2 Endpoint
Closed
FLIP-224: Blocklist Mechanism 1.16 FLINK-28130
-
FLIP-224: Blocklist Mechanism
Closed
FLIP-225: Implement standalone mode support in the kubernetes operator kubernetes-operator-1.2.0 FLINK-27416
-
FLIP-225: Implement standalone mode support in the kubernetes operator
Closed
Released
Title Release JIRA FLIP-1: Fine Grained Recovery from Task Failures 1.9 FLINK-4256
-
Fine-grained recovery
Closed
FLIP-2: Extending Window Function Metadata 1.3 FLINK-4997
-
Extending Window Function Metadata
Closed
FLIP-3: Organization of Documentation 1.2 FLINK-4463
-
FLIP-3: Restructure Documentation
Closed
FLIP-4: Enhance Window Evictor 1.2 FLINK-4174
-
Enhance Window Evictor
Closed
FLIP-6: Flink Deployment and Process Model - Standalone, Yarn, Mesos, Kubernetes, etc. 1.5 FLINK-4319
-
Rework Cluster Management (FLIP-6)
Resolved
FLIP-7: Expose metrics to WebInterface 1.2 FLINK-4389
-
Expose metrics to Webfrontend
Closed
FLIP-8: Rescalable Non-Partitioned State 1.2 FLINK-4379
-
Add Rescalable Non-Partitioned State
Closed
FLIP-10: Unify Checkpoints and Savepoints 1.2 FLINK-4484
-
FLIP-10: Unify Savepoints and Checkpoints
Closed
FLIP-11: Table API Stream Aggregations 1.3 FLINK-4557
-
Table API Stream Aggregations
Resolved
FLIP-12: Asynchronous I/O Design and Implementation 1.2 FLINK-4391
-
Provide support for asynchronous operations over streams
Resolved
FLIP-13: Side Outputs in Flink 1.3 FLINK-4460
-
Side Outputs in Flink
Closed
FLIP-19: Improved BLOB storage architecture 1.4 FLIP-20: Integration of SQL and CEP 1.7 FLINK-6935
-
Integration of SQL and CEP
Open
FLIP-24: SQL Client 1.5 FLINK-7594
-
Add a SQL client
Closed
FLIP-25: Support User State TTL Natively 1.6 FLINK-3089
-
State API Should Support Data Expiration (State TTL)
Closed
FLIP-27: Refactor Source Interface 1.12 FLINK-10740
-
FLIP-27: Refactor Source Interface
Open
FLIP-29: Support map/flatMap/aggregate/flatAggregate on TableAPI 1.9 FLINK-10972
-
Enhancements to Flink Table API
Closed
FLINK-13470
-
Enhancements to Flink Table API for blink planner
Open
FLIP-30: Unified Catalog APIs 1.9 FLINK-11275
-
Unified Catalog APIs
Closed
FLIP-31: Pluggable Shuffle Service 1.9 FLINK-10653
-
Introduce Pluggable Shuffle Service Architecture
Closed
FLIP-32: Restructure flink-table for future contributions 1.14 FLINK-11439
-
INSERT INTO flink_sql SELECT * FROM blink_sql
Closed
FLIP-33: Standardize Connector Metrics 1.14 FLINK-11576
-
FLIP-33: Standardize connector metrics
Resolved
FLIP-34: Terminate/Suspend Job with Savepoint 1.9 FLINK-11458
-
Add TERMINATE/SUSPEND Job with Savepoint (FLIP-34)
Closed
FLIP-35: Support Chinese Documents and Website 1.9 FLINK-11526
-
Support Chinese Website for Apache Flink
Open
FLINK-11529
-
Support Chinese Documentation for Apache Flink
Open
FLIP-38: Python Table API 1.10 FLINK-12308
-
Support python language in Flink Table API
Closed
FLIP-41: Unify Binary format for Keyed State 1.13 FLINK-20976
-
Unify binary format for Keyed State savepoints
Resolved
FLIP-43: State Processor API 1.9 FLINK-12047
-
State Processor API (previously named Savepoint Connector) to read / write / process savepoints
Open
FLIP-49: Unified Memory Configuration for TaskExecutors 1.10 FLINK-13980
-
FLIP-49 Unified Memory Configuration for TaskExecutors
Resolved
FLIP-52 : Remove legacy Program interface. 1.10 FLINK-13713
-
Remove legacy Program interface.
Closed
FLIP-53: Fine Grained Operator Resource Management 1.10 FLINK-14058
-
FLIP-53 Fine-grained Operator Resource Management
Closed
FLIP-55: Introduction of a Table API Java Expression DSL 1.11 FLINK-11890
-
Replace Table API string-based expressions by a Java DSL
Closed