Status

Current state: Under Discussion

Discussion thread: here 

JIRA: KAFKA-3729

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In this particular issue, we wish to allow users to also be able to configure Serdes that are passed along in the topology builder. However, after the initial attempt to implement a change, several issues were discovered which warranted a public API change. Hence, why this KIP is necessary. Several of the major problems were identified as follows:

  • The configure() method present is called twice. The first time, it is called by the user. The second one by Kafka Steams, which overwrites the preexisting configuration (that was the correct one).
  • Another problem is that only one set of configurations is available in StreamsConfig for all Serdes in Kafka Streams, which means that Serdes with the same config parameter names would have the same values even though the user wishes for different ones.

Taking an in-depth look at these problems, there are several possible ways by which these could be resolved. Approaches worthy of consideration are listed in detail below.

Public Interfaces

A main problem noted in the motivation is that configure()  is called twice. Below are two ways which we could resolve this issue:

Configuration passed through constructor.

In this case, we would deprecate the old configuration() method and pass the configuration through the constructor. In which case, this would mean that a configuration is effectively final. In this manner, once the configuration of a serde is set, it cannot be changed.

Add a isConfigured() flag

A boolean flag could be added in the Serde, where upon the first time configure() has been called, the second time it is called, it will be a no-op (because isConfigured  has been set from false to true). This approach might be simpler to implement because the upgrade path is more straightforward.

It is important that the configure() operation is idempotent.

Another issue which is present is that two Serdes could not be configured with different config values. In order to resolve this issue, the global StreamsConfig must contain not only one configuration of Serdes, but several, each for a distinctive Serde. This would allow individual Serdes to possess different configuration values. 

Migration Plan

Upgrade path will be decided once the alternative has been selected.

Rejected Alternatives

TBD. Alternatives to be accepted and rejected still needs discussion.

  • No labels