Introduction

Apache NiFi 2.0.0 includes the removal of multiple deprecated extension components and legacy product features. Version 1.18.0 introduced deprecation logging to provide runtime warnings for elements targeted for removal. Upgrading to the latest version 1 release provides the most accurate set of deprecation warnings.

Configuration changes must be completed prior to upgrading from version 1 to avoid failures. Application settings and flow design changes should be completed and tested before upgrading to version 2.

Migrating from deprecated components and features requires different steps depending on configured properties. The level of difficulty varies based on the number of properties to be changed and the steps involved to maintain similar behavior.

Features

Migrating deprecated features requires changes to application properties or custom component implementations. The steps involved will be different based on implementation details and deployment use cases.

Legacy and Alternative Sensitive Properties Algorithms

Migration

The nifi.sensitive.props.algorithm property in the nifi.properties configuration has supported a large number of possible values, including improved algorithms introduced in NiFi 1.14.0.

Starting with NiFi 1.14.0, the default value of the Sensitive Properties Algorithm changed from PBEWITHMD5AND256BITAES-CBC-OPENSSL to NIFI_PBKDF2_AES_GCM_256.

NiFi 1.18.0 added deprecation warnings for all algorithms starting with PBE.

NiFi 1.21.0 added deprecation for the following values using the bcrypt or scrypt algorithms, as well as other values using 128 bit keys.

• NIFI_ARGON2_AES_GCM_128
• NIFI_BCRYPT_AES_GCM_128
• NIFI_BCRYPT_AES_GCM_256
• NIFI_PBKDF2_AES_GCM_128
• NIFI_SCRYPT_AES_GCM_128
• NIFI_SCRYPT_AES_GCM_256

NiFi 2.0.0 supports the following options:

• NIFI_ARGON2_AES_GCM_256
• NIFI_PBKDF2_AES_GCM_256

Running the nifi.sh command supports migrating from a deprecated algorithm to a supported algorithm.

nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256

The Administrator's Guide provides additional details related to setting the Sensitive Properties Algorithm.

ProcessContext encrypt and decrypt methods

Migration

The ProcessContext interface provides property and component configuration information to Processors through the onTrigger method. The encrypt and decrypt methods supported enciphering and deciphering component-provided strings using the Sensitive Properties Key and Sensitive Properties Algorithm configured in nifi.properties.

These methods did not provide any persistent storage mechanism and also did not provide any guarantees of compatible behavior when changing application properties. Most components depend on framework capabilities to store and retrieve sensitive property values, obviating the need for these methods.

Custom components should use Property Descriptors with the sensitive field enabled to store and retrieve sensitive property values. Alternative uses of the encrypt and decrypt methods require custom implementation.

SSLContextService createSSLContext methods

Migration

The SSLContextService interface is a standard Controller Service that supports TLS communication for network services. The createSSLContext methods accepted a Client Authentication argument that did not apply to SSLContext instances. The replacement createContext method provides the same runtime behavior without requiring any arguments. Updating custom code to compile against Apache NiFi 1.13.0 or greater and switching to the createContext method provides compatibility with Apache NiFi 2.0.0.

CRON Scheduling Changes

Migration

Component run scheduling using cron expression changed from using Quartz to Spring Framework CronExpression for parsing the scheduling period in NIFI-12290.

The new parsing approach does not impact most cron expressions, but does introduce the following changes:

• Removed support for the optional Year field as the seventh field in an expression
• Integer definitions for day of the week span from 0 to 7 instead of 1 to 7

Components can no longer be scheduled to run for a particular year, so any referenced to a seventh field in an expression must be removed. Using named day of the week references such as SUN or MON are preferred over integer references.

ExpressionLanguageScope VARIABLE_REGISTRY

In Apache NiFi 2.0, the ExpressionLanguageScope VARIABLE_REGISTRY has been removed.  An alternate expression language scope ENVIRONMENT is now available, allowing for similar configurability via JVM system properties or system environment variables.  Further detail on this change may be found in NIFI-12079.

Migration

• Logic of existing flow definitions should be updated to no longer depend on the presence of variables at any process group scope, or in the file registry file.  Usage of parameter contexts and parameters is recommended to mitigate any gaps in flow definition logic.
• Existing custom NARs should be rebuilt using the NiFi 2.0 API.  Any component properties that provide configurability via VARIABLE_REGISTRY should be adjusted to reference one of the current ExpressionLanguageScope options.

Processors

Migrating deprecated Processors and properties requires flow design changes. The steps required range from setting different property values to replacing one Processor with one or more Processors providing similar capabilities.

The section for each deprecation provides the component class name as well as the Deprecation Type and Migration Difficulty. Each section includes Migration steps and example Flow Definitions with selected configuration properties.

Base64EncodeContent

Alternatives

Migration

Base64EncodeContent supports a single Mode property that can be set to Encode or Decode.

The EncodeContent Processor supports Base64 and other formats for both encoding and decoding.

EncodeContent provides a direct replacement for Base64EncodeContent using standard properties.

Flow Definitions

The Deprecated Flow Definition includes Base64EncodeContent configured for encoding.

• Base64EncodeContent-Deprecated.json

The Migrated Flow Definition replaces Base64EncodeContent with EncodeContent configured with Base64 encoding.

• Base64EncodeContent-Migrated.json

GetHTTP

Alternatives

Migration

GetHTTP supports a number of configurable properties to retrieve data using the HTTP GET method. The URL property controls the primary behavior of the Processor.

GetHTTP requires the Filename property to set the corresponding FlowFile attribute for content retrieved. The Processor requires an SSL Context Service when the URL starts with the HTTPS scheme.

The standard InvokeHTTP Processor supports standard HTTP methods including GET. Unlike GetHTTP, InvokeHTTP does not require an SSL Context Service for accessing HTTPS locations as it defaults to the Java Runtime default certificate authorities.

InvokeHTTP has a required HTTP URL property and defaults to setting the FlowFile filename attribute to a random UUID. The filename attribute can be derived from the URL path using the Response FlowFile Naming Strategy property set to URL_PATH.

InvokeHTTP has lower default socket timeout settings than GetHTTP.

Setting standard InvokeHTTP properties provides functionality similar to GetHTTP,

Flow Definitions

The Deprecated Flow Definition includes GetHTTP configured to retrieve the SHA-256 checksum for the NiFi 1.20.0 distribution. The Flow Definition also includes a required SSL Context Service configured with the Java Runtime cacerts Trust Store.

• GetHTTP-Deprecated.json

The Migrated Flow Definition includes InvokeHTTP configured with the same URL but without the SSL Context Service, which is not required for public trusted HTTPS services.

• GetHTTP-Migrated.json

GetJMSQueue

Alternatives

Migration

GetJMSQueue supports retrieving messages using the Java Message Service Queue interface to access Apache ActiveMQ.

GetJMSQueue bundles the ActiveMQ libraries and requires the URL property to specify the JMS server location, and the Destination Name property to specify the JMS queue for retrieval. Additional optional properties support TLS communication and authentication as well as batching.

The standard ConsumeJMS Processor similar functionality to GetJMSQueue, but requires additional configuration. ConsumeJMS does not bundle JMS libraries and requires referencing a local directory containing implementation JAR files.

ConsumeJMS should be configured with a Connection Factory Service implementation of JMSConnectionFactoryProvider with required properties. The ActiveMQ Artemis library provides the current supported version of JMS access.

ConsumeJMS uses the Connection Factory Service along with several other properties to access JMS Queues.

Flow Definitions

The Deprecated Flow Definition includes GetJMSQueue configured for ActiveMQ running on the localhost address with a specified Destination Name.

• GetJMSQueue-Deprecated.json

The Migrated Flow Definition includes ConsumeJMS configured with a JMSConnectionFactoryProvider using the same JMS broker information. The Migrated Flow Definition includes a JMS Parameters context with a Parameter named JmsClientLibrariesPath referencing the local directory containing the Apache ActiveMQ Artemis libraries.

• GetJMSQueue-Migrated.json

GetJMSTopic

Alternatives

Migration

GetJMSTopic supports retrieving messages using the Java Message Service Topic interface to access Apache ActiveMQ.

GetJMSTopic bundles the ActiveMQ libraries and requires the URL property to specify the JMS server location, and the Destination Name property to specify the JMS topic for retrieval. Additional optional properties support TLS communication and authentication as well as batching.

The standard ConsumeJMS Processor similar functionality to GetJMSTopic, but requires additional configuration. ConsumeJMS does not bundle JMS libraries and requires referencing a local directory containing implementation JAR files.

ConsumeJMS should be configured with a Connection Factory Service implementation of JMSConnectionFactoryProvider with required properties. The ActiveMQ Artemis library provides the current supported version of JMS access.

ConsumeJMS uses the Connection Factory Service along with several other properties to access JMS Queues.

Flow Definitions

The Deprecated Flow Definition includes GetJMSTopic configured for ActiveMQ running on the localhost address with a specified Destination Name.

• GetJMSTopic-Deprecated.json

The Migrated Flow Definition includes ConsumeJMS configured with a JMSConnectionFactoryProvider using the same JMS broker information. The Migrated Flow Definition includes a JMS Parameters context with a Parameter named JmsClientLibrariesPath referencing the local directory containing the Apache ActiveMQ Artemis libraries.

• GetJMSTopic-Migrated.json

GetTwitter

Alternatives

Migration

GetTwitter uses Version 1 of the Twitter REST API and relies on the retired Twitter Hosebird Client library.

GetTwitter supports several different endpoints for streaming Tweets, including Sample realtime Tweets and Filter realtime Tweets.

GetTwitter also supports configurable properties for filtering Tweets, but this filtering approach is not support in Version 2 of the Twitter REST API, and is not supported in ConsumeTwitter.

ConsumeTwitter uses Version 2 of the Twitter REST API. Twitter documentation provides a general guide for migrating from Version 1 to Version 2 for filtered streams and sampled streams.

ConsumeTwitter with Twitter API Version 2 requires a single OAuth 2.0 Bearer Token for app-only authentication. This approach replaces the API Key and Access Token approach required for GetTwitter.

Configuring ConsumeTwitter with the Sample Stream setting provides similar behavior to GetTwitter Sample Endpoint setting.

Configuring ConsumeTwitter with the Search Stream setting requires creating Filter Rules using the Twitter REST API outside of processor properties.

Twitter Filter Rules

Twitter Filter Rules can be created using JSON sent as an HTTP POST to the Twitter REST API Version 2.

• https://api.twitter.com/2/tweets/search/stream/rules

The Languages property in GetTwitter can be applied to a Filter Rule using the lang operator in a Filter value definition.

{
  "add": [
    {
      "value": "Apache lang:en",
      "tag": "English Language Filter"
    }
  ]
}

The Terms to Filter On property in GetTwitter can be applied to a Filter Rule using the Filter value definition with optional Boolean operators.

{
  "add": [
    {
      "value": "Apache AND NiFi",
      "tag": "Boolean Terms Filter"
    }
  ]
}

The IDs to Filter On property in GetTwitter can be applied to a Filter Rule using the from operator in a Filter value definition.

{
  "add": [
    {
      "value": "from:1625695877326340102",
      "tag": "Twitter User ID Filter"
    }
  ]
}

Flow Definitions

The Deprecated Flow Definition includes GetTwitter configured with the Sample Endpoint and without filters.

• GetTwitter-Deprecated.json

The Migrated Flow Definition includes ConsumeTwitter configured with the Sample Stream.

• GetTwitter-Migrated.json

HashAttribute

Alternatives

Migration

HashAttribute supports generating an MD5 digest of one more FlowFile attribute name and value pairs. The MD5 algorithm is not secure for cryptographic operations, and alternatives such as the SHA-2 or SHA-3 algorithms should be used.

The UpdateAttribute Processor supports generating a digest of one or more FlowFile attributes using the hash Expression Language function.

HashAttribute uses the Hash Value Attribute Key property to specify the destination FlowFile attribute name for the computed digest. HashAttribute uses one or more dynamic properties to provide source FlowFile attribute name, and a regular expression value to determine whether the FlowFile attribute should be included in the digest calculation. HashAttribute converts all attribute values to bytes using UTF-8 encoding.

Generating a hash attribute from a single FlowFile attribute name and value pair requires the both standard and dynamic properties.

UpdateAttribute uses dynamic properties to specify the the destination FlowFile attribute name, and the attribute value as an expression.

Generating a hash attribute from a single FlowFile attribute name and value requires one dynamic property.

Flow Definitions

The Deprecated Flow Definition includes HashAttribute configured to write a FlowFile attribute named label-hash using the label attribute name and the value of the attribute from the incoming FlowFile.

• HashAttribute-Deprecated.json

The Migrated Flow Definition includes CryptographicHashAttribute configured to write a FlowFile attribute named label-hash using the value of the label attribute from the incoming FlowFile.

• HashAttribute-Migrated.json

HashContent

Alternatives

Migration

HashContent supports generating a digest of FlowFile content using a configurable algorithm and writing the hash to a specified FlowFile attribute.

CryptographicHashContent supports the same basic digest capabilities and writes the hash to a FlowFile attribute name based on the configured algorithm.

The CryptographicHashContent FlowFile attribute name begins with content. When configured with the SHA-256 algorithm, CryptographicHashContent writes a FlowFile attribute named content_SHA-256.

Both Processors have a Hash Algorithm property, but the CryptographicHashContent Processor supports a scoped set of well-defined algorithms instead of depending on the algorithms that the Java Runtime Security Provider enumerates.

Flow Definitions

The Deprecated Flow Definition includes HashContent configured to write a FlowFile attribute named hash.value using the SHA-256 algorithm.

• HashContent-Deprecated.json

The Migrated Flow Definition includes CryptographicHashContent configured with the SHA-256 algorithm. The Migrated Flow Definition also includes an UpdateAttribute Processor that writes a FlowFile attribute named hash.value using the value of content_SHA-256, providing behavior equivalent to the HashContent Hash Attribute Name property.

• HashContent-Migrated.json

InvokeHTTP

Migration

InvokeHTTP includes several deprecated properties for configuring access through a proxy server.

• Proxy Host
• Proxy Port
• Proxy Type
• Proxy Username
• Proxy Password

InvokeHTTP supports a Controller Service property that abstracts proxy server access.

• Proxy Configuration Service

Creating and configuring an instance of StandardProxyConfigurationService with the corresponding property values is required for upgrading.

Flow Definitions

The Deprecated Flow Definition includes InvokeHTTP configured with Proxy Host and Proxy Port properties.

• InvokeHTTP-Proxy-Deprecated.json

The Migrated Flow Definition includes InvokeHTTP configured with a StandardProxyConfigurationService containing the same proxy property values.

• InvokeHTTP-Proxy-Migrated.json

PostHTTP

Alternatives

Migration

PostHTTP supports sending FlowFile content to a configurable HTTP URL using the HTTP POST method.

PostHTTP has a required URL property and also requires an SSL Context Service when the URL indicates an HTTPS location.

PostHTTP supports a specialized mode for transmitting binary packages containing FlowFile attributes and content to the ListenHTTP Processor. This behavior can be enabled using the Send as FlowFile property. There are various approach to implementing similar behavior, including configurable the MergeContent Processor with the FlowFile Version 3 format prior to sending through the InvokeHTTP Processor. Other options include passing selected FlowFile attribute values as HTTP headers, subject to ASCII encoding and length limitations.

The InvokeHTTP Processor supports standard HTTP methods including POST. Unlike PostHTTP, InvokeHTTP does not require an SSL Context Service for accessing HTTPS locations as it defaults to the Java Runtime default certificate authorities.

InvokeHTTP has a required HTTP URL property and also has lower default socket timeout settings than PostHTTP.

Flow Definitions

The Deprecated Flow Definition includes PostHTTP configured to send to ListenHTTP without custom FlowFile binary packaging.

• PostHTTP-Deprecated.json

The Migrated Flow Definition includes InvokeHTTP configured to send to ListenHTTP using equivalent settings.

• PostHTTP-Migrated.json

PutJMS

Alternatives

Migration

PutJMS supports sending messages using the Java Message Service interface to access Apache ActiveMQ.

PutJMS bundles the ActiveMQ libraries and requires the URL property to specify the JMS server location, and the Destination Name property to specify the JMS destination. PutJMS also supports a configurable Destination Type of Queue or Topic.

The PublishJMS Processor supports similar functionality to PutJMS, but requires additional configuration. PublishJMS does not bundle JMS libraries and requires referencing a local directory containing implementation JAR files.

PublishJMS should be configured with a Connection Factory Service implementation of JMSConnectionFactoryProvider with required properties. The ActiveMQ Artemis library provides the current supported version of JMS access.

PublishJMS uses the Connection Factory Service along with several other properties to access JMS destinations.

Flow Definitions

The Deprecated Flow Definition includes PutJMS configured for ActiveMQ running on the localhost address with a specified Destination Name.

• PutJMS-Deprecated.json

The Migrated Flow Definition includes PublishJMS configured with a JMSConnectionFactoryProvider using the same JMS broker information. The Migrated Flow Definition includes a JMS Parameters context with a Parameter named JmsClientLibrariesPath referencing the local directory containing the Apache ActiveMQ Artemis libraries.

• PutJMS-Migrated.json