Cloud Data Fusion release notes

The Cloud SQL for MySQL and Cloud SQL for PostgreSQL plugins, version 1.11.14, are available in Cloud Data Fusion version 6.10.x. Version 1.12.5 of these plugins is available in Cloud Data Fusion version 6.11.x.

This release includes the following feature:

• You can now configure transaction isolation levels in the Cloud SQL for MySQL and Cloud SQL for PostgreSQL plugins. This configuration applies to both batch sources and sinks, providing more precise control over data consistency and database locking (PLUGIN-1779).

The Windows Share Copy Action plugin version 2.12.5 is available in Cloud Data Fusion version 6.9.1 and later. Version 2.13.2 of this plugin is available in Cloud Data Fusion version 6.11.0 and later.

This release includes the following change:

• Added support for the modern SMBv2 and SMBv3 protocols to improve connection performance and reliability, while maintaining backward compatibility with pipelines that use SMBv1 (PLUGIN-1960).

Cloud Data Fusion version 6.11.1.4 is generally available (GA).

Fixed in Cloud Data Fusion 6.11.1.4:

• Fixed a race condition where successfully completed Dataproc jobs were incorrectly marked as failed on the Dataproc console due to premature program cancellation (CDAP-21219).
• Fixed an issue where the Pipeline list page would hang due to a race condition in an internal service (CDAP-21241).
• Fixed a security vulnerability in log downloads by enforcing strict validation on requested log paths and query parameters to prevent unauthorized access (CDAP-21260).

Changes in Cloud Data Fusion 6.11.1.4:

• Increased the default Wrangler browsing limit to 2,000 items (CDAP-21259).

The Salesforce plugin version 1.7.4 is available in Cloud Data Fusion version 6.8.0 and later.

This release includes the following change:

• Added support for the client credentials grant type in the Salesforce plugin. To select the grant type, use the new OAuth Grant Type field in the console (PLUGIN-1953).

Cloud Data Fusion version 6.11.1.3 is generally available (GA).

Fixed in Cloud Data Fusion 6.11.1.3:

• Fixed an issue that caused pipeline preview runs to fail with an InaccessibleObjectException when using certain plugins, such as Cloud SQL for PostgreSQL (CDAP-21212).

• Fixed an issue causing custom plugins to lose their logging context when running in parallel pipeline branches, ensuring consistent log propagation across both linear and parallel branched pipeline executions (CDAP-21245).

• Fixed critical security vulnerabilities in CDAP (CDAP-21250).

• Improved the latency of the List pipelines page (CDAP-21244).

• Fixed an issue causing intermittent service unavailability after instance upgrades (CDAP-21254).

Changes in Cloud Data Fusion 6.11.1.3:

• Introduced a deployStrategy query parameter in the Deploy Application API to skip the re-deployment of an existing pipeline if its configuration hasn't changed (CDAP-21246).

The Snowflake plugin version 1.2.1 is available in Cloud Data Fusion version 6.11.1 and later. This release includes the following change:

• Upgraded the Snowflake JDBC driver version from 3.14.4 to 4.0.2 and adapted to breaking API changes. This update migrates stream upload and download methods to use the new UploadStreamConfig and DownloadStreamConfig builders, updates OAuth2 authentication logic, and adds passphrase validation for key-pair authentication (PLUGIN-1954).

The HTTP plugin version 1.5.2 is available in Cloud Data Fusion version 6.11.1. This release includes the following change:

• Fixed an issue in the HTTP plugin where OAuth2 token requests bypassed configured proxies, causing connection failures in restricted environments. Authentication requests now correctly route through the configured proxy (PLUGIN-1956).

The Multi-table plugin version 1.4.2 is available in Cloud Data Fusion version 6.11.1 and later. This release includes the following changes:

• Introduced informational logging to flag when source tables or custom SQL queries yield zero records, which improves data pipeline observability. The split-generation logic is also updated to handle empty tables and prevent false-alarm logs during multi-split processing (PLUGIN-1950).

Cloud Data Fusion version 6.11.1.2 is generally available (GA).

This release includes the following changes:

• Fixed an issue that caused pipelines to get stuck in the provisioning state when processing large run records (CDAP-21234).

• Fixed the triggers panel in the pipeline details page to display the correct triggers count on initial load (CDAP-21230).

• Updated the GraphQL query mechanism to cache the pipelines list and fix the long loading screen in the deployed pipelines list page (CDAP-21229).

The SAP ODP plugin version 0.12.1 is available in Cloud Data Fusion version 6.10 and later. This release includes the following features and changes:

• Hierarchy data sources are now supported for extraction.

• ODQ fields are populated with TSN, package number, and record number for full extraction.

• Structured SAP messages are now displayed according to the message type (for example, type E as error and type I as information). Earlier, all messages from SAP displayed as errors.

• You can now use the TVARVC table entries with parameter /GOOG/ODP_TRANSFER_MODE (for all data sources) or /GOOG/M_DatasourceName (for individual data sources) to determine if data is transferred in CHAR or XML format. Maintain 'C' to transfer data in CHAR format. Maintain an empty value to transfer data in XML format.

Cloud Data Fusion version 6.11.1.1 is generally available (GA). This release includes the following feature:

• InstanceV3 monitored-resource: Introduced datafusion.googleapis.com/InstanceV3 as the default monitored resource for instance-level metrics and system service logs. This resource excludes the org_id and namespace labels found in InstanceV2. Emission of InstanceV2 metrics and logs is disabled by default for new and upgraded instances but can be re-enabled using the REST API.

  For more information, see Metrics overview and View pipeline logs.

Fixed in Cloud Data Fusion 6.11.1.1:

• Fixed retries in message publishing when the messaging service is temporarily unavailable (CDAP-21043).
• Fixed a security vulnerability where user-provided code in preview runners could access sensitive data from other preview runs (CDAP-21211).
• Fixed an issue where internal task workers running user code could hang indefinitely. The system now forces completed tasks to exit and uses a health check to restart unresponsive workers (CDAP-21213).
• Fixed an issue where the list apps API endpoint failed to return all deployed pipelines when used with pagination (CDAP-21220).

The SAP Table Reader plugin version 0.11.5 is available in Cloud Data Fusion version 6.10 and later.

This release includes the following change:

• Fixed an issue in the SAP Table Reader where running in Snapshot mode caused the Clean Up RFM process to generate an unnecessary Job abort failed error message in the detail logs.

The Salesforce plugin version 1.7.0 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

• Upgrade of Salesforce Bulk API V1 version from 62.0 to 64.0 (PLUGIN-1926).

Salesforce has deprecated certain fields in the API version 64.0. Upgrading to Salesforce plugin version 1.7.0 might cause pipelines that use these fields, to fail. To ensure your pipelines continue to work, you must manually update your pipeline schema to either load a new schema or remove the deprecated fields. For more information, see Prerequisites for upgrading to Salesforce plugin version 1.7.0.

The ServiceNow plugin version 1.2.7 is available in Cloud Data Fusion version 6.10.1. This release includes the following change:

• Fixed an issue related to schema backward compatibility while upgrading from plugin version 1.1.0 (PLUGIN-1902).

Cloud Data Fusion version 6.11.1 is generally available (GA). This release includes the following features:

• Added support for HTTP access tokens (Bearer authentication) in Bitbucket Server for source control management (CDAP-21049).
• A new API is available to retrieve the application count for each namespace (CDAP-21161).

Fixed in Cloud Data Fusion 6.11.1:

• To prevent storage issues on static Dataproc clusters, temporary pipeline data is automatically deleted after a successful run (CDAP-21076).
• Fixed an issue that prevented Expression Language (EL) from being used within user-defined directives (CDAP-21204).
• Fixed an issue where a pipeline would fail when reading an encrypted file from Cloud Storage using Tink (PLUGIN-1717).

Dataproc 2.0 is no longer supported in Cloud Data Fusion version 6.11.1 and later.

Changes in Cloud Data Fusion 6.11.1:

• The Java runtime environment is upgraded from Java 8 to Java 11 (CDAP-21184).
• To create ephemeral Dataproc clusters, Cloud Data Fusion uses the Dataproc 2.3 image by default (CDAP-21187).
• The pipeline JSON size limit for creating new pipelines and importing pipelines as JSON is increased to 5MB (previously 2MB) (CDAP-21194).
• On the Pipeline details page, the inbound triggers sidebar features a paginated list of pipelines where you can select the pipelines you want to add to the trigger. Additionally, a refresh button is added to update the existing list of triggers and pipelines (CDAP-21195).

You can now attach tags to Cloud Data Fusion instances during instance creation. This ensures immediate metadata for better organization, cost tracking, and policy automation. For more information, see Attach tags during instance creation.

The Cloud Data Fusion version 6.10.1.5 patch revision is generally available (GA). 6.10.1.5 includes reliability fixes.

The SAP ODP plugin version 0.12 is available in Cloud Data Fusion version 6.10 and later. This release includes the following changes:

• Support for old SAP ODP RFCs is removed to align with SAP Note 3255746.
• Messages have been updated when attempting to extract data from unsupported hierarchy data sources.

We recommend upgrading all your pipelines to the latest SAP ODP plugin version, as earlier versions are no longer compliant with SAP Guidelines.

The Oracle plugin version 1.12.3 is available in Cloud Data Fusion (via Hub) versions 6.11.0 and later, and 1.11.8 is available in Cloud Data Fusion (via Hub) version 6.10.

This release provides backward compatibility for recent schema changes, including the following:

• Precisionless numbers: In version 1.10.0, precisionless numbers were handled as strings.
• Improved timestamp handling: Version 1.11.0 introduced improvements to timestamp handling.

To address backward compatibility for these changes, two new hidden fields are introduced in Oracle batch source configurations: treatPrecisionlessNumAsDeci and treatAsOldTimestamp. Both flags default to false. To enable these flags, edit the respective values in your exported connection JSON (if using connections) or pipeline JSON (if not using connections) before re-importing or re-deploying (PLUGIN-1893).

The Elasticsearch plugin version 1.11.0 is available in Cloud Data Fusion version 6.11.0. This release includes the following change:

• Upgraded Hadoop version for Elasticsearch plugin compatibility (PLUGIN-1881).

The HTTP plugin version 1.4.4 is available in Cloud Data Fusion version 6.10.1. This release includes the following changes:

• Implemented the Client Credentials Grant flow for HTTP OAuth2, enabling authorized clients to securely access data using the client_credentials grant type. Client credentials can be passed through Basic Authentication header, in the request body, or as query parameters (PLUGIN-1872).

• Fixed an issue causing the HTTP Source plugin to throw a NullPointerException when the BasePageIterator received a null response (PLUGIN-1894).

Cloud Data Fusion is available in the northamerica-south1 (Mexico) region. For more information, see Pricing.

The Salesforce plugin version 1.6.10 is available in Cloud Data Fusion versions 6.10.1 and 6.11.0. This release includes the following changes:

• Upgraded Salesforce API version from 53.0 to 62.0 (PLUGIN-1891).
• Added a retry mechanism for all Salesforce API calls from the Salesforce plugin (PLUGIN-1892).

The DB plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following change:

• Added the TRANSACTION_ISOLATION_LEVEL property to DB Plugins (PostgreSQL, MySQL, and MSSQL).

The MariaDB Batch Source plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following changes:

• Fixed an issue related to the MariaDB YEAR data type causing pipeline failures when the date type was manually changed to DATE. Validation for this specific scenario is now skipped (PLUGIN-1888).

• Fixed an issue that caused pipeline failures with a java.lang.ClassNotFoundException: Class io.cdap.plugin.db.DBRecord not found error, which was due to a class loading problem (PLUGIN-1874).

• Added a committer to Database plugins to manage committing or rolling back entire tasks, depending on whether the task completes successfully or is aborted (PLUGIN-1815).

Cloud Data Fusion version 6.9 is no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

The Python Transform plugin (version 2.3.2) is available in CDAP version 6.10.1. This includes bug fix for the deprecated PROTOCOL_TLSv1 on Dataproc 2.0 and later. The issue occurs when earlier TLS versions, such as TLSv1 and TLSv1.1 are disabled by default due to security concerns. Applications relying on ssl.PROTOCOL_TLSv1 in Python might fail and requires updates to use ssl.PROTOCOL_TLSv1_2 or later.

Cloud Data Fusion version 6.11.0 is available in Preview.

Cloud Data Fusion 6.11.0 offers high availability with reduced upgrade downtime.

When a pipeline run fails, you can retrieve detailed error information on the pipeline details page of the Cloud Data Fusion web interface.

Cloud Data Fusion classifies pipeline errors by category, reason, and message. This classification speeds up resolution and reduces the need to examine complex logs. For more information, see Retrieve error information for a failed pipeline run.

You can view instance metrics and pipeline metrics in Cloud Monitoring and in the dashboard provided by Cloud Data Fusion. For more information, see Metrics overview and Monitor Cloud Data Fusion system, instance, and pipeline health.

You can view instance logs and pipeline logs in Cloud Logging, and in the dashboard provided by Cloud Data Fusion. For more information, see View Cloud Data Fusion logs.

The following APIs for downloading system service and pipeline run logs are deprecated in 6.11.0:

• GET /v3/namespaces/<NAMESPACE_ID>/apps/<APP_ID>/<PROGRAM_TYPE>/<PROGRAM_ID>/logs
• GET /v3/system/services/<SERVICE_ID>/logs

The ability to retrieve all applications without pagination using the GET /v3/namespaces/<NAMESPACE_ID>/apps endpoint is deprecated in 6.11.0.

Fixed in Cloud Data Fusion 6.11.0:

• Fixed a null pointer exception in the BigQuery multi-sink plugin when used without a reference name (PLUGIN-1843).

• Fixed Joiner plugin failures observed on Dataproc 2.2-debian12 instances (CDAP-21075).

• Fixed an issue that prevented pipelines from accepting empty input in Amazon S3 and Google Cloud Storage source plugins (PLUGIN-1742).

• Fixed an issue in RBAC-enabled instances where the pipeline details page displayed an incorrect author name (CDAP-21069).

Changes in Cloud Data Fusion 6.11.0:

• To create ephemeral clusters, Cloud Data Fusion uses the Dataproc 2.2 image by default. For more information about its limitations in Cloud Data Fusion, see Change the Dataproc image to version 2.1.

• The maximum concurrent runs limit for triggers is displayed in the console (CDAP-21072).

• Added support for destination table write preference in the BigQuery Execute plugin (PLUGIN-1438).

The following APIs for searching and querying metrics are deprecated in 6.11.0:

• POST v3/metrics/query
• POST v3/metrics/search

A soft limit of 2 MB for pipeline JSON size is introduced in 6.11.0. Pipelines exceeding this size might encounter deployment failures.

The SAP OData plugin version 0.11.6 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

Fixed an issue causing pipeline deployments to fail due to SAP memory dumps when processing large datasets with macro filters: ERROR Stage 'SAP OData' encountered : CDF_SAP_ODATA_01534 - Service validation failed. Root Cause:Invalid parametertype used at function.

You can use custom constraints with Organization Policy to provide more granular control over specific fields for some Cloud Data Fusion resources. For more information, see Create custom organization policy constraints.

The SAP SuccessFactors plugin version 1.2.4 is available in Cloud Data Fusion version 6.8.0 and later. This release lets you use OAuth 2.0 for ODATA API authentication (PLUGIN-1741).

The Cloud Data Fusion version 6.10.1.2 patch revision is generally available (GA). 6.10.1.2 includes the following changes:

• You can generate audit logs that record data plane activities within your Cloud Data Fusion instance. Data plane audit logging is available in Preview for RBAC-enabled instances.

• To improve the API response time, by default, all program activity records older than 30 days are cleaned up. Any activity older than 30 days isn't visible in the Cloud Data Fusion studio (CDAP-14950).

• When using role-based access control, performing the List Pipelines operation requires datafusion.pipelines.list permission, in addition to datafusion.namespaces.get permission. For more information, see RBAC roles and permissions (CDAP-20931).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to overcount in cases where servers were restarted when a pipeline run was in progress (CDAP-21046).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to be stale after a restart when no pipelines were running (CDAP-21048).

• Fixed an issue causing the default max concurrent runs limit for triggers not to appear in the web interface, making it difficult to tell if triggers were working as intended (CDAP-21072).

• Fixed an issue causing the top panel of the Studio tab to disappear when you edited a pipeline draft that's based on a pipeline from an earlier Cloud Data Fusion version (CDAP-21073).

• Improved performance by removing a call to the list apps API during pipeline deployment when checking if a pipeline already exists (CDAP-21074).

Cloud Data Fusion supports the CMEK organization policy.

The Snowflake plugin version 1.1.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1816):

• Fixed an issue in the Snowflake source causing pipelines to fail if fields contained decimals.
• Fixed an issue in the Snowflake source causing pipelines to fail if data contained a backslash (\). You can set a new escape character using the cdap.snowflake.source.escape runtime argument.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.10.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017).

The HTTP plugin version 1.4.3 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1810):

• In the HTTP streaming source, batch source, and batch sink, a PATCH option was added to the HTTP Method field.
• Fixed an issue in the HTTP sink causing data loss when a pipeline didn't fail by default when a non-2xx response code was received.
• Fixed an issue in the HTTP source causing a pipeline not to fail when the get_schema method was called and a non-2xx response code was received.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.8.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017). This note is incorrect; see entry for November 27, 2024.

The SAP table batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The Multiple table plugin version 1.4.1 is available in Cloud Data Fusion versions 6.10.1 and later. This release fixes an issue causing pipelines to fail if a Multiple database tables batch source's Reference Name field contains spaces. The field no longer accepts spaces (PLUGIN-1752).

The Oracle plugin version 1.11.4 is available in Cloud Data Fusion versions 6.10.1 and later. This release includes the following change:

• Fixed an issue causing pipelines with an Oracle sink that has date columns in the input schema to fail (PLUGIN-1812).

Using Dataproc version 2.2 in your Cloud Data Fusion pipeline can fail in some cases with the following error: ERROR [Driver:o.a.s.d.y.ApplicationMaster@97] - User class threw exception: java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.encoders.ExpressionEncoder org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(org.apache.spark.sql.types.StructType)' at io.cdap.cdap.etl.spark.batch.OpaqueDatasetCollection.toDataframeCollection(OpaqueDatasetCollection.java:111).
To avoid this issue, change the Dataproc image to 2.1 (CDAP-21075).

The SAP ODATA batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing pipelines to fail when the source plugin contained nested fields (an array of records): ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task io.cdap.cdap.api.data.format.UnexpectedFormatException: Schema must be a record with at least one field.

• Added a Read Timeout plugin property, which lets you configure the read timeout value for long-running pipelines.

The