Cloud Data Fusion release notes

The Cloud SQL for MySQL and Cloud SQL for PostgreSQL plugins, version 1.11.14, are available in Cloud Data Fusion version 6.10.x. Version 1.12.5 of these plugins is available in Cloud Data Fusion version 6.11.x.

This release includes the following feature:

• You can now configure transaction isolation levels in the Cloud SQL for MySQL and Cloud SQL for PostgreSQL plugins. This configuration applies to both batch sources and sinks, providing more precise control over data consistency and database locking (PLUGIN-1779).

The Windows Share Copy Action plugin version 2.12.5 is available in Cloud Data Fusion version 6.9.1 and later. Version 2.13.2 of this plugin is available in Cloud Data Fusion version 6.11.0 and later.

This release includes the following change:

• Added support for the modern SMBv2 and SMBv3 protocols to improve connection performance and reliability, while maintaining backward compatibility with pipelines that use SMBv1 (PLUGIN-1960).

Cloud Data Fusion version 6.11.1.4 is generally available (GA).

Fixed in Cloud Data Fusion 6.11.1.4:

• Fixed a race condition where successfully completed Dataproc jobs were incorrectly marked as failed on the Dataproc console due to premature program cancellation (CDAP-21219).
• Fixed an issue where the Pipeline list page would hang due to a race condition in an internal service (CDAP-21241).
• Fixed a security vulnerability in log downloads by enforcing strict validation on requested log paths and query parameters to prevent unauthorized access (CDAP-21260).

Changes in Cloud Data Fusion 6.11.1.4:

• Increased the default Wrangler browsing limit to 2,000 items (CDAP-21259).

The Salesforce plugin version 1.7.4 is available in Cloud Data Fusion version 6.8.0 and later.

This release includes the following change:

• Added support for the client credentials grant type in the Salesforce plugin. To select the grant type, use the new OAuth Grant Type field in the console (PLUGIN-1953).

Cloud Data Fusion version 6.11.1.3 is generally available (GA).

Fixed in Cloud Data Fusion 6.11.1.3:

• Fixed an issue that caused pipeline preview runs to fail with an InaccessibleObjectException when using certain plugins, such as Cloud SQL for PostgreSQL (CDAP-21212).

• Fixed an issue causing custom plugins to lose their logging context when running in parallel pipeline branches, ensuring consistent log propagation across both linear and parallel branched pipeline executions (CDAP-21245).

• Fixed critical security vulnerabilities in CDAP (CDAP-21250).

• Improved the latency of the List pipelines page (CDAP-21244).

• Fixed an issue causing intermittent service unavailability after instance upgrades (CDAP-21254).

Changes in Cloud Data Fusion 6.11.1.3:

• Introduced a deployStrategy query parameter in the Deploy Application API to skip the re-deployment of an existing pipeline if its configuration hasn't changed (CDAP-21246).

The Snowflake plugin version 1.2.1 is available in Cloud Data Fusion version 6.11.1 and later. This release includes the following change:

• Upgraded the Snowflake JDBC driver version from 3.14.4 to 4.0.2 and adapted to breaking API changes. This update migrates stream upload and download methods to use the new UploadStreamConfig and DownloadStreamConfig builders, updates OAuth2 authentication logic, and adds passphrase validation for key-pair authentication (PLUGIN-1954).

The HTTP plugin version 1.5.2 is available in Cloud Data Fusion version 6.11.1. This release includes the following change:

• Fixed an issue in the HTTP plugin where OAuth2 token requests bypassed configured proxies, causing connection failures in restricted environments. Authentication requests now correctly route through the configured proxy (PLUGIN-1956).

The Multi-table plugin version 1.4.2 is available in Cloud Data Fusion version 6.11.1 and later. This release includes the following changes:

• Introduced informational logging to flag when source tables or custom SQL queries yield zero records, which improves data pipeline observability. The split-generation logic is also updated to handle empty tables and prevent false-alarm logs during multi-split processing (PLUGIN-1950).

Cloud Data Fusion version 6.11.1.2 is generally available (GA).

This release includes the following changes:

• Fixed an issue that caused pipelines to get stuck in the provisioning state when processing large run records (CDAP-21234).

• Fixed the triggers panel in the pipeline details page to display the correct triggers count on initial load (CDAP-21230).

• Updated the GraphQL query mechanism to cache the pipelines list and fix the long loading screen in the deployed pipelines list page (CDAP-21229).

The SAP ODP plugin version 0.12.1 is available in Cloud Data Fusion version 6.10 and later. This release includes the following features and changes:

• Hierarchy data sources are now supported for extraction.

• ODQ fields are populated with TSN, package number, and record number for full extraction.

• Structured SAP messages are now displayed according to the message type (for example, type E as error and type I as information). Earlier, all messages from SAP displayed as errors.

• You can now use the TVARVC table entries with parameter /GOOG/ODP_TRANSFER_MODE (for all data sources) or /GOOG/M_DatasourceName (for individual data sources) to determine if data is transferred in CHAR or XML format. Maintain 'C' to transfer data in CHAR format. Maintain an empty value to transfer data in XML format.

Cloud Data Fusion version 6.11.1.1 is generally available (GA). This release includes the following feature:

• InstanceV3 monitored-resource: Introduced datafusion.googleapis.com/InstanceV3 as the default monitored resource for instance-level metrics and system service logs. This resource excludes the org_id and namespace labels found in InstanceV2. Emission of InstanceV2 metrics and logs is disabled by default for new and upgraded instances but can be re-enabled using the REST API.

  For more information, see Metrics overview and View pipeline logs.

Fixed in Cloud Data Fusion 6.11.1.1:

• Fixed retries in message publishing when the messaging service is temporarily unavailable (CDAP-21043).
• Fixed a security vulnerability where user-provided code in preview runners could access sensitive data from other preview runs (CDAP-21211).
• Fixed an issue where internal task workers running user code could hang indefinitely. The system now forces completed tasks to exit and uses a health check to restart unresponsive workers (CDAP-21213).
• Fixed an issue where the list apps API endpoint failed to return all deployed pipelines when used with pagination (CDAP-21220).

The SAP Table Reader plugin version 0.11.5 is available in Cloud Data Fusion version 6.10 and later.

This release includes the following change:

• Fixed an issue in the SAP Table Reader where running in Snapshot mode caused the Clean Up RFM process to generate an unnecessary Job abort failed error message in the detail logs.

The Salesforce plugin version 1.7.0 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

• Upgrade of Salesforce Bulk API V1 version from 62.0 to 64.0 (PLUGIN-1926).

Salesforce has deprecated certain fields in the API version 64.0. Upgrading to Salesforce plugin version 1.7.0 might cause pipelines that use these fields, to fail. To ensure your pipelines continue to work, you must manually update your pipeline schema to either load a new schema or remove the deprecated fields. For more information, see Prerequisites for upgrading to Salesforce plugin version 1.7.0.

The ServiceNow plugin version 1.2.7 is available in Cloud Data Fusion version 6.10.1. This release includes the following change:

• Fixed an issue related to schema backward compatibility while upgrading from plugin version 1.1.0 (PLUGIN-1902).

Cloud Data Fusion version 6.11.1 is generally available (GA). This release includes the following features:

• Added support for HTTP access tokens (Bearer authentication) in Bitbucket Server for source control management (CDAP-21049).
• A new API is available to retrieve the application count for each namespace (CDAP-21161).

Fixed in Cloud Data Fusion 6.11.1:

• To prevent storage issues on static Dataproc clusters, temporary pipeline data is automatically deleted after a successful run (CDAP-21076).
• Fixed an issue that prevented Expression Language (EL) from being used within user-defined directives (CDAP-21204).
• Fixed an issue where a pipeline would fail when reading an encrypted file from Cloud Storage using Tink (PLUGIN-1717).

Dataproc 2.0 is no longer supported in Cloud Data Fusion version 6.11.1 and later.

Changes in Cloud Data Fusion 6.11.1:

• The Java runtime environment is upgraded from Java 8 to Java 11 (CDAP-21184).
• To create ephemeral Dataproc clusters, Cloud Data Fusion uses the Dataproc 2.3 image by default (CDAP-21187).
• The pipeline JSON size limit for creating new pipelines and importing pipelines as JSON is increased to 5MB (previously 2MB) (CDAP-21194).
• On the Pipeline details page, the inbound triggers sidebar features a paginated list of pipelines where you can select the pipelines you want to add to the trigger. Additionally, a refresh button is added to update the existing list of triggers and pipelines (CDAP-21195).

You can now attach tags to Cloud Data Fusion instances during instance creation. This ensures immediate metadata for better organization, cost tracking, and policy automation. For more information, see Attach tags during instance creation.

The Cloud Data Fusion version 6.10.1.5 patch revision is generally available (GA). 6.10.1.5 includes reliability fixes.

The SAP ODP plugin version 0.12 is available in Cloud Data Fusion version 6.10 and later. This release includes the following changes:

• Support for old SAP ODP RFCs is removed to align with SAP Note 3255746.
• Messages have been updated when attempting to extract data from unsupported hierarchy data sources.

We recommend upgrading all your pipelines to the latest SAP ODP plugin version, as earlier versions are no longer compliant with SAP Guidelines.

The Oracle plugin version 1.12.3 is available in Cloud Data Fusion (via Hub) versions 6.11.0 and later, and 1.11.8 is available in Cloud Data Fusion (via Hub) version 6.10.

This release provides backward compatibility for recent schema changes, including the following:

• Precisionless numbers: In version 1.10.0, precisionless numbers were handled as strings.
• Improved timestamp handling: Version 1.11.0 introduced improvements to timestamp handling.

To address backward compatibility for these changes, two new hidden fields are introduced in Oracle batch source configurations: treatPrecisionlessNumAsDeci and treatAsOldTimestamp. Both flags default to false. To enable these flags, edit the respective values in your exported connection JSON (if using connections) or pipeline JSON (if not using connections) before re-importing or re-deploying (PLUGIN-1893).

The Elasticsearch plugin version 1.11.0 is available in Cloud Data Fusion version 6.11.0. This release includes the following change:

• Upgraded Hadoop version for Elasticsearch plugin compatibility (PLUGIN-1881).

The HTTP plugin version 1.4.4 is available in Cloud Data Fusion version 6.10.1. This release includes the following changes:

• Implemented the Client Credentials Grant flow for HTTP OAuth2, enabling authorized clients to securely access data using the client_credentials grant type. Client credentials can be passed through Basic Authentication header, in the request body, or as query parameters (PLUGIN-1872).

• Fixed an issue causing the HTTP Source plugin to throw a NullPointerException when the BasePageIterator received a null response (PLUGIN-1894).

Cloud Data Fusion is available in the northamerica-south1 (Mexico) region. For more information, see Pricing.

The Salesforce plugin version 1.6.10 is available in Cloud Data Fusion versions 6.10.1 and 6.11.0. This release includes the following changes:

• Upgraded Salesforce API version from 53.0 to 62.0 (PLUGIN-1891).
• Added a retry mechanism for all Salesforce API calls from the Salesforce plugin (PLUGIN-1892).

The DB plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following change:

• Added the TRANSACTION_ISOLATION_LEVEL property to DB Plugins (PostgreSQL, MySQL, and MSSQL).

The MariaDB Batch Source plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following changes:

• Fixed an issue related to the MariaDB YEAR data type causing pipeline failures when the date type was manually changed to DATE. Validation for this specific scenario is now skipped (PLUGIN-1888).

• Fixed an issue that caused pipeline failures with a java.lang.ClassNotFoundException: Class io.cdap.plugin.db.DBRecord not found error, which was due to a class loading problem (PLUGIN-1874).

• Added a committer to Database plugins to manage committing or rolling back entire tasks, depending on whether the task completes successfully or is aborted (PLUGIN-1815).

Cloud Data Fusion version 6.9 is no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

The Python Transform plugin (version 2.3.2) is available in CDAP version 6.10.1. This includes bug fix for the deprecated PROTOCOL_TLSv1 on Dataproc 2.0 and later. The issue occurs when earlier TLS versions, such as TLSv1 and TLSv1.1 are disabled by default due to security concerns. Applications relying on ssl.PROTOCOL_TLSv1 in Python might fail and requires updates to use ssl.PROTOCOL_TLSv1_2 or later.

Cloud Data Fusion version 6.11.0 is available in Preview.

Cloud Data Fusion 6.11.0 offers high availability with reduced upgrade downtime.

When a pipeline run fails, you can retrieve detailed error information on the pipeline details page of the Cloud Data Fusion web interface.

Cloud Data Fusion classifies pipeline errors by category, reason, and message. This classification speeds up resolution and reduces the need to examine complex logs. For more information, see Retrieve error information for a failed pipeline run.

You can view instance metrics and pipeline metrics in Cloud Monitoring and in the dashboard provided by Cloud Data Fusion. For more information, see Metrics overview and Monitor Cloud Data Fusion system, instance, and pipeline health.

You can view instance logs and pipeline logs in Cloud Logging, and in the dashboard provided by Cloud Data Fusion. For more information, see View Cloud Data Fusion logs.

The following APIs for downloading system service and pipeline run logs are deprecated in 6.11.0:

• GET /v3/namespaces/<NAMESPACE_ID>/apps/<APP_ID>/<PROGRAM_TYPE>/<PROGRAM_ID>/logs
• GET /v3/system/services/<SERVICE_ID>/logs

The ability to retrieve all applications without pagination using the GET /v3/namespaces/<NAMESPACE_ID>/apps endpoint is deprecated in 6.11.0.

Fixed in Cloud Data Fusion 6.11.0:

• Fixed a null pointer exception in the BigQuery multi-sink plugin when used without a reference name (PLUGIN-1843).

• Fixed Joiner plugin failures observed on Dataproc 2.2-debian12 instances (CDAP-21075).

• Fixed an issue that prevented pipelines from accepting empty input in Amazon S3 and Google Cloud Storage source plugins (PLUGIN-1742).

• Fixed an issue in RBAC-enabled instances where the pipeline details page displayed an incorrect author name (CDAP-21069).

Changes in Cloud Data Fusion 6.11.0:

• To create ephemeral clusters, Cloud Data Fusion uses the Dataproc 2.2 image by default. For more information about its limitations in Cloud Data Fusion, see Change the Dataproc image to version 2.1.

• The maximum concurrent runs limit for triggers is displayed in the console (CDAP-21072).

• Added support for destination table write preference in the BigQuery Execute plugin (PLUGIN-1438).

The following APIs for searching and querying metrics are deprecated in 6.11.0:

• POST v3/metrics/query
• POST v3/metrics/search

A soft limit of 2 MB for pipeline JSON size is introduced in 6.11.0. Pipelines exceeding this size might encounter deployment failures.

The SAP OData plugin version 0.11.6 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

Fixed an issue causing pipeline deployments to fail due to SAP memory dumps when processing large datasets with macro filters: ERROR Stage 'SAP OData' encountered : CDF_SAP_ODATA_01534 - Service validation failed. Root Cause:Invalid parametertype used at function.

You can use custom constraints with Organization Policy to provide more granular control over specific fields for some Cloud Data Fusion resources. For more information, see Create custom organization policy constraints.

The SAP SuccessFactors plugin version 1.2.4 is available in Cloud Data Fusion version 6.8.0 and later. This release lets you use OAuth 2.0 for ODATA API authentication (PLUGIN-1741).

The Cloud Data Fusion version 6.10.1.2 patch revision is generally available (GA). 6.10.1.2 includes the following changes:

• You can generate audit logs that record data plane activities within your Cloud Data Fusion instance. Data plane audit logging is available in Preview for RBAC-enabled instances.

• To improve the API response time, by default, all program activity records older than 30 days are cleaned up. Any activity older than 30 days isn't visible in the Cloud Data Fusion studio (CDAP-14950).

• When using role-based access control, performing the List Pipelines operation requires datafusion.pipelines.list permission, in addition to datafusion.namespaces.get permission. For more information, see RBAC roles and permissions (CDAP-20931).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to overcount in cases where servers were restarted when a pipeline run was in progress (CDAP-21046).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to be stale after a restart when no pipelines were running (CDAP-21048).

• Fixed an issue causing the default max concurrent runs limit for triggers not to appear in the web interface, making it difficult to tell if triggers were working as intended (CDAP-21072).

• Fixed an issue causing the top panel of the Studio tab to disappear when you edited a pipeline draft that's based on a pipeline from an earlier Cloud Data Fusion version (CDAP-21073).

• Improved performance by removing a call to the list apps API during pipeline deployment when checking if a pipeline already exists (CDAP-21074).

Cloud Data Fusion supports the CMEK organization policy.

The Snowflake plugin version 1.1.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1816):

• Fixed an issue in the Snowflake source causing pipelines to fail if fields contained decimals.
• Fixed an issue in the Snowflake source causing pipelines to fail if data contained a backslash (\). You can set a new escape character using the cdap.snowflake.source.escape runtime argument.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.10.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017).

The HTTP plugin version 1.4.3 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1810):

• In the HTTP streaming source, batch source, and batch sink, a PATCH option was added to the HTTP Method field.
• Fixed an issue in the HTTP sink causing data loss when a pipeline didn't fail by default when a non-2xx response code was received.
• Fixed an issue in the HTTP source causing a pipeline not to fail when the get_schema method was called and a non-2xx response code was received.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.8.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017). This note is incorrect; see entry for November 27, 2024.

The SAP table batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The Multiple table plugin version 1.4.1 is available in Cloud Data Fusion versions 6.10.1 and later. This release fixes an issue causing pipelines to fail if a Multiple database tables batch source's Reference Name field contains spaces. The field no longer accepts spaces (PLUGIN-1752).

The Oracle plugin version 1.11.4 is available in Cloud Data Fusion versions 6.10.1 and later. This release includes the following change:

• Fixed an issue causing pipelines with an Oracle sink that has date columns in the input schema to fail (PLUGIN-1812).

Using Dataproc version 2.2 in your Cloud Data Fusion pipeline can fail in some cases with the following error: ERROR [Driver:o.a.s.d.y.ApplicationMaster@97] - User class threw exception: java.lang.NoSuchMethodError: 'org.apache.spark.sql.catalyst.encoders.ExpressionEncoder org.apache.spark.sql.catalyst.encoders.RowEncoder.apply(org.apache.spark.sql.types.StructType)' at io.cdap.cdap.etl.spark.batch.OpaqueDatasetCollection.toDataframeCollection(OpaqueDatasetCollection.java:111).
To avoid this issue, change the Dataproc image to 2.1 (CDAP-21075).

The SAP ODATA batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing pipelines to fail when the source plugin contained nested fields (an array of records): ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task io.cdap.cdap.api.data.format.UnexpectedFormatException: Schema must be a record with at least one field.

• Added a Read Timeout plugin property, which lets you configure the read timeout value for long-running pipelines.

The SAP ODP batch source plugin version 0.11.3 is available in Cloud Data Fusion versions 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage. To address the issue, you must upgrade the Cloud Storage client library to version 2.3.0 or later.

• Fixed an issue causing memory errors in the SAP system. You can choose to load changed data without loading historical data first. You can select this option in the plugin properties.

The Cloud Data Fusion version 6.10.1.1 patch revision is generally available (GA). 6.10.1.1 includes the following changes:

• Cloud Data Fusion stores lineage-related information for 30 days by default, in addition to cleaning up run records (CDAP-21053).
• Added support to disable Field level lineage. For more information, see Explore Data Lineage using metadata (CDAP-21007).
• Cloud Data Fusion supports running pipelines using Dataproc 2.2. For more information about its limitations in Cloud Data Fusion, which are unchanged from Dataproc 2.1, see Change the Dataproc image to version 2.1 (CDAP-21047). This note is incorrect; see entry for October 24, 2024.

The Cloud Data Fusion version 6.9.2.4 patch revision is GA. 6.9.2.4 includes the following changes:

• Cloud Data Fusion stores lineage-related information for 30 days by default, in addition to cleaning up run records (CDAP-21053).
• Added support to disable Field level lineage. For more information, see Explore Data Lineage using metadata (CDAP-21007).
• Fixed an issue causing the maximum concurrent runs setting not to work as expected for scheduled pipeline runs (CDAP-20988).
• Fixed an issue causing upgrades to fail when the schedule name had hyphens, spaces, or other symbols (CDAP-20999).

The SAP SLT No RFC Replication plugin version 0.11.3 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The CloudSQL MySQL plugin version 1.10.7 is available in Cloud Data Fusion versions 6.9.0 and 6.10.0. This plugin version lets you use a macro to specify the name of the CloudSQL instance in the plugin's Connection name field.

Cloud Data Fusion version 6.8 is no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

Excel plugin version 2.12.3 is available in Cloud Data Fusion 6.10.0 and later. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

Excel plugin version 2.11.5 is available in Cloud Data Fusion 6.9 versions. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

Excel plugin version 2.10.3 is available in Cloud Data Fusion 6.8 versions. This version fixes an issue in the Excel batch source causing pipelines with large XLSX files to consume high memory and fail (PLUGIN-1771 and PLUGIN-1795).

The Python transform plugin version 2.3.1 is available in Cloud Data Fusion version 6.10.1. The release lets you use a macro in the Script plugin property field to pass runtime arguments (CDAP-21054).

Cloud Data Fusion versions 6.9 and later store pipeline run records for 30 days by default. For more information, see View run records.

The Cloud Storage Copy/Move plugin version 0.23.2, which is bundled with Google Cloud Platform plugin, is available in Cloud Data Fusion versions 6.10.0 and later. The release lets you use a wildcard character (*) in the source path to copy and move multiple files. For example, the source path gs://demo0/prod/reports/*.csv copies and moves all CSV files in the reports directory (PLUGIN-698).

You can configure maintenance windows for Cloud Data Fusion instances, in versions 6.8 and later, in Preview.

The Cloud Storage Multi File sink plugin version 0.23.2 is available in Cloud Data Fusion version 6.10.1 and later. The release fixes an issue in the Cloud Storage Multi File sink causing pipelines to fail when the Flexible schema property was set to true (PLUGIN-1780).

The Oracle sink plugin version 1.10.7 is available in Cloud Data Fusion version 6.9. The release fixes an issue in the Oracle sink causing null values to be assigned to fields in the input schema that have lowercase letters in the field name (PLUGIN-1793).

The Google Sheets plugin version 1.4.3, which is bundled with the Google Drive plugins, is available in the Cloud Data Fusion Hub. The release includes the following changes:

• Fixed an issue causing the Google Sheets plugin to incorrectly parse column names that have special characters (PLUGIN-1785).

• Fixed an issue causing pipelines to fail when the Google Sheets plugin is used with Wrangler and any of the fields required to fetch schema was a macro (PLUGIN-1791).

Cloud Data Fusion supports annotating resources with tags in Preview. For more information, see the Tags overview and Control access with tags.

The Decompress plugin version 1.2.1 is available in the Hub in Cloud Data Fusion version 6.10.1 and later. The release fixes an issue in the Decompress plugin causing concatenated GZIP files (.gz) to not decompress as intended. In version 1.2.1, decompression occurs until EOF is reached (PLUGIN-1743).

The Cloud Storage Multi File sink plugin version 0.22.8 is available in Cloud Data Fusion version 6.9.2. The release fixes an issue in the Cloud Storage Multi File sink causing pipelines to fail when a Flexible schema was set to true (PLUGIN-1780).

Cloud Data Fusion version 6.10.1 is generally available (GA). This release is in parallel with the CDAP 6.10.1 release.

Creating a private instance with Private Service Connect is GA in Cloud Data Fusion version 6.10.1.

Per Namespace Service Accounts are GA in Cloud Data Fusion version 6.10.1. For more information, see Access control with namespace service accounts.

Syncing multiple pipelines from a namespace is GA in Cloud Data Fusion version 6.10.1, For more information, see Sync Cloud Data Fusion pipelines with a remote repository.

The SAP SuccessFactors batch source version 1.2.3 is available in the Enterprise edition of Cloud Data Fusion 6.7.0 and later. This release lets you configure a proxy URL and SuccessFactors authentication properties.

Fixed in Cloud Data Fusion 6.10.1:

• Fixed an issue causing runtime arguments of pipeline triggers to not propagate to downstream pipelines (CDAP-20947).
• Fixed an issue in Wrangler causing the send-to-error-and-continue directive to not initialize dq_failure when the condition is false (PLUGIN-1736).
• Fixed an issue that occurs if running a replication pipeline when task workers are enabled (CDAP-20951).
• Improved error reporting in the BigQuery Sink. Fixed an issue in BigQuery Argument Setter where validation error wasn't displayed correctly (PLUGIN-788, PLUGIN-781, PLUGIN-782, PLUGIN-1318).
• Improved retries in BigQuery plugin (PLUGIN-1715).
• Fixed an issue with the Python plugin, where running in native mode doesn't work as intended (PLUGIN-1617).
• Fixed an issue causing certain connection parameters to not propagate in a MySQL connection (PLUGIN-1728).
• Fixed an issue causing the Cloud Storage Copy action to timeout while working with large files (PLUGIN-1735).
• Fixed an issue causing Copy and Move plugins to not create buckets at the destination path as expected, resulting in a runtime error (PLUGIN-1738).
• Fixed an issue causing empty source input to fail in multiple plugins (PLUGIN-1742).
• Fixed an issue with remote execution of Wrangler directives causing type information to not be emitted (PLUGIN-1778).
• Fixed an issue causing a No record field providederror (CDAP-21024).
• Streaming pipelines in Cloud Data Fusion support the Excel source. Batch pipelines with an Excel source can consume high memory and fail in large pipelines (PLUGIN-1771).
• Fixed an issue with using the Conditional plugin as a source for Wrangler, causing CDAP not to fetch the necessary schema (CDAP-20890).
• Fixed an issue with instance upgrades causing existing schedule names to be improperly encoded in the URL, resulting in pre-upgrade failure (CDAP-20999).
• Fixed an issue with schedules causing the maximum concurrent run property to not work as intended (CDAP-20988).
• Fixed an issue causing committed ID to incorrectly propagate when pushing pipeline configurations to Git (CDAP-20932).

Changed in Cloud Data Fusion 6.10.1:

• Source Control Management supports Bitbucket and Gitlab.
• Cloud Data Fusion uses the subnet used by the shared VPC network attachment in the default compute profile.
• Added support for option string field (keep-strings) in parse-xml-to-json Wrangler directive (CDAP-20934).
• The BigQuery sink plugin doesn't provide the Dedupe By option while in insert mode (PLUGIN-900).
• The BigQuery plugin supports the JSON type (PLUGIN-1563).
• Improved error messages in the Spanner source (PLUGIN-1748).
• Improved retries in Pub/Sub plugin (PLUGIN-1769).

Cloud Data Fusion version 6.10.1 has a known issue in the Cloud Storage plugin causing pipelines to intermittently fail if the plugin contains a * regex pattern and uses Dataproc 2.0. To mitigate this issue:

• Change the Dataproc image to version 2.1.
• Use an older plugin version.
• Increase memory for the executor.

The HTTP plugin (version 1.4.2) is available in Cloud Data Fusion versions 6.8.0 and later. The release fixed an issue in the HTTP source causing an error in the retrieved schema when one of the retrieved columns contained a quoted value with a delimiter, such as a comma (PLUGIN-1781).

The Salesforce Marketing Cloud plugin (version 1.3.1) is available in Cloud Data Fusion version 6.8.0 and later. The release fixed an issue in the Salesforce Marketing sink plugin causing upsert operations to fail (PLUGIN-1773).

Cloud Data Fusion is available in the africa-south1 region. For more information, see Pricing.

The Google Sheets plugin version 1.4.2 (bundled with the Google Drive plugins) is available in all Cloud Data Fusion versions. The release includes the following changes:

• Macros are supported for the following OAuth fields: Client ID, Client secret, and Refresh token (PLUGIN-1762).
• You can specify a single file ID in the File identifier field (PLUGIN-1763).
• Added an Access token field, which supports macros (PLUGIN-1764).
• You can turn on auto detection for the number of rows and columns (PLUGIN-1766).

The Amazon Redshift batch source connector version 1.11.1 is available in Preview in Cloud Data Fusion 6.10.0 and later. This source lets you load batch data from your Redshift dataset to a destination, such as BigQuery.

Cloud Data Fusion is available in the following regions:

• asia-south2
• me-central2

For more information, see Pricing.

The Amazon Redshift batch source connector version 1.10.6 is available in Preview in Cloud Data Fusion 6.9 versions. This source lets you load batch data from your Redshift dataset to a destination, such as BigQuery.

The SAP Ariba batch source plugin version 1.3.0 is available in the Cloud Data Fusion Enterprise edition, versions 6.7.0 and later. In plugin version 1.3.0, if a failure occurs, the plugin retries API calls. You can configure the retry parameters using the hidden fields in the pipeline configuration JSON.

Salesforce plugin version 1.6.3 is available in Cloud Data Fusion versions 6.8.0 and later. The release includes the following changes:

• Fixed an issue in the Salesforce sink plugin causing an unsupported type datetime error for DateTime type fields in the input schema. In this version, the Salesforce sink plugin supports datetime and decimal logical types (PLUGIN-1749).
• Fixed an issue in all Salesforce plugins causing a pipeline to fail when using an OAuth macro because the oAuth macro value didn't get passed to the plugin as intended. In this version, all Salesforce plugins support an OAuth macro (PLUGIN-1767).
• At the time of failure on the Salesforce sink side, if the Error handling property is set to the Fail on error option, the Salesforce job is aborted, which stops newer batches from being added to the job due to spark retry settings in CDAP (PLUGIN-1768).
• To make debugging easier, additional debug logs and batch results in logs are available.

The Cloud Data Fusion version 6.9.2.3 patch revision is generally available (GA). 6.9.2.3 includes the following fixes:

• Skipped running MetadataConsumerSubscriberService when Dataplex Data Lineage Integration is disabled (CDAP-20947).

• Fixed an issue causing runtime arguments of pipeline triggers to not propagate to downstream pipelines (CDAP-20943).

• Fixed an issue causing pipelines to fail in starting state when the system worker service is intermittently unavailable (CDAP-20956).

• Fixed an issue causing pipelines to fail in starting state when the Compute Engine metadata server is intermittently unavailable (CDAP-20955).

Cloud Data Fusion version 6.7 is no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

Cloud Data Fusion is available in the following regions:

• europe-southwest1
• europe-west8
• europe-west9
• europe-west12
• me-central1
• southamerica-west1

For more information, see Pricing.

Cloud Data Fusion supports patch revision upgrades in the Google Cloud console. For more information, see Manage patch revisions for instances.

Cloud Data Fusion lets you enable and disable Dataplex Lineage, as needed. When you create a new instance in Cloud Data Fusion version 6.8.0 and later, Dataplex Lineage is disabled by default. For more information, see View lineage in Dataplex.

Cloud Data Fusion version 6.10.0 is available in Preview. This release is in parallel with the CDAP 6.10.0 release.

Source control management using GitHub is generally available (GA) in Cloud Data Fusion version 6.10.0. With this feature, you can use GitHub to maintain version histories of your ETL and ELT pipelines.

To simplify the experience of synchronizing pipelines between Cloud Data Fusion and GitHub in bulk, pushing and pulling multiple pipelines is available in Preview.

In versions before 6.10.0, you can only establish connectivity between private Cloud Data Fusion instances and customer resources with VPC Peering. Starting in version 6.10.0, you can use Private Service Connect interfaces to establish connectivity using internal IP addresses. Private Service Connect interfaces are easier to set up, use fewer internal IP addresses, and don't require the use of proxies to reach customer resources in peered VPCs, the internet, or on-premises.

Changes in Cloud Data Fusion 6.10.0:

• Role-based access control in Cloud Data Fusion supports using the Cloud Data Fusion API Service Agent service account while emitting lineage events in RBAC-enabled instances. For more information, see View lineage in Dataplex.
• To create ephemeral clusters, Cloud Data Fusion uses the Dataproc 2.1 image by default. For more information about its limitations in Cloud Data Fusion, see Change the Dataproc image to version 2.1 (CDAP-20841).
• Cloud Data Fusion uses the autoscaling Dataproc profile by default for ephemeral clusters.
• In Source control management, GitHub PAT was removed from the Cloud Data Fusion web interface for repository configurations (CDAP-20904 and CDAP-20581).
• Improved latency when BigQuery pushdown is enabled by fetching artifacts from a local cache (CDAP-20846).
• The BigQuery sink supports flexible table names and column names (PLUGIN-1718).
• BigQuery sinks support ingesting data to JSON data type fields(PLUGIN-1692).
• In BigQuery sink jobs, you can add labels in the form of key-value pairs (PLUGIN-1705).
• In BigQuery execute jobs, you can add labels in the form of key-value pairs (PLUGIN-1729).
• Added capability to run concurrent tasks in task workers (CDAP-20868).
• If a task is stuck, task workers are forcefully restarted (CDAP-20831).
• The Cloud Storage Java Client is upgraded to version 2.3 and later (PLUGIN-1293).
• Wrangler is schema aware (CDAP-15361).
• Added validation for incorrect credentials in the Amazon S3 source (PLUGIN-1694).

You can control access to Google Cloud resources at the Cloud Data Fusion namespace level using an IAM service account. Per Namespace Service Accounts are available in Preview in Cloud Data Fusion version 6.10.0. For more information, see Access control with namespace service accounts.

To support the Private Service Connect integration, the following permissions are added to the Cloud Data Fusion API Service Agent role:

• compute.networkAttachments.get
• compute.networkAttachments.update
• compute.networkAttachments.list

Fixed in Cloud Data Fusion 6.10.0:

• Fixed an issue in the Postgres DB plugin causing macros to be unsupported for database configuration (PLUGIN-1681).
• Fixed an issue causing slowness in the API while fetching runs for all applications in a namespace (CDAP-20587).
• Made the following fixes to Wrangler grammar (CDAP-20839):
  • The NUMERIC token type supports negative numbers.
  • The PROPERTIES token type supports one or more properties.
• Fixed an issue causing columns that have all null values to be dropped in Wrangler (CDAP-20521).
• Fixed an issue causing pipeline upgrades to not have the intended description (CDAP-20815).

The Spark compute engine running on Scala 2.11 is not supported in Cloud Data Fusion version 6.10.0.

Dataproc 1.5 isn't supported in Cloud Data Fusion version 6.10.0.

The SAP BW Open Hub batch source plugin version 0.11.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes:

• Fixed an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage. To address the issue, you must upgrade the Google Storage Client library to version 2.3.0 or later.

• Add a new plugin property, Max. retries for read enqueue.

The Salesforce plugin version 1.6.2 is available in Cloud Data Fusion versions 6.8.0 and later. This version includes the following changes:

• Fixed an issue in the Salesforce plugin causing the following error in some pipelines that run more than 4 hours: java.lang.IllegalStateException: SSLException reading next record: javax.net.ssl.SSLException: Connection reset. The Connection timeout property was added to the Salesforce plugin properties in the web interface with the default value of 3600 seconds (PLUGIN-1719).

• For accuracy, fixed schema handling for referenced object fields: child fields are explicitly marked as non-nullable, regardless of the schema values in the referenced object.

In earlier versions, when retrieving schema information for fields in referenced objects, such as contact.account_lastmodifieddate, the schema inherits properties from the referenced object, causing incorrect non-nullable assumptions (PLUGIN-1720).

• A retry mechanism was added in the Salesforce batch source and Multi-Source plugins for connection timeout issues (PLUGIN-1706).

The SAP ODP batch source plugin version 0.11.0 is available in Cloud Data Fusion versions 6.8.3 and later. The release includes the following changes:

• Improved error logging, providing more information about the occurrence and mode of errors.
• Improved the placeholder values in the Filter options fields.
• Added support for fetching Operational Delta Queue (ODQ) fields by turning on the Fetch delta fields toggle.
• On the Properties page for the plugin, changed the name of the ObjectType field to Context. For more information, see the ODP batch source properties.

The Cloud Data Fusion version 6.9.2.2 patch revision is generally available (GA). 6.9.2.2 includes the following fixes:

• Increased the speed of the batch /runs API call for pipelines that are run thousands of times (CDAP-20587).
• Fixed an issue causing draft pipelines to load incorrectly when you enable an accelerator that automatically installs plugins from a custom hub (CDAP-20628).
• Fixed an issue causing the CDAP service IP to be cached forever (CDAP-20781).
• Fixed an issue causing a KubeTwillRunnerService error on shutdown (CDAP-20792).
• Fixed an issue causing pipelines and streaming jobs to fail after an AppFabric restart. (CDAP-20797).
• Fixed an issue causing a slowdown in deploying applications (CDAP-20820).
• Fixed an issue causing refreshed OAuth tokens to not be logged as expected, making it difficult to identify the root cause of some issues (CDAP-20861).
• Fixed an issue in replication where you couldn't select the Tink transformation in the web interface. (CDAP-20804).

Cloud Data Fusion versions 6.5 and 6.6 are no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

You can update the patch revision version of an instance by adding the optional --patch_revision argument to the gcloud beta data-fusion instances update command. For more information, see Manage patch revisions for instances.

You can apply a patch revision version when you create a new Cloud Data Fusion instance by adding the optional --patch_revision argument to the gcloud beta data-fusion instances create command. For more information, see Manage patch revisions for instances.

The Cloud Data Fusion version 6.9.2.1 patch revision is generally available (GA). It fixes a regression that causes a pipeline to fail when using Dataproc secondary workers (CDAP-20807).

The Cloud Data Fusion version 6.8.3.1 patch revision is generally available (GA). It fixes a regression that causes a pipeline to fail when using Dataproc secondary workers (CDAP-20807).

Cloud Data Fusion supports patch revisions. These revisions apply bug fixes between major releases. For more information, see Versioning in Cloud Data Fusion.

The Cloud Data Fusion version 6.7.3.1 patch revision is generally available (GA). It introduces retries in the platform transaction layer to handle a PSQLException error thrown from broken database connections (CDAP-19949 and CDAP-20722).

The BigQuery Replication plugin versions 0.8.4 and 0.7.3 are available in Cloud Data Fusion Enterprise edition versions 6.9 and 6.8, respectively. These releases include the following change: the BigQuery Replication sink lets you write data to other projects.

The Cloud Data Fusion SAP SLT No RFC Replication plugin version 0.11.0 is available in the Hub in Cloud Data Fusion enterprise edition versions 6.8.0 and later. It differs from the existing SAP SLT Replication plugin in the following ways:

• All data and metadata file formats are in JSON.
• No SAP RFC inbound calls occur in the SAP SLT No RFC Replication plugin. Accessing schemas and data from the SAP system no longer requires an SAP connection. Metadata and data extraction are sourced from the Cloud Storage bucket.

Cloud Data Fusion version 6.9.2 is generally available (GA). This release is in parallel with the CDAP 6.9.2 release.

Features in Cloud Data Fusion 6.9.2:

• Editing deployed batch pipelines is generally available (GA).
• Using Pub/Sub sources in streaming data pipelines is GA.
• Wrangler Filter Pushdown is GA.
• Pushdown for window aggregations is GA.

Changes in Cloud Data Fusion 6.9.2:

• Cloud Data Fusion supports setting custom scopes when creating a Dataproc cluster (CDAP-19428).
• You can set common metadata labels for Dataproc clusters and jobs using the Common Labels property in the Ephemeral Dataproc compute profile (CDAP-20698).
• You can set labels for the Dataproc jobs using the Common Labels property in the Existing Dataproc compute profile (CDAP-20698).
• You can set a pipeline runtime argument with the key system.profile.properties.labels and a value representing the labels in the following format: key1|value1;key2|value2. This setting overrides the common labels set in the compute profile for pipeline runs (CDAP-20698).
• Cloud Data Fusion supports using Dataproc temp buckets in compute profiles (CDAP-20712).

Fixed in Cloud Data Fusion 6.9.2:

• Added a retry for Pub/Sub snapshot creation and deletion in a real-time pipeline with a Pub/Sub source when a retryable internal error is thrown (PLUGIN-1660).
• Fixed a bug causing the Dynamic Spark plugins to fail when running on Dataproc 1.5 (CDAP-20674).
• Fixed a discrepancy in warning and error counts reported between the pipeline summary tab and system logs (CDAP-20680).
• Fixed a problem when, in rare cases, a cluster couldn't be found with Cluster Reuse (CDAP-20759).
• Fixed a bug causing the JavaScript transform to fail on Dataproc 2.1 (CDAP-20778).

The SAP ODP plugin version 0.7.5 is available in Cloud Data Fusion versions 6.6.0 to 6.8.0. This release includes the following changes:

• Fixed an issue causing duplicate records or loss of records due to package acknowledgement occurring too early.
• Filters that you apply are viewable in the logs.