Managed Service for Apache Spark release notes

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.119-debian11, 2.1.119-rocky8, 2.1.119-ubuntu20, 2.1.119-ubuntu20-arm
• 2.2.87-debian12, 2.2.87-rocky9, 2.2.87-ubuntu22, 2.2.87-ubuntu22-arm
• 2.3.36-debian12, 2.3.36-ml-ubuntu22, 2.3.36-rocky9, 2.3.36-ubuntu22, 2.3.36-ubuntu22-arm
• 3.0.2-debian13, 3.0.2-ml-ubuntu24, 3.0.2-rocky9, 3.0.2-ubuntu24

Key updates in these image versions include:

• Feature updates:
  • Apache Hudi: Added support for the Apache Hudi optional component in 3.0 images. Version 1.2.0 is available in 3.0 images.
  • Parquet footer caching: Enabled Parquet footer caching by default for Lightning Engine (Velox). If executor out-of-memory (OOM) or task failure spikes on tiny files are observed, these can be mitigated by setting spark.gluten.sql.columnar.backend.velox.cacheParquetFooters=false.
  • Apache Iceberg 1.10: Added support for Apache Iceberg 1.10 in 2.2 images. Users can opt-in by setting the cluster property, dataproc:dataproc.iceberg.version=1.10.
  • Lakehouse catalog: Lakehouse catalog auto-loading is supported for image versions 2.2 and later.
• Library updates:
  • Cloud Storage connector: Upgraded the Cloud Storage connector to 4.0.4 in 3.0 images.
  • OpenLineage: Upgraded OpenLineage to 1.49 in 3.0 images to support lineage for tables created using the Lakehouse Runtime catalog.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

• Preconfigured Conda channels removed: Preconfigured conda channels (such as conda-forge) have been removed from configurations. All image version aliases now point to the latest image without conda channel configuration. Support for earlier images with conda channels configuration to be announced in upcoming release notes. Recommendation: Migrate to the latest image versions as soon as possible.
  • Affected image versions: 1.3.96+, 1.4.81+, 1.5.92+, 2.0.161+, 2.1.119+, 2.2.87+, and 2.3.36+.
  • Impact: Additional conda package installation using the dataproc:conda.packages cluster property or direct conda install <package> command will fail.
  • Workaround:
    • Use the <channel>::<package>==<version> specification for the dataproc:conda.packages property (for example: dataproc:conda.packages=conda-forge::pip==24.0).
    • Specify the channel name on the command line when running conda install (for example: conda install <packages> -c conda-forge).
• google-guest-agent upgraded: Upgraded google-guest-agent on Debian and Rocky Linux images (remediating CVE-2026-33186).
• SSH metadata restriction: Stricter SSH metadata restrictions are enforced by default; metadata SSH keys mapped directly to root are ignored. Connect using a standard non-root user (such as dataproc) with sudo, or use Google Cloud OS Login.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings with the Lakehouse Runtime catalog.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on August 19, 2026 have been rolled back:

• 2.2.86-debian12, 2.2.86-rocky9, 2.2.86-ubuntu22, 2.2.86-ubuntu22-arm

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.118-debian11, 2.1.118-rocky8, 2.1.118-ubuntu20, 2.1.118-ubuntu20-arm
• 2.2.86-debian12, 2.2.86-rocky9, 2.2.86-ubuntu22, 2.2.86-ubuntu22-arm
• 2.3.35-debian12, 2.3.35-ml-ubuntu22, 2.3.35-rocky9, 2.3.35-ubuntu22, 2.3.35-ubuntu22-arm
• 3.0.1-debian13, 3.0.1-ml-ubuntu24, 3.0.1-rocky9, 3.0.1-ubuntu24

Key updates in these image versions include:

• Iceberg updates: In the 2.3 image version, 2.3 clusters with Lightning Engine now use Iceberg version 1.10 by default.
• OpenLineage updates: In the 2.2 and 2.3 image versions:
  • Upgraded OpenLineage to version 1.49 to support lineage for tables created using the Lakehouse Runtime catalog.

Rollback Notice: The 2.2.86 image versions were rolled back on August 31, 2026.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.86
• 2.2.86
• 2.3.39

Key updates in these runtime versions include:

• OpenLineage updates: In the 2.3 runtime:
  • Upgraded OpenLineage to version 1.49 to support lineage for tables created using the Lakehouse Runtime catalog.
  • Fixed a segmentation fault when OpenLineage parses complex SQL query strings.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

A critical bug related to Conda channels has been fixed in-place in image versions 1.4.81, 1.5.92, 2.1.117, and 2.2.85. These image versions were released without pre-configured Conda channels.

Required customer actions: To comply with Google requirements, recreate the following resources if they were created using these image versions on or before August 10, 2026:

• Custom images
• Clusters

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.85
• 2.2.85
• 2.3.38

Notes:

• Apache Spark upgraded to 3.5.3 in 2.2 runtime.

• Apache Gluten upgraded to 1.6 in 2.3 runtime.

Managed Service for Apache Spark latest image and runtime versions:

• Configured spark.scheduler.listenerbus.exitTimeout to 30s.

New Managed Service for Apache Spark on Google Kubernetes Engine (formerly Dataproc on Google Kubernetes Engine) subminor image version:

• 3.5-dataproc-28

Key updates in this image version include:

• Conda channels: The new 3.5-dataproc-28 subminor image version doesn't have preconfigured Conda channels, and is mapped to default aliases (such as 3.5 and latest).
  • Impact: When creating clusters with 3.5-dataproc-28 or using default aliases (3.5, latest), packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels, pin your clusters to the previous image versions before August 25, 2026.
  • Default change schedule: All workloads must transition to image versions without preconfigured Conda channels after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Starting with Managed Service for Apache Spark image version 3.0, clusters that are created without a specified machine type for a node are created with a Flex VM configuration for the node.

Announcing the General Availability (GA) release of Managed Service for Apache Spark cluster image version 3.0, as follows:

• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, and 3.0.0-ubuntu24.

  • The 3.0.0-ml-ubuntu24 image extends the 3.0 base image with ML-specific libraries.

Image version 3.0 is a lightweight image that contains the following pre-installed core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs):

• Apache Hadoop 3.5.0.
• Apache Hive 4.2.0
• Apache Spark 4.1.2
• Apache Tez 0.10.5
• Cloud Storage Connector 3.1.13
• Conscrypt 2.6
• Java 21
• Python 3.12
• R 4.5
• Scala 2.13.17
• Spark-BigQuery Connector 0.44.1-preview

You can add listed optional components when you create a 3.0 image version cluster.

Recommendation: Use the 3.0 (and later) image versions to meet security compliance requirements.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.34-debian12, 2.3.34-ml-ubuntu22, 2.3.34-rocky9, 2.3.34-ubuntu22, 2.3.34-ubuntu22-arm
• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, 3.0.0-ubuntu24

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

• The 2.1, 2.2 and 2.3 cluster image versions now support Confidential Compute for the g4-standard-48 GPU machine type.

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):

• The 3.0 runtime now uses fewer executors, as follows:

  • 0 min executors for spark.dynamicAllocation.minExecutors property
  • 1 min executor for spark.executor.instances and spark.dynamicAllocation.initialExecutors properties

• All runtimes now configure spark.scheduler.listenerbus.exitTimeout to 30 seconds.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.117-debian11, 2.1.117-rocky8, 2.1.117-ubuntu20, 2.1.117-ubuntu20-arm
• 2.2.85-debian12, 2.2.85-rocky9, 2.2.85-ubuntu22, 2.2.85-ubuntu22-arm
• 2.3.33-debian12, 2.3.33-ml-ubuntu22, 2.3.33-rocky9, 2.3.33-ubuntu22, 2.3.33-ubuntu22-arm

Key updates in these image versions include:

• Conda channels: The new 2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
  • Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions (for example, 2.1.116-debian11 or 2.2.84-debian12).
  • Default change schedule: After August 25, 2026, the default aliases for 2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32, don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026. The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels, but note the following Default change schedule:, which will impact all supported image versions.

• Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
• Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions.
• Default change schedule: The subminor versions 1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026. Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases. All workloads must use new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 1.3.96-debian10, 1.3.96-ubuntu18
• 1.4.81-debian10, 1.4.81-ubuntu18
• 1.5.92-debian10, 1.5.92-rocky8, 1.5.92-ubuntu18
• 2.0.161-debian10, 2.0.161-rocky8, 2.0.161-ubuntu18
• 2.1.115-debian11, 2.1.115-rocky8, 2.1.115-ubuntu20, 2.1.115-ubuntu20-arm
• 2.1.116-debian11, 2.1.116-rocky8, 2.1.116-ubuntu20, 2.1.116-ubuntu20-arm
• 2.2.83-debian12, 2.2.83-rocky9, 2.2.83-ubuntu22, 2.2.83-ubuntu22-arm
• 2.2.84-debian12, 2.2.84-rocky9, 2.2.84-ubuntu22, 2.2.84-ubuntu22-arm
• 2.3.31-debian12, 2.3.31-ml-ubuntu22, 2.3.31-rocky9, 2.3.31-ubuntu22, 2.3.31-ubuntu22-arm
• 2.3.32-debian12, 2.3.32-ml-ubuntu22, 2.3.32-rocky9, 2.3.32-ubuntu22, 2.3.32-ubuntu22-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:

• Iceberg support: Added support for Iceberg 1.10 in Dataproc 2.3 images. This change only applies to 2.3 clusters created with the ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.
• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default in new GCE image versions.
• Scheduler exitTimeout: Configured spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.82
• 2.2.82
• 2.3.35

Key updates in these runtime versions include:

• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.

• Introduced confidentialInstanceType enum in the API.
• The --enable-confidential-compute flag and enableConfidentialCompute field are deprecated in favor of the new type-specific flag/field.
• Clusters created with the deprecated boolean flag will default to SEV.
• Added validation for machine type compatibility for SEV, SEV-SNP, and TDX.
• Updated live migration logic to support compatible machine types and CPU platforms for each technology, including N2D and C3D for SEV.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on May 19, 2026 have been rolled back:

• 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm
• 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm
• 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm

Rollback Notice: The 2.2.82 image versions were rolled back on May 25, 2026.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.

This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.

• Impact: This change only affects users who are programmatically setting the configuration in code using spark.conf.set() with an integer literal.
  • Impacted example: spark.conf.set("spark.sql.shuffle.partitions", 100)
• User action: Update your code to pass a string literal instead of an integer.
  • Example fix: spark.conf.set("spark.sql.shuffle.partitions", "100")
• Not impacted: Setting the configuration via command-line arguments (e.g., spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.80
• 2.2.80
• 2.3.33

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):

• 3.0.0-RC2-debian13, 3.0.0-RC2-ml-ubuntu24, 3.0.0-RC2-rocky9, 3.0.0-RC2-ubuntu24

Key updates in this release:

• The debian image for 3.0 is upgraded to Debian 13.
• The default Java runtime is upgraded to Java 21.
• Apache Hadoop upgraded to 3.5.0.
• Apache Spark upgraded to 4.1.1.
• Apache Hive upgraded to 4.2.0.
• Trino upgraded to 480.
• Apache Solr upgraded to 9.10.1.
• Apache Kafka upgraded to 3.9.2.
• Docker upgraded to 28.1.
• Apache Flink upgraded to 2.2.0.
• Scala upgraded to 2.13.17.
• Cloud Storage Connector upgraded to 3.1.13.
• Apache Zookeeper upgraded to 3.9.5.
• BigQuery Connector upgraded to 0.44.1-Preview.
• Zeppelin Notebook upgraded to 0.12.0.
• JupyterLab upgraded to 4.5.7.
• Pixi is used as python package manager instead of conda.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.79
• 2.2.79
• 2.3.32

Key updates in this release:

• Upgraded Metastore Proxy version to v0.0.79.
• Upgraded Spark RAPIDS to version 26.04.0 version in the Managed Service for Apache Spark version 3.0 serverless runtime.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.113-debian11, 2.1.113-rocky8, 2.1.113-ubuntu20, 2.1.113-ubuntu20-arm
• 2.2.81-debian12, 2.2.81-rocky9, 2.2.81-ubuntu22, 2.2.81-ubuntu22-arm
• 2.3.29-debian12, 2.3.29-ml-ubuntu22, 2.3.29-rocky9, 2.3.29-ubuntu22, 2.3.29-ubuntu22-arm

Key updates in this release:

• The native runtime engine is now supported on base Ubuntu cluster images starting from image version 2.3.29-ubuntu22 (not supported on Arm and ML images).

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:

• Flexible VMs: Minimize stockouts and improve machine obtainability by defining prioritized lists for your master, primary, and secondary worker VM types. Managed Spark for Apache Spark utilizes these lists to create your cluster, and selects the optimal VM type based on current capacity, quotas, and existing reservations.
• Cluster Scheduled Stop: Optimize cost and maintain your cluster configuration by stopping clusters after a specified idle period, at a specified future time, or after a specified period from cluster creation or a cluster update request. All custom cluster configurations are restored once you restart the cluster.
• Zero-scale clusters: Reduce costs by creating only secondary workers, scaling them down to zero when they are not in use.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.78
• 2.2.78
• 2.3.31

Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.28-debian12, 2.3.28-ml-ubuntu22, 2.3.28-rocky9, 2.3.28-ubuntu22, 2.3.28-ubuntu22-arm

• Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Upgraded Apache Zookeeper to version 3.9.5 in image version 2.3.

• Upgraded Dataproc Metastore Proxy to v0.0.79 to fix CVEs.
• Fixed CVEs CVE-2026-24308 and CVE-2026-24281.

New Serverless for Apache Spark runtime versions:

• 1.2.77
• 2.2.77
• 2.3.30
• 3.0.13

New Dataproc on Compute Engine subminor image versions:

• 2.1.112-debian11, 2.1.112-rocky8, 2.1.112-ubuntu20, 2.1.112-ubuntu20-arm
• 2.2.80-debian12, 2.2.80-rocky9, 2.2.80-ubuntu22, 2.2.80-ubuntu22-arm
• 2.3.27-debian12, 2.3.27-ml-ubuntu22, 2.3.27-rocky9, 2.3.27-ubuntu22, 2.3.27-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.76
• 2.2.76
• 2.3.29
• 3.0.12

New Dataproc on Compute Engine subminor image versions:

• 2.1.111-debian11, 2.1.111-rocky8, 2.1.111-ubuntu20, 2.1.111-ubuntu20-arm
• 2.2.79-debian12, 2.2.79-rocky9, 2.2.79-ubuntu22, 2.2.79-ubuntu22-arm
• 2.3.26-debian12, 2.3.26-ml-ubuntu22, 2.3.26-rocky9, 2.3.26-ubuntu22, 2.3.26-ubuntu22-arm

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924, and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.
• Default JDK is set to Temurin JDK - 11.0.30 in all 2.1, 2.2 and 2.3 images.

New Serverless for Apache Spark runtime versions:

• 1.2.75
• 2.2.75
• 2.3.28
• 3.0.11

Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

Rollback Notice: These image versions were rolled back on March 11, 2026.

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924 (partial), and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.

New Serverless for Apache Spark runtime versions:

• 1.2.74
• 2.2.74
• 2.3.27
• 3.0.10

Serverless for Apache Spark 3.0 runtime version:

• Updated Spark version to 4.0.1.
• Updated BigQuery Spark Connector version to 0.44.0.
• Applied patch for SPARK-52601.
• Applied patch for SPARK-52614.

Serverless for Apache Spark 1.2, 2.2 and 2.3 runtime versions:

• Updated Delta Lake version to 3.2.1.

New Serverless for Apache Spark runtime versions:

• 1.2.73
• 2.2.73
• 2.3.26

New Serverless for Apache Spark runtime versions:

• 1.2.72
• 2.2.72
• 2.3.25
• 3.0.9

New Dataproc on Compute Engine subminor image versions:

• 2.0.160-debian10, 2.0.160-rocky8, 2.0.160-ubuntu18
• 2.1.109-debian11, 2.1.109-rocky8, 2.1.109-ubuntu20, 2.1.109-ubuntu20-arm
• 2.2.77-debian12, 2.2.77-rocky9, 2.2.77-ubuntu22, 2.2.77-ubuntu22-arm
• 2.3.24-debian12, 2.3.24-ml-ubuntu22, 2.3.24-rocky9, 2.3.24-ubuntu22, 2.3.24-ubuntu22-arm

Fixed an issue that could cause gsutil failures on clusters using custom images built from new base images. This problem occurred when the gcloud core/universe_domain property was not set in the build environment, resulting in a malformed /etc/boto.cfg. The image build process now ensures a correct default configuration, restoring proper Google Cloud Storage connectivity.

New Serverless for Apache Spark runtime versions:

• 1.2.71
• 2.2.71
• 2.3.24
• 3.0.8

Hyperdisks can now be added to a Dataproc cluster.

New Dataproc on Compute Engine subminor image versions:

• 2.0.159-debian10, 2.0.159-rocky8, 2.0.159-ubuntu18
• 2.1.108-debian11, 2.1.108-rocky8, 2.1.108-ubuntu20, 2.1.108-ubuntu20-arm
• 2.2.76-debian12, 2.2.76-rocky9, 2.2.76-ubuntu22, 2.2.76-ubuntu22-arm
• 2.3.23-debian12, 2.3.23-ml-ubuntu22, 2.3.23-rocky9, 2.3.23-ubuntu22, 2.3.23-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.70
• 2.2.70
• 2.3.23
• 3.0.7

New Serverless for Apache Spark runtime versions:

• 1.2.69
• 2.2.69
• 2.3.22
• 3.0.5

Serverless for Apache Spark: Added support for removing conscrypt from Serverless for Apache Spark 2.3+ runtimes using the dataproc.artifacts.remove property .

Dataproc on Compute Engine: Sharing snapshot diagnostic data: Setting the --tarball-access=GOOGLE_DATAPROC_DIAGNOSE flag with the gcloud dataproc clusters diagnose command shares all of the output Cloud Storage bucket contents with Google Cloud support if uniform bucket-level access is enabled on the output Cloud Storage bucket. If object-level access control is enabled on the output Cloud Storage bucket, only the generated diagnostic tar file is shared.

New Dataproc on Compute Engine subminor image versions:

• 2.0.158-debian10, 2.0.158-ubuntu18, 2.0.158-rocky8
• 2.1.107-debian11, 2.1.107-ubuntu20, 2.1.107-ubuntu20-arm, 2.1.107-rocky8
• 2.2.75-debian12, 2.2.75-ubuntu22, 2.2.75-ubuntu22-arm, 2.2.75-rocky9
• 2.3.22-debian12, 2.3.22-ml-ubuntu22, 2.3.22-rocky9, 2.3.22-ubuntu22, 2.3.22-ubuntu22-arm

Zookeeper version upgraded to 3.9.4 in Dataproc image 2.3.22.

Parquet CLI version upgraded to 1.15.2 in 2.1 and 2.2 images.

Delta subminor version upgraded to 3.2.1 in Dataproc on Compute Engine image 2.2 and 2.3.

Apache Pig is now available in ARM images.

Added a new dataproc:pypi.repository property to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.

Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

Upcoming Spark data lineage changes See the upcoming May, 2026 Dataproc and Serverless for Apache Spark release notes for an announcement of a change that will automatically enable Dataproc Spark data lineage and Serverless for Apache Spark data lineage when you enable the Data Lineage API (see Control lineage ingestion for a service) without requiring additional project, cluster, batch workload, or interactive session settings.

New Serverless for Apache Spark runtime versions:

• 3.0.4

Dataproc on Compute Engine: The following subminor image versions announced on January 24, 2026 have been rolled back:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

Rollback Notice: These image versions were rolled back on January 30, 2026.

Upgraded the Delta subminor version to 3.2.1 in images 2.2 and 2.3.

Apache Pig is now available in ARM images.

Added a new dataproc:pypi.repository property to customize the PyPI repository that pip uses. The value can be a internet-accessible URL, or you can specify google to use a Google-hosted cache of PyPI, which is accessible without public internet connectivity. Starting with Managed Service for Apache Spark image version 3.1, google will be the default (you can specify the pypi property value to use public PyPI instead of the default google 3.1 value).

Removed the use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

New Serverless for Apache Spark runtime versions:

• 1.2.67
• 2.2.67
• 2.3.20
• 3.0.3

• Applied patch for SPARK-48292 in Serverless for Apache Spark 1.2 and 2.2 runtimes.

Dataproc on Compute Engine: The following subminor image versions announced on January 06, 2026 have been rolled back:

• 2.0.156-debian10, 2.0.156-ubuntu18, 2.0.156-rocky8
• 2.1.105-debian11, 2.1.105-ubuntu20, 2.1.105-ubuntu20-arm, 2.1.105-rocky8
• 2.2.73-debian12, 2.2.73-ubuntu22, 2.2.73-ubuntu22-arm, 2.2.73-rocky9
• 2.3.20-debian12, 2.3.20-ml-ubuntu22, 2.3.20-rocky9, 2.3.20-ubuntu22, 2.3.20-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.66
• 2.2.66
• 2.3.19
• 3.0.1

New Dataproc on Compute Engine subminor image versions:

• 2.0.156-debian10, 2.0.156-ubuntu18, 2.0.156-rocky8
• 2.1.105-debian11, 2.1.105-ubuntu20, 2.1.105-ubuntu20-arm, 2.1.105-rocky8
• 2.2.73-debian12, 2.2.73-ubuntu22, 2.2.73-ubuntu22-arm, 2.2.73-rocky9
• 2.3.20-debian12, 2.3.20-ml-ubuntu22, 2.3.20-rocky9, 2.3.20-ubuntu22, 2.3.20-ubuntu22-arm

Rollback Notice: These image versions were rolled back on January 13, 2026.

Added a new property dataproc:pypi.repository to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.

Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

New Serverless for Apache Spark runtime versions:

• 1.2.65
• 2.2.65
• 2.3.18

The following Dataproc on Compute Engine subminor image versions, released on December 05, 2025, are now blocklisted:

• 2.2.72-debian12, 2.2.72-ubuntu22, 2.2.72-ubuntu22-arm, 2.2.72-rocky9
• 2.3.19-debian12, 2.3.19-ubuntu22, 2.3.19-ubuntu22-arm, 2.3.19-ml-ubuntu22, 2.3.19-rocky9

New Dataproc on Compute Engine subminor image versions:

• 2.0.155-debian10, 2.0.155-ubuntu18, 2.0.155-rocky8
• 2.1.104-debian11, 2.1.104-ubuntu20, 2.1.104-ubuntu20-arm, 2.1.104-rocky8
• 2.2.72-debian12, 2.2.72-ubuntu22, 2.2.72-ubuntu22-arm, 2.2.72-rocky9
• 2.3.19-debian12, 2.3.19-ubuntu22, 2.3.19-ubuntu22-arm, 2.3.19-ml-ubuntu22, 2.3.19-rocky9

dataproc-ml is available by default in 2.3 -ml images.

Serverless for Apache Spark: Runtime version 3.0 is now generally available. This version simplifies onboarding, improves reliability, reduces startup latency, and adds support for Spark 4.0.

• Features and improvements:

  • Regional and multi-zonal workloads are used by default to increase obtainability of compute resources
  • Faster startup than previous runtimes
  • Fast resource cleanup that allows faster release of VPC IPs after workload completion
  • End-user credentials are used for all workloads by default
  • New bigquery Spark catalog, pre-configured for out-of-the-box BigQuery native table interactions
  • New Spark Serverless-specific IAM roles
  • New dataproc-rm.googleapis.com API enablement is required

New Dataproc on Compute Engine subminor image versions:

• 2.0.154-debian10, 2.0.154-ubuntu18, 2.0.154-rocky8
• 2.1.103-debian11, 2.1.103-ubuntu20, 2.1.103-ubuntu20-arm, 2.1.103-rocky8
• 2.2.71-debian12, 2.2.71-ubuntu22, 2.2.71-ubuntu22-arm, 2.2.71-rocky9
• 2.3.18-debian12, 2.3.18-ubuntu22, 2.3.18-ubuntu22-arm, 2.3.18-ml-ubuntu22, 2.3.18-rocky9

Dataproc on Compute Engine: Updated Cloud Storage connector:

• Updated to 3.1.10 in image version 2.3.18.
• Updated to 3.0.15 in image version 2.2.71.
• Updated to hadoop3-2.2.30 in image versions 2.1.103 and 2.0.154.

New Serverless for Apache Spark runtime versions:

• 1.2.62
• 2.2.62
• 2.3.15

Serverless for Apache Spark: Added the dataproc.artifacts.remove property, which lets users remove default artifacts, such as spark-bigquery-connector, iceberg, and delta-lake from a Serverless for Apache Spark runtime.

Serverless for Apache Spark: Apache Spark upgrade to version 3.5.3 for the 1.2 and 2.2 Serverless for Apache Spark runtime versions announced on October 13, 2025 has been rolled back.

Dataproc on Compute Engine subminor image version 2.3.16, announced on October 20, 2025 has been blocklisted and cannot be used when creating a new cluster.

New Dataproc on Compute Engine subminor image versions:

• 2.0.153-debian10, 2.0.153-ubuntu18, 2.0.153-rocky8
• 2.1.102-debian11, 2.1.102-ubuntu20, 2.1.102-ubuntu20-arm, 2.1.102-rocky8
• 2.2.70-debian12, 2.2.70-ubuntu22, 2.2.70-ubuntu22-arm, 2.2.70-rocky9
• 2.3.17-debian12, 2.3.17-ubuntu22, 2.3.17-ubuntu22-arm, 2.3.17-ml-ubuntu22, 2.3.17-rocky9

Fixed a Jupyter Kernel Gateway bug that caused failures while restarting kernels.

Announcing the General Availability (GA) of Lightning Engine for Google Cloud Serverless for Apache Spark. Lightning Engine is a high-performance query accelerator that delivers up to 4.3x faster performance for Spark workloads compared to open-source Spark, as measured on TPC-H-like benchmarks.

Lightning Engine and Native Query Execution are not available in 3.0 runtimes. For more details on enabling Lightning Engine and its advanced features like Native Query Execution (NQE), see the official documentation.

Serverless for Apache Spark: With the Lightning Engine GA release, the property to enable Native Query Execution (NQE) feature has been updated.

In order to use Lightning Engine, submit your jobs in the Premium tier. Under Lightning Engine, if you would like to use the NQE feature, set the new flag: spark.dataproc.lightningEngine.runtime=native. Users are encouraged to try this feature to explore the full potential of Lightning Engine.

For backward compatibility, the legacy property that was used to enable NQE spark.dataproc.runtimeEngine=native will continue to be honored in the existing runtimes 1.2, 2.2 and 2.3, but it's not supported in future releases (3.0+ runtimes).

New Dataproc on Compute Engine subminor image versions:

• 2.0.152-debian10, 2.0.152-ubuntu18, 2.0.152-rocky8
• 2.1.101-debian11, 2.1.101-ubuntu20, 2.1.101-ubuntu20-arm, 2.1.101-rocky8
• 2.2.69-debian12, 2.2.69-ubuntu22, 2.2.69-ubuntu22-arm, 2.2.69-rocky9
• 2.3.16-debian12, 2.3.16-ubuntu22, 2.3.16-ubuntu22-arm, 2.3.16-ml-ubuntu22, 2.3.16-rocky9

New Serverless for Apache Spark runtime versions:

• 3.0.0-RC6

Dataproc on Compute Engine: The default image version of premium tier clusters is now 2.3.

New Dataproc on Compute Engine subminor image versions:

• 2.0.151-debian10, 2.0.151-ubuntu18, 2.0.151-rocky8
• 2.1.100-debian11, 2.1.100-ubuntu20, 2.1.100-ubuntu20-arm, 2.1.100-rocky8
• 2.2.68-debian12, 2.2.68-ubuntu22, 2.2.68-ubuntu22-arm, 2.2.68-rocky9
• 2.3.15-debian12, 2.3.15-ubuntu22, 2.3.15-ubuntu22-arm, 2.3.15-ml-ubuntu22, 2.3.15-rocky9

Fixed startup race condition in multi-tenant clusters using the dataproc:pip.packages property that could cause authentication failures while starting Jupyter notebook kernels.

Serverless for Apache Spark: Runtimes rollout with Apache Spark upgrade to version 3.5.3 in the latest 1.2 and 2.2 Serverless for Apache Spark runtime versions has started and is expected to finish by October 16th.

New Serverless for Apache Spark runtime versions:

• 2.3.13
• 3.0.0-RC5

Serverless for Apache Spark: Upgraded Apache Spark to version 3.5.3 in the latest 2.3 Serverless for Apache Spark runtime versions.

Dataproc on Compute Engine: The following diagnostic properties are now enabled by default for new Dataproc clusters created with 2.0+ image versions:

• dataproc:diagnostic.capture.enabled: Collects checkpoint diagnostic data in the cluster temp bucket.
• dataproc:dataproc.logging.extended.enabled: Collects logs for the Knox, Zeppelin, Ranger-usersync, Jupyter_notebook, Jupyter_kernel_gateway components and the Spark History-Server in Cloud Logging.
• dataproc:dataproc.logging.syslog.enabled: Collects VM syslogs in Cloud Logging.

Note: To disable any of these features, set the corresponding property to false during cluster creation.

To continue using the Ops Agent initialization action opsagent.sh to ingest syslogs from Dataproc cluster nodes, do one of the following:

• Recommended: Use opsagent_nosyslog.sh since VM syslogs are emitted by default from Dataproc clusters.
• Set the dataproc:dataproc.logging.syslog.enabled=false and continue using opsagent.sh to ingest syslogs.

New Dataproc on Compute Engine subminor image versions:

• 2.0.150-debian10, 2.0.150-ubuntu18, 2.0.150-rocky8
• 2.1.99-debian11, 2.1.99-ubuntu20, 2.1.99-ubuntu20-arm, 2.1.99-rocky8
• 2.2.67-debian12, 2.2.67-ubuntu22, 2.2.67-ubuntu22-arm, 2.2.67-rocky9
• 2.3.14-debian12, 2.3.14-ubuntu22, 2.3.14-ubuntu22-arm, 2.3.14-ml-ubuntu22, 2.3.14-rocky9

New Dataproc on Compute Engine subminor image versions:

• 2.0.149-debian10, 2.0.149-ubuntu18, 2.0.149-rocky8
• 2.1.98-debian11, 2.1.98-ubuntu20, 2.1.98-ubuntu20-arm, 2.1.98-rocky8
• 2.2.66-debian12, 2.2.66-ubuntu22, 2.2.66-ubuntu22-arm, 2.2.66-rocky9
• 2.3.13-debian12, 2.3.13-ubuntu22, 2.3.13-ubuntu22-arm, 2.3.13-ml-ubuntu22, 2.3.13-rocky9

New Serverless for Apache Spark runtime versions:

• 1.2.61
• 2.2.61
• 2.3.12
• 3.0.0-RC4

Announcing the Preview release of Dataproc on Compute Engine image version 3.0.0-RC1:

• Spark 4.0.0
• Hadoop 3.4.1
• Hive 4.1.0
• Tez 0.10.5
• Cloud Storage Connector 3.1.4
• Conda 24.11
• Java 17
• Python 3.11
• R 4.3
• Scala 2.13

Announcing the Preview release of Serverless for Apache Spark 3.0.0-RC3 runtime:

• Spark 4.0.0
• BigQuery Spark Connector 0.42.3
• Cloud Storage Connector 3.1.5
• Conda 25.3.0
• Java 21
• Python 3.12
• R 4.4
• Scala 2.13

New Dataproc on Compute Engine subminor image versions:

• 2.3.11-debian12, 2.3.11-ubuntu22, 2.3.11-ubuntu22-arm, 2.3.11-ml-ubuntu22, 2.3.11-rocky9

New Dataproc Serverless for Spark runtime versions:

• 1.2.60
• 2.2.60
• 2.3.11

Multi-tenant clusters are now available in Preview. Many data engineers and scientists can share a multi-tenant cluster to execute their workloads in isolation from each other.

New Dataproc on Compute Engine subminor image versions:

• 2.0.147-debian10, 2.0.147-ubuntu18, 2.0.147-rocky8
• 2.1.96-debian11, 2.1.96-ubuntu20, 2.1.96-ubuntu20-arm, 2.1.96-rocky8
• 2.2.64-debian12, 2.2.64-ubuntu22, 2.2.64-ubuntu22-arm, 2.2.64-rocky9
• 2.3.10-debian12, 2.3.10-ubuntu22, 2.3.10-ubuntu22-arm, 2.3.10-ml-ubuntu22, 2.3.10-rocky9

New Dataproc Serverless for Spark runtime versions:

• 1.2.59
• 2.2.59
• 2.3.10

New Dataproc Serverless for Spark runtime versions:

• 1.2.58
• 2.2.58
• 2.3.9

Serverless for Apache Spark: Fixed a bug in Dataproc Batches that occasionally caused higher latency before an application was started.

New Dataproc on Compute Engine subminor image versions:

• 2.0.146-debian10, 2.0.146-ubuntu18, 2.0.146-rocky8
• 2.1.95-debian11, 2.1.95-ubuntu20, 2.1.95-ubuntu20-arm, 2.1.95-rocky8
• 2.2.63-debian12, 2.2.63-ubuntu22, 2.2.63-ubuntu22-arm, 2.2.63-rocky9
• 2.3.9-debian12, 2.3.9-ubuntu22, 2.3.9-ubuntu22-arm, 2.3.9-ml-ubuntu22, 2.3.9-rocky9

New Dataproc Serverless for Spark runtime versions:

• 1.2.57
• 2.2.57
• 2.3.8

New Dataproc on Compute Engine subminor image versions:

• 2.0.145-debian10, 2.0.145-ubuntu18, 2.0.145-rocky8
• 2.1.94-debian11, 2.1.94-ubuntu20, 2.1.94-ubuntu20-arm, 2.1.94-rocky8
• 2.2.62-debian12, 2.2.62-ubuntu22, 2.2.62-ubuntu22-arm, 2.2.62-rocky9
• 2.3.8-debian12, 2.3.8-ubuntu22, 2.3.8-ubuntu22-arm, 2.3.8-ml-ubuntu22, 2.3.8-rocky9

Dataproc on Compute Engine: Sharing checkpoint diagnostic data: Setting the dataproc:diagnostic.capture.access=GOOGLE_DATAPROC_DIAGNOSE property during cluster creation shares all of the temp bucket contents with Google Cloud support if uniform bucket-level access is enabled on temp bucket. If object-level access control is in effect on the temp bucket, only the checkpoint diagnostic data folder corresponding to the cluster in Cloud Storage is shared.

New Dataproc Serverless for Spark runtime versions:

• 1.2.56
• 2.2.56
• 2.3.7

Dataproc on Compute Engine: Image versions 2.2 and 2.3: The Iceberg optional component supports the BigLake Iceberg REST catalog.

New Dataproc on Compute Engine subminor image versions:

• 2.1.93-debian11, 2.1.93-rocky8, 2.1.93-ubuntu20, 2.1.93-ubuntu20-arm
• 2.2.61-debian12, 2.2.61-rocky9, 2.2.61-ubuntu22, 2.2.61-ubuntu22-arm

New Dataproc Serverless for Spark runtime versions:

• 1.1.111
• 1.2.55
• 2.2.55
• 2.3.6

Dataproc Serverless for Spark: Subminor version 1.1.111 is the last release of runtime version 1.1, which will no longer be supported and will not receive new releases.

New Dataproc on Compute Engine subminor image versions:

2.3.7-debian12, 2.3.7-ubuntu22, 2.3.7-ubuntu22-arm, 2.3.7-ml-ubuntu22, and 2.3.7-rocky9

The 2.3.7-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.

New Dataproc on Compute Engine subminor image versions:

2.3.6-debian12, 2.3.6-ubuntu22, 2.3.6-ml-ubuntu22, and 2.3.6-rocky9

The 2.3.6-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.

Dataproc on Compute Engine: Starting August 18, 2025, the following diagnostic properties will be enabled by default for newly created Dataproc clusters:

• dataproc:diagnostic.capture.enabled: Enables the collection of checkpoint data in the cluster temp bucket.
• dataproc:dataproc.logging.extended.enabled: Enables the collection of logs for the Knox, Zeppelin, Ranger-usersync, Jupyter_notebook, Jupyter_kernel_gateway components and the Spark History-Server in Cloud Logging.

• dataproc:dataproc.logging.syslog.enabled: Enables the collection of VM syslogs in Cloud Logging.

  To continue using the Ops Agent initialization action opsagent.sh to ingest syslogs from Dataproc cluster nodes, do one of the following:

  • Recommended: Use opsagent_nosyslog.sh since VM syslogs will now be emitted by default from Dataproc clusters.
  • Set the dataproc:dataproc.logging.syslog.enabled=false and continue using opsagent.sh to ingest syslogs.

Note: To disable any of these features, set the corresponding property to false during cluster creation.

Dataproc now allows Dynamic update of multi-tenancy clusters.

The Cluster Scheduled Stop feature is available in preview. You can use this feature to stop clusters after a specified idle period, at a specified future time, or after a specified period from the cluster creation or update request.

New Dataproc on Compute Engine subminor image versions:

2.3.5-debian12, 2.3.5-ubuntu22, 2.3.5-ml-ubuntu22, and 2.3.5-rocky9

The 2.3.5-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.

Serverless for Apache Spark (formerly known as Dataproc Serverless for Spark) now supports OS Login organization policy. Organizations, folders, and projects that enforce the OS Login policy can now use Serverless for Apache Spark.

New Dataproc Serverless for Spark runtime versions:

• 1.1.110
• 1.2.54
• 2.2.54
• 2.3.5

Dataproc Serverless for Spark: The built-in Iceberg now supports the BigLake Iceberg REST catalog on the 2.2 runtime.

Dataproc Serverless for Spark: Upgraded the Cloud Storage connector version to 2.2.28 in the 1.1 runtime.

New Dataproc Serverless for Spark runtime versions:

• 1.1.109
• 1.2.53
• 2.2.53
• 2.3.4

Dataproc on Compute Engine: Upgraded the Cloud Storage connector version to 2.2.28 in the latest 2.0 and 2.1 images.

New Dataproc on Compute Engine subminor image versions:

• 2.0.144-debian10, 2.0.144-rocky8, 2.0.144-ubuntu18
• 2.1.92-debian11, 2.1.92-rocky8, 2.1.92-ubuntu20, 2.1.92-ubuntu20-arm
• 2.2.60-debian12, 2.2.60-rocky9, 2.2.60-ubuntu22
• 2.3.4-debian12, 2.3.4-rocky9, 2.3.4-ubuntu22, and 2.3.4-ml-ubuntu22.

The 2.3.4-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.

Dataproc on Compute Engine: Dataproc now automatically configures Knox Gateway configuration properties gateway.dispatch.whitelist.services and gateway.dispatch.whitelist for component web UIs within the cluster.

Dataproc on Compute Engine: Fixed a bug in trino-jvm cluster properties. To configure Trino JVM options prefixed with trino-jvm, follow these guidelines:

• Configure JVM options starting with -XX:, without :. For JVM flags without a value, add = at the end. For example, add trino-jvm:-XX+HeapDumpOnOutOfMemoryError= as -XX:+HeapDumpOnOutOfMemoryError in the jvm.config.
• Specify JVM options system properties with a -D prefix the same way. For example, trino-jvm:-Dsystem.property.name=value.
• Any value containing : cannot be provided as a cluster property.

Dataproc on Compute Engine & Dataproc Serverless: Backported GH-3198 in Parquet addressing CVE-2025-46762.

New Dataproc Serverless for Spark runtime versions:

• 1.1.108
• 1.2.52
• 2.2.52
• 2.3.3

Announcing the GA release of Dataproc on Compute Engine image version 2.3:

Image Version 2.3 is a lightweight image that contains only core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher security compliance requirements, use the image version 2.3 or later when creating a Dataproc cluster. Optional components can still be deployed on-demand.

The following images are the latest available 2.3 subminor image versions:

• 2.3.3-debian12, 2.3.3-rocky9, 2.3.3-ubuntu22, and 2.3.3-ml-ubuntu22.

The 2.3.3-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.

New Dataproc Serverless for Spark runtime versions:

• 1.1.107
• 1.2.51
• 2.2.51
• 2.3.2

Dataproc Serverless for Spark: Fixed a bug that prevented the spark.executorEnv property from correctly setting specific executor environment variables across all runtimes.

New Dataproc on Compute Engine subminor image versions:

• 2.0.143-debian10, 2.0.143-rocky8, 2.0.143-ubuntu18
• 2.1.91-debian11, 2.1.90-rocky8, 2.1.91-ubuntu20, 2.1.91-ubuntu20-arm
• 2.2.59-debian12, 2.2.59-rocky9, 2.2.59-ubuntu22

Dataproc on Compute Engine: Fixed the ordering of log entries generated from clusters created with 2.2+ image versions by assigning timestamps closer to the log generation time.

The support dates for Dataproc on Compute Engine image versions 2.0, 2.1, and 2.2 have been extended, as follows:

• Image version 2.2: Supported until 03/31/2027
• Image version 2.1: Supported until 03/31/2026
• Image version 2.0 Supported until 09/30/2025

New Dataproc Serverless for Spark runtime versions:

• 1.1.106
• 1.2.50
• 2.2.50
• 2.3.1

Announcing the General Availability release of Spark on BigQuery, which lets you create a serverless Spark session in a BigQuery Studio notebook. Use this feature to create, run, and test Spark jobs quickly and easily. For more information, see Run PySpark code in BigQuery Studio notebooks.

Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime versions 2.3, which include the following components:

• Spark 3.5.1
• BigQuery Spark Connector 0.42.3
• Cloud Storage Connector 3.1.2
• Java 17
• Python 3.11
• R 4.3
• Scala 2.13

New Dataproc Serverless for Spark runtime versions:

• 1.1.105
• 1.2.49
• 2.2.49
• 2.3.0

Dataproc now supports the creation of zero-scale clusters, available in preview. This feature provides a cost-effective way to use Dataproc clusters, as they utilize only secondary workers that can be scaled down to zero when not in use.

New Dataproc on Compute Engine subminor image versions:

• 2.0.142-debian10, 2.0.142-rocky8, 2.0.142-ubuntu18
• 2.1.90-debian11, 2.1.90-rocky8, 2.1.90-ubuntu20, 2.1.90-ubuntu20-arm
• 2.2.58-debian12, 2.2.58-rocky9, 2.2.58-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.104
• 1.2.48
• 2.2.48

New Dataproc on Compute Engine subminor image versions:

• 2.0.141-debian10, 2.0.141-rocky8, 2.0.141-ubuntu18
• 2.1.89-debian11, 2.1.89-rocky8, 2.1.89-ubuntu20, 2.1.89-ubuntu20-arm
• 2.2.57-debian12, 2.2.57-rocky9, 2.2.57-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.103
• 1.2.47
• 2.2.47

Dataproc Serverless for Spark: Spark UI for Dataproc Serverless batches and interactive sessions, which lets you monitor and debug your serverless Spark workloads, now features Event Timeline and Task Quantile views for enhanced troubleshooting.

New Dataproc on Compute Engine subminor image versions:

• 2.0.140-debian10, 2.0.140-rocky8, 2.0.140-ubuntu18
• 2.1.88-debian11, 2.1.88-rocky8, 2.1.88-ubuntu20, 2.1.88-ubuntu20-arm
• 2.2.56-debian12, 2.2.56-rocky9, 2.2.56-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.102
• 1.2.46
• 2.2.46

Dataproc on Compute Engine: The default enabling of the following cluster properties previously announced to occur on May 10, 2025 (see the February 10, 2025 release note) has been postponed to a future date. The future date will be announced in a release note at least one month in advance of the change. Until then, these diagnostic properties will continue to be set to false by default unless set to true by the user.

• dataproc:diagnostic.capture.enabled
• dataproc:dataproc.logging.extended.enabled
• dataproc:dataproc.logging.syslog.enabled

Dataproc on Compute Engine: Upgraded NodeProblemDetector to 0.8.20 based version for 2.2 image.

New Dataproc on Compute Engine subminor image versions:

• 2.0.139-debian10, 2.0.139-rocky8, 2.0.139-ubuntu18
• 2.1.87-debian11, 2.1.87-rocky8, 2.1.87-ubuntu20, 2.1.87-ubuntu20-arm
• 2.2.55-debian12, 2.2.55-rocky9, 2.2.55-ubuntu22

Dataproc on Compute Engine: Upgraded oauth2l to v1.3.3 to address CVEs.

Dataproc on Compute Engine: Fixed an issue with Apache Hudi that caused failure in Hudi CLI.

Dataproc Serverless: Backported GH-3168 in Parquet addressing CVE-2025-30065.

New Dataproc Serverless for Spark runtime versions:

• 1.1.101
• 1.2.45
• 2.2.45

Native Query Execution now supports reading Apache ORC complex types.

Dataproc on Compute Engine: Fixed Job ID retrieval in Dataproc job logs for clusters created with 2.0, 2.1 image versions, by ignoring timestamp prefix.

New Dataproc on Compute Engine subminor image versions:

2.0.138-debian10, 2.0.138-rocky8, 2.0.138-ubuntu18

2.1.86-debian11, 2.1.86-rocky8, 2.1.86-ubuntu20, 2.1.86-ubuntu20-arm

2.2.54-debian12, 2.2.54-rocky9, 2.2.54-ubuntu22

Dataproc on Compute Engine: Backported GH-3168 in Parquet addressing CVE-2025-30065.

Dataproc on Compute Engine: Added an temporary object hold on the spark-job-history folder in Cloud Stroage to prevent deletion by Cloud Storage life cycling.

New Dataproc Serverless for Spark runtime versions:

• 1.1.99
• 1.2.43
• 2.2.43

Fixed a bug in which Jupyter fails to restart upon cluster restart on Personal Authentication clusters.

Dataproc on Compute Engine: The Spark BigQuery connector has been upgraded to version 0.34.1 in the latest 2.2 image version.

New Dataproc on Compute Engine subminor image versions:

• 2.0.137-debian10, 2.0.137-rocky8, 2.0.137-ubuntu18
• 2.1.85-debian11, 2.1.85-rocky8, 2.1.85-ubuntu20, 2.1.85-ubuntu20-arm
• 2.2.53-debian12, 2.2.53-rocky9, 2.2.53-ubuntu22

Dataproc Serverless for Spark: Gemini Cloud Assist Investigations is available in Preview for the following runtimes:

• 1.1
• 1.2
• 2.2

Dataproc on Compute Engine: Fixed an issue with the retrieval of an Access token when using the ranger-gcs-plugin with 2.2 images.

New Dataproc on Compute Engine subminor image versions:

• 2.2.52-debian12, 2.2.52-rocky9, 2.2.52-ubuntu22

Dataproc Serverless for Spark: Installed CUDA, cuDNN and NCCL NVIDIA libraries in 1.2 and 2.2 runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.98
• 1.2.42
• 2.2.42

New Dataproc on Compute Engine subminor image versions:

• 2.2.51-debian12, 2.2.51-rocky9, 2.2.51-ubuntu22

Dataproc on Compute Engine: Hyperdisk-Balanced is now the default primary disk type when creating a cluster from the console.

Dataproc on Compute Engine: Fixed incorrectly attributed Dataproc job logs in Cloud Logging for clusters created with 2.2+ image versions. This happened when multiple Dataproc jobs were running concurrently on the same cluster.

New Dataproc Serverless for Spark runtime versions:

• 1.1.97
• 1.2.41
• 2.2.41

Dataproc Serverless for Spark: Hadoop Native libraries are installed by default in all runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.96
• 1.2.40
• 2.2.40

New Dataproc on Compute Engine subminor image versions:

• 2.0.136-debian10, 2.0.136-rocky8, 2.0.136-ubuntu18
• 2.1.84-debian11, 2.1.84-rocky8, 2.1.84-ubuntu20, 2.1.84-ubuntu20-arm
• 2.2.50-debian12, 2.2.50-rocky9, 2.2.50-ubuntu22

Dataproc on Compute Engine: Spark upgraded to version 3.5.3 in the latest Dataproc image version 2.2.

Dataproc on Compute Engine: Added support for Enhanced Flexibility Mode (EFM) with primary worker shuffle mode on Spark for image version 2.2.50 and above.

Dataproc on Compute Engine: The latest Dataproc 2.2 image version now supports Spark data lineage.

New Dataproc Serverless for Spark runtime versions:

• 1.1.95
• 1.2.39
• 2.2.39

New Dataproc on Compute Engine subminor image versions:

• 2.0.135-debian10, 2.0.135-rocky8, 2.0.135-ubuntu18
• 2.1.83-debian11, 2.1.83-rocky8, 2.1.83-ubuntu20, 2.1.83-ubuntu20-arm
• 2.2.49-debian12, 2.2.49-rocky9, 2.2.49-ubuntu22

Dataproc is now available in the europe-north2 region (Stockholm, Sweden).

New Dataproc Serverless for Spark runtime versions:

• 1.1.94
• 1.2.38
• 2.2.38

New Dataproc on Compute Engine subminor image versions:

• 2.0.134-debian10, 2.0.134-rocky8, 2.0.134-ubuntu18
• 2.1.82-debian11, 2.1.82-rocky8, 2.1.82-ubuntu20, 2.1.82-ubuntu20-arm
• 2.2.48-debian12, 2.2.48-rocky9, 2.2.48-ubuntu22

Dataproc on Compute Engine: Explicitly disabled sha1, md5 algorithms for use with kex and kex-gss sshd features.

New Dataproc Serverless for Spark runtime versions:

• 1.1.93
• 1.2.37
• 2.2.37

New Dataproc on Compute Engine subminor image versions:

• 2.0.133-debian10, 2.0.133-rocky8, 2.0.133-ubuntu18
• 2.1.81-debian11, 2.1.81-rocky8, 2.1.81-ubuntu20, 2.1.81-ubuntu20-arm
• 2.2.47-debian12, 2.2.47-rocky9, 2.2.47-ubuntu22

New Dataproc on Compute Engine subminor image versions:

• 2.0.132-debian10, 2.0.132-rocky8, 2.0.132-ubuntu18
• 2.1.80-debian11, 2.1.80-rocky8, 2.1.80-ubuntu20, 2.1.80-ubuntu20-arm
• 2.2.46-debian12, 2.2.46-rocky9, 2.2.46-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.92
• 1.2.36
• 2.2.36

Data Lineage for Dataproc Hive is now in Public Preview, which can be enabled using the Hive Lineage initialization action.

Dataproc on Compute Engine: To help diagnose Dataproc clusters, you can set the following cluster properties to true when you create a cluster:

• dataproc:diagnostic.capture.enabled: When set to true, enables the collection of checkpoint data in the cluster temp bucket.
• dataproc:dataproc.logging.extended.enabled: When set to true, enables the collection of logs for the Knox, Zeppelin, Solr, Trino, Presto, Ranger-usersync, Jupyter_notebook components and the Spark History-Server in Cloud Logging.
• dataproc:dataproc.logging.syslog.enabled: When set to true, enables the collection of VM syslogs in Cloud Logging.

Note: starting May 10, 2025, these properties will be set to true by default.

New Dataproc Serverless for Spark runtime versions:

• 1.1.91
• 1.2.35
• 2.2.35

New Dataproc on Compute Engine subminor image versions:

• 2.0.131-debian10, 2.0.131-rocky8, 2.0.131-ubuntu18
• 2.1.79-debian11, 2.1.79-rocky8, 2.1.79-ubuntu20, 2.1.79-ubuntu20-arm
• 2.2.45-debian12, 2.2.45-rocky9, 2.2.45-ubuntu22

Spark UI for Dataproc Serverless Batches and Interactive sessions, which lets you to monitor and debug your serverless Spark workloads, is now available for CMEK (Customer-Managed Encryption Keys) and Assured Workloads. The Spark UI is available by default and free of cost.

New Dataproc Serverless for Spark runtime versions:

• 1.1.90
• 1.2.34
• 2.2.34

New Dataproc on Compute Engine subminor image versions:

• 2.0.130-debian10, 2.0.130-rocky8, 2.0.130-ubuntu18
• 2.1.78-debian11, 2.1.78-rocky8, 2.1.78-ubuntu20, 2.1.78-ubuntu20-arm
• 2.2.44-debian12, 2.2.44-rocky9, 2.2.44-ubuntu22

• New Hyperdisk Balanced primary disk type available on Dataproc clusters.
• New machine types available for Hyperdisk Balanced disk type on clusters: C4, C4A, and N4.

New Dataproc Serverless for Spark runtime versions:

• 1.1.89
• 1.2.33
• 2.2.33

Dataproc on Compute Engine: Private Google Access is now automatically enabled in the configured subnetwork when creating clusters with internal IP addresses.

Dataproc Serverless for Spark: Private Google Access is now automatically enabled in the configured subnetwork when running batch workloads and interactive sessions.

Dataproc cluster caching now supports ARM images.

Zeppelin component added to 2.1-Ubuntu20-arm images.

New Dataproc on Compute Engine subminor image versions:

• 2.0.129-debian10, 2.0.129-rocky8, 2.0.129-ubuntu18
• 2.1.77-debian11, 2.1.77-rocky8, 2.1.77-ubuntu20, 2.1.77-ubuntu20-arm
• 2.2.43-debian12, 2.2.43-rocky9, 2.2.43-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.88
• 1.2.32
• 2.2.32

New Dataproc Serverless for Spark runtime versions:

• 1.1.87
• 1.2.31
• 2.2.31

New Dataproc on Compute Engine subminor image versions:

• 2.0.128-debian10, 2.0.128-rocky8, 2.0.128-ubuntu18
• 2.1.76-debian11, 2.1.76-rocky8, 2.1.76-ubuntu20, 2.1.76-ubuntu20-arm
• 2.2.42-debian12, 2.2.42-rocky9, 2.2.42-ubuntu22

Dataproc Serverless for Spark:

• Added support for XGBoost 2.1 in 2.2 runtime.
• Change spark.sql.maxMetadataStringLength default value to 5000 for 1.2 and 2.2 runtimes

Dataproc Serverless for Spark: On March 10, 2025, the Dataproc Resource Manager API will be enabled as part of General Availability (GA) for Dataproc Serverless 3.0+ versions.

User action will not be required in response to this API enablement change.

The Dataproc Resource Manager will be implemented as a stand-alone Google Cloud API, dataprocrm.googleapis.com. It will allow Dataproc distributions of open source software, ,particularly Apache Spark, to directly communicate resource requirements.

New Dataproc on Compute Engine subminor image versions:

• 2.0.127-debian10, 2.0.127-rocky8, 2.0.127-ubuntu18
• 2.1.75-debian11, 2.1.75-rocky8, 2.1.75-ubuntu20, 2.1.75-ubuntu20-arm
• 2.2.41-debian12, 2.2.41-rocky9, 2.2.41-ubuntu22

New Dataproc on Compute Engine subminor image versions:

• 2.0.126-debian10, 2.0.126-rocky8, 2.0.126-ubuntu18
• 2.1.74-debian11, 2.1.74-rocky8, 2.1.74-ubuntu20, 2.1.74-ubuntu20-arm
• 2.2.40-debian12, 2.2.40-rocky9, 2.2.40-ubuntu22

Dataproc on Compute Engine: Updated Dataproc Metastore (DPMS) gRPC proxy image version to v. 0.0.70

Dataproc Serverless for Spark: Spark Lineage is available for all supported Dataproc Serverless for Spark runtime versions.

Dataproc is now available in the northamerica-south1 region (Queretaro, Mexico).

Announcing the General Availability (GA) of Spot and non-preemptible VM mixing for Dataproc secondary workers which allows you to mix spot and non-preemptible secondary workers when you create a Dataproc cluster.

Announcing the General Availability (GA) of Flexible shapes for Dataproc secondary workers which allows you to provide a ranked selection of machine types to use for the creation of VMs.

New Dataproc Serverless for Spark runtime versions:

• 1.1.86
• 1.2.30
• 2.2.30

Disabled HiveServer2 Ranger policy synchronization in non-HA clusters for latest image version 2.1 and later. Policy synchronization is causing instability of the HiveServer2 process while trying to connect to ZooKeeper, which is not active by default in non-HA clusters.

New Dataproc on Compute Engine subminor image versions:

• 2.0.125-debian10, 2.0.125-rocky8, 2.0.125-ubuntu18
• 2.1.73-debian11, 2.1.73-rocky8, 2.1.73-ubuntu20, 2.1.73-ubuntu20-arm
• 2.2.39-debian12, 2.2.39-rocky9, 2.2.39-ubuntu22

Note: When using Dataproc version 2.0.125 with the ranger-gcs-plugin, please create a customer support request for your project to use the enhanced version of the plugin prior to its GA release. This note does not apply Dataproc on Compute Engine image versions 2.1 and 2.2.

Dataproc Serverless for Spark: The Hadoop Google Secret Manager Credential Provider feature is now available in the Dataproc Serverless for Spark 1.2 and 2.2 runtimes.

New Dataproc on Compute Engine subminor image versions:

• 2.0.124-debian10, 2.0.124-rocky8, 2.0.124-ubuntu18
• 2.1.72-debian11, 2.1.72-rocky8, 2.1.72-ubuntu20, 2.1.72-ubuntu20-arm
• 2.2.38-debian12, 2.2.38-rocky9, 2.2.38-ubuntu22

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 3.0.3 version in the latest 1.2 and 2.2 runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.85
• 1.2.29
• 2.2.29

Dataproc Serverless for Spark: Added common AI/ML Python packages by default to Dataproc Serverless for Spark 1.2 and 2.2 runtimes.

Announcing the General Availability (GA) release of Spark UI for Dataproc Serverless Batches and Interactive sessions which allows you to monitor and debug your serverless Spark workloads. Spark UI is available by default and free of cost for all Dataproc Serverless workloads.

• New Dataproc on Compute Engine subminor image versions:

  • 2.0.123-debian10, 2.0.123-rocky8, 2.0.123-ubuntu18
  • 2.1.71-debian11, 2.1.71-rocky8, 2.1.71-ubuntu20, 2.1.71-ubuntu20-arm
  • 2.2.37-debian12, 2.2.37-rocky9, 2.2.37-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.84
• 1.2.28
• 2.2.28

Dataproc Clusters created with image versions 2.0.57+, 2.1.5+, or 2.2+: Secondary workers' control plane operations are made by the Dataproc Service Agent service account (service-<project-number>@dataproc-accounts.iam.gserviceaccount.com). They will no longer use the Google APIs Service Agent service account (<project-number>@cloudservices.gserviceaccount.com).

• New Dataproc on Compute Engine subminor image versions:

  • 2.0.122-debian10, 2.0.122-rocky8, 2.0.122-ubuntu18
  • 2.1.70-debian11, 2.1.70-rocky8, 2.1.70-ubuntu20, 2.1.70-ubuntu20-arm
  • 2.2.36-debian12, 2.2.36-rocky9, 2.2.36-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.83
• 1.2.27
• 2.2.27

• New Dataproc on Compute Engine subminor image versions:

  • 2.0.121-debian10, 2.0.121-rocky8, 2.0.121-ubuntu18
  • 2.1.69-debian11, 2.1.69-rocky8, 2.1.69-ubuntu20, 2.1.69-ubuntu20-arm
  • 2.2.35-debian12, 2.2.35-rocky9, 2.2.35-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.82
• 1.2.26
• 2.2.26

New Dataproc Serverless for Spark runtime versions:

• 1.1.81
• 1.2.25
• 2.2.25

Blocklisted the following Dataproc on Compute Engine subminor image versions:

• 2.0.120-debian10, 2.0.120-rocky8, 2.0.120-ubuntu18
• 2.1.68-debian11, 2.1.68-rocky8, 2.1.68-ubuntu20, 2.1.68-ubuntu20-arm
• 2.2.34-debian12, 2.2.34-rocky9, 2.2.34-ubuntu22

Dataproc Serverless for Spark: Added the google-cloud-dlp Python package by default to the Dataproc Serverless for Spark runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.79
• 1.2.23
• 2.2.23

Dataproc Serverless for Spark: In runtime versions 1.2 and 2.2, minimized the dynamic memory footprint of the Spark application by setting XX:MaxHeapFreeRatio to 30% and XX:MinHeapFreeRatio to 10%.

Dataproc Serverless for Spark: Fixed an issue that would cause some batches and sessions to fail to start when using the premium compute tier.

Blocklisted the following Dataproc on Compute Engine subminor image versions:

• 2.0.119-debian10, 2.0.103-rocky8, 2.0.103-ubuntu18
• 2.1.67-debian11, 2.1.51-rocky8, 2.1.51-ubuntu20, 2.1.51-ubuntu20-arm
• 2.2.33-debian12, 2.2.17-rocky9, 2.2.17-ubuntu22

New Dataproc on Compute Engine subminor image versions:

• 2.0.118-debian10, 2.0.118-rocky8, 2.0.118-ubuntu18
• 2.1.66-debian11, 2.1.66-rocky8, 2.1.66-ubuntu20, 2.1.66-ubuntu20-arm
• 2.2.32-debian12, 2.2.32-rocky9, 2.2.32-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.78
• 1.2.22
• 2.2.22

Dataproc Serverless for Spark: Fixed a bug that caused some batches and sessions to fail to start when using the premium compute tier.

New Dataproc on Compute Engine subminor image versions:

• 2.0.117-debian10, 2.0.117-rocky8, 2.0.117-ubuntu18
• 2.1.65-debian11, 2.1.65-rocky8, 2.1.65-ubuntu20, 2.1.65-ubuntu20-arm
• 2.2.31-debian12, 2.2.31-rocky9, 2.2.31-ubuntu22

Dataproc on Compute Engine: The latest 2.2 image versions support Hudi Trino integration natively. If both components are selected when you create a Dataproc cluster, Trino will be configured to support Hudi automatically.

Dataproc on Compute Engine: The latest 2.2 image versions now support Hudi 0.15.0.

Dataproc on Compute Engine: Dataproc image version 2.2 will become the default Dataproc on Compute Engine image version on September 6, 2024.

New Dataproc on Compute Engine subminor image versions:

• 2.0.116-debian10, 2.0.116-rocky8, 2.0.116-ubuntu18
• 2.1.64-debian11, 2.1.64-rocky8, 2.1.64-ubuntu20, 2.1.64-ubuntu20-arm
• 2.2.30-debian12, 2.2.30-rocky9, 2.2.30-ubuntu22,

Dataproc on GKE runtime versions 2.0 (Spark 3.1) is deprecated.

Dataproc on Compute Engine: Apache Spark upgraded to version 3.5.1 in image version 2.2 starting with image version 2.2.30.

New Dataproc on Compute Engine subminor image versions:

• 2.0.115-debian10, 2.0.115-rocky8, 2.0.115-ubuntu18
• 2.1.63-debian11, 2.1.63-rocky8, 2.1.63-ubuntu20, 2.1.63-ubuntu20-arm
• 2.2.29-debian12, 2.2.29-rocky9, 2.2.29-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.77
• 1.2.21
• 2.0.85
• 2.2.21

Dataproc Serverless for Spark: Subminor version 2.0.85 is the last release of runtime version 2.0, which will no longer be supported and will not receive new releases.

New Dataproc on Compute Engine subminor image versions:

• 2.0.114-debian10, 2.0.114-rocky8, 2.0.114-ubuntu18
• 2.1.62-debian11, 2.1.62-rocky8, 2.1.62-ubuntu20, 2.1.62-ubuntu20-arm
• 2.2.28-debian12, 2.2.28-rocky9, 2.2.28-ubuntu22

syslog is now available for Dataproc cluster nodes in Cloud Logging. See Dataproc logs for cluster and job log information.

New Dataproc Serverless for Spark runtime versions:

• 1.1.76
• 1.2.20
• 2.0.84
• 2.2.20

New Dataproc Serverless for Spark runtime versions:

• 1.1.75
• 1.2.19
• 2.0.83
• 2.2.19

New Dataproc Serverless for Spark runtime versions:

• 1.1.74
• 1.2.18
• 2.0.82
• 2.2.18

Dataproc Serverless for Spark: Upgraded Spark BigQuery connector to version 0.36.4 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.73
• 1.2.17
• 2.0.81
• 2.2.17

New Dataproc on Compute Engine subminor image versions:

• 2.0.113-debian10, 2.0.113-rocky8, 2.0.113-ubuntu18
• 2.1.61-debian11, 2.1.61-rocky8, 2.1.61-ubuntu20, 2.1.61-ubuntu20-arm
• 2.2.27-debian12, 2.2.27-rocky9, 2.2.27-ubuntu22

Enabled user sync by default for clusters using Ranger.

Replaced Spark external packages with connector folder on Dataproc 2.2 clusters.

Fixed a bug that caused intermittent delays and failures in clusters with 3 HDFS.

Added support for N4 and C4 machine types for Dataproc image versions 2.1 and above. The following default configurations are now applied to clusters created with N4 or C4 machine types:

• bootdisktype = "hyperdisk-balanced"
• nictype = "gvnic"

Hyperdisks for Dataproc clusters are now created with default throughput and IOPS. When this behavior becomes configurable, it will be announced in a future release note.

When a Cluster, Job, AutoscalingPolicy, or WorkflowTemplate API resource does not exist and the requestor does not have access to the project, a 403 error code is now issued instead of a 404 error code.

New Dataproc Serverless for Spark runtime versions:

• 1.1.72
• 1.2.16
• 2.0.80
• 2.2.16

Note: Dataproc Serverless for Spark runtime versions 1.1.71, 1.2.15, 2.0.79, and 2.2.15 were not released.

New Dataproc on Compute Engine subminor image versions:

• 2.0.112-debian10, 2.0.112-rocky8, 2.0.112-ubuntu18
• 2.1.60-debian11, 2.1.60-rocky8, 2.1.60-ubuntu20, 2.1.60-ubuntu20-arm
• 2.2.26-debian12, 2.2.26-rocky9, 2.2.26-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.70
• 1.2.14
• 2.0.78
• 2.2.14

New Dataproc on Compute Engine subminor image versions:

• 2.0.111-debian10, 2.0.112-rocky8, 2.0.112-ubuntu18
• 2.1.59-debian11, 2.1.60-rocky8, 2.1.60-ubuntu20, 2.1.60-ubuntu20-arm
• 2.2.25-debian12, 2.2.26-rocky9, 2.2.26-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.69
• 1.2.13
• 2.0.77
• 2.2.13

New Dataproc on Compute Engine subminor image versions:

• 2.0.110-debian10, 2.0.110-rocky8, 2.0.110-ubuntu18
• 2.1.58-debian11, 2.1.58-rocky8, 2.1.58-ubuntu20, 2.1.58-ubuntu20-arm
• 2.2.24-debian12, 2.2.24-rocky9, 2.2.24-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.68
• 1.2.12
• 2.0.76
• 2.2.12

New Dataproc on Compute Engine subminor image versions:

• 2.0.109-debian10, 2.0.109-rocky8, 2.0.109-ubuntu18
• 2.1.57-debian11, 2.1.57-rocky8, 2.1.57-ubuntu20, 2.1.57-ubuntu20-arm
• 2.2.23-debian12, 2.2.23-rocky9, 2.2.23-ubuntu22

Added Cloud Profiler support in Dataproc Serverless for Spark. Enable profiling via the dataproc.profiling.enabled=true property and configure it via dataproc.profiling.name=<PROFILE_NAME>

Dataproc on Compute Engine: Apache Hadoop upgraded to version 3.2.4 in image version 2.0 starting with image version 2.0.109.

New Dataproc on Compute Engine subminor image versions:

• 2.0.108-debian10, 2.0.108-rocky8, 2.0.108-ubuntu18
• 2.1.56-debian11, 2.1.56-rocky8, 2.1.56-ubuntu20, 2.1.56-ubuntu20-arm
• 2.2.22-debian12, 2.2.22-rocky9, 2.2.22-ubuntu22

Backported fixes for HIVE-25958 and HIVE-20220 (new configuration hive.groupby.enable.deterministic.distribution=false/true).

New Dataproc Serverless for Spark runtime versions:

• 1.1.67
• 1.2.11
• 2.0.75
• 2.2.11

Dataproc Serverless for Spark: To fix compatibility with open table formats (Apache Iceberg, Apache Hudi and Delta Lake), the ANTLR version downgraded from 4.13.1 to 4.9.3 in Dataproc Serverless for Spark runtime versions 1.2 and 2.2.

The Dataproc Component Gateway is now activated by default when you create a Dataproc on Compute Engine cluster using the Google Cloud console.

New Dataproc on Compute Engine subminor image versions:

• 2.0.107-debian10, 2.0.107-rocky8, 2.0.107-ubuntu18
• 2.1.55-debian11, 2.1.55-rocky8, 2.1.55-ubuntu20, 2.1.55-ubuntu20-arm
• 2.2.21-debian12, 2.2.21-rocky9, 2.2.21-ubuntu22

Dataproc Serverless for Spark: To fix compatibility with open table formats (Apache Iceberg, Apache Hudi and Delta Lake), the ANTLR version will be downgraded from 4.13.1 to 4.9.3 in Dataproc Serverless for Spark runtime versions 1.2 and 2.2 on June 26, 2024.

Dataproc Serverless for Spark: Spark runtime version 2.2 will become the default Dataproc Serverless for Spark runtime version on September 6, 2024.

New Dataproc Serverless for Spark runtime versions:

• 1.1.66
• 1.2.10
• 2.0.74
• 2.2.10

New Dataproc on Compute Engine subminor image versions:

• 2.0.106-debian10, 2.0.106-rocky8, 2.0.106-ubuntu18
• 2.1.54-debian11, 2.1.54-rocky8, 2.1.54-ubuntu20, 2.1.54-ubuntu20-arm
• 2.2.20-debian12, 2.2.20-rocky9, 2.2.20-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.65
• 1.2.9
• 2.0.73
• 2.2.9

Support configuration to prevent HiveMetaStore metrics expensive database queries. To prevent expensive queries during HiveMetaStore startup, set Hive property metastore.initial.metadata.count.enabled to false.

Dataproc Serverless for Spark: Upgraded Spark BigQuery connector to version 0.36.3 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

The Apache Spark in BigQuery feature is available in Private Preview. This feature lets you create a Spark session in a BigQuery notebook that you can use to develop and submit PySpark code from BigQuery. To access this feature, fill in and submit the Dataproc Preview access request form.

New Dataproc on Compute Engine subminor image versions:

• 2.0.105-debian10, 2.0.105-rocky8, 2.0.105-ubuntu18
• 2.1.53-debian11, 2.1.53-rocky8, 2.1.53-ubuntu20, 2.1.53-ubuntu20-arm
• 2.2.19-debian12, 2.2.19-rocky9, 2.2.19-ubuntu22

Dataproc on Compute Engine: When creating a cluster with the latest Dataproc on Compute Engine image versions, the secondary worker boot disk type now defaults to the primary worker boot disk type, which is pd-standard if the primary worker boot disk type is not specified.

New Dataproc Serverless for Spark runtime versions:

• 1.1.64
• 1.2.8
• 2.0.72
• 2.2.8

Dataproc on Compute Engine: Update restartable job error messages to include job IDs.

Dataproc Serverless for Spark: Automatically apply goog-dataproc-session-id, goog-dataproc-session-uuid and goog-dataproc-location labels for a session resource.

New Dataproc on Compute Engine subminor image versions:

• 2.0.104-debian10, 2.0.104-rocky8, 2.0.104-ubuntu18
• 2.1.52-debian11, 2.1.52-rocky8, 2.1.52-ubuntu20, 2.1.52-ubuntu20-arm
• 2.2.18-debian12, 2.2.18-rocky9, 2.2.18-ubuntu22

Dataproc Serverless for Spark: Removed Spark data lineage support for runtime version 1.2.

New Dataproc Serverless for Spark runtime versions:

• 1.1.63
• 1.2.7
• 2.0.71
• 2.1.50
• 2.2.7

Dataproc Serverless for Spark: Enabled Spark checkpoint (spark.checkpoint.compress) and RDD (spark.rdd.compress) compression in the latest 1.2 and 2.2 runtime versions.

Dataproc Serverless for Spark: Subminor version 2.1.50 is the last release of runtime version 2.1, which will no longer be supported and will not receive new releases.

Blocklisted the following Dataproc on Compute Engine subminor image versions:

• 2.0.103-debian10, 2.0.103-rocky8, 2.0.103-ubuntu18
• 2.1.51-debian11, 2.1.51-rocky8, 2.1.51-ubuntu20, 2.1.51-ubuntu20-arm
• 2.2.17-debian12, 2.2.17-rocky9, 2.2.17-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.62
• 1.2.6
• 2.0.70
• 2.1.49
• 2.2.6

Upgraded Spark BigQuery connector to version 0.36.2 in the latest 1.2 and 2.2 Dataproc Serverless for Spark runtime versions.

New Dataproc on Compute Engine subminor image versions:

• 2.0.102-debian10, 2.0.102-rocky8, 2.0.102-ubuntu18

• 2.1.50-debian11, 2.1.50-rocky8, 2.1.50-ubuntu20, 2.1.50-ubuntu20-arm

• 2.2.16-debian12, 2.2.16-rocky9, 2.2.16-ubuntu22

Anaconda's default channel is disabled for package installations on Dataproc on Compute Engine.

New Dataproc on Compute Engine subminor image versions:

• 2.0.101-debian10, 2.0.101-rocky8, 2.0.101-ubuntu18

• 2.1.49-debian11, 2.1.49-rocky8, 2.1.49-ubuntu20, 2.1.49-ubuntu20-arm

• 2.2.15-debian12, 2.2.15-rocky9, 2.2.15-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.61
• 1.2.5
• 2.0.69
• 2.1.48
• 2.2.5

New Dataproc on Compute Engine subminor image versions:

• 2.0.100-debian10, 2.0.100-rocky8, 2.0.100-ubuntu18
• 2.1.48-debian11, 2.1.48-rocky8, 2.1.48-ubuntu20, 2.1.48-ubuntu20-arm
• 2.2.14-debian12, 2.2.14-rocky9, 2.2.14-ubuntu22

Dataproc on Compute Engine:

• Backported patches for HIVE-14557, HIVE-19326, HIVE-20514, HIVE-21100, HIVE-22165, HIVE-22416, HIVE-24435.
• Hive: Improved ORC split generation.

New Dataproc Serverless for Spark runtime versions:

• 1.1.60
• 1.2.4
• 2.0.68
• 2.1.47
• 2.2.4

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS to version 24.04.0 in 1.2 and 2.2 Dataproc Serverless for Spark runtimes.

When you submit a Dataproc Serverless Batch with a CMEK key:

• In addition to encrypting disk and Cloud Storage data, Dataproc Serverless will use your CMEK to also encrypt batch job arguments. This change will require you to do the following:
  • Assign the Cloud KMS CryptoKey Encrypter/Decrypter role to the Dataproc Service Agent service account.
  • Enable the Cloud KMS API on the project that runs Dataproc Batches resources.
  • If the Dataproc Service Agent role is not attached to the Dataproc Service Agent service account, then add the serviceusage.services.use permission to the custom role attached to the Dataproc Service Agent service account.
• batches.list will return an unreachable field that lists any batches with job arguments that couldn't be decrypted. You can issue a batches.get request to obtain more information on an unreachable batch.
• Multi-regional and cross-regional CMEKs will no longer be permitted. The key (CMEK) must be located in the same location as the encrypted resource. For example, the CMEK used to encrypt a batch that runs in the us-central1 region must also be located in the us-central1 region.

New Dataproc on Compute Engine subminor image versions:

• 2.0.99-debian10, 2.0.99-rocky8, 2.0.99-ubuntu18
• 2.1.47-debian11, 2.1.47-rocky8, 2.1.47-ubuntu20, 2.1.47-ubuntu20-arm
• 2.2.13-debian12, 2.2.13-rocky9, 2.2.13-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.59
• 1.2.3
• 2.0.67
• 2.1.46
• 2.2.3

New Dataproc on Compute Engine subminor image versions:

• 2.0.98-debian10, 2.0.98-rocky8, 2.0.98-ubuntu18
• 2.1.46-debian11, 2.1.46-rocky8, 2.1.46-ubuntu20, 2.1.46-ubuntu20-arm
• 2.2.12-debian12, 2.2.12-rocky9, 2.2.12-ubuntu22

Announcing Dataproc Workflow Templates supports the CMEK organization policy.

New Dataproc Serverless for Spark runtime versions:

• 1.1.58
• 1.2.2
• 2.0.66
• 2.1.45
• 2.2.2

Updated the default autoscaling V2 cool-down time from 2m to 1m to reduce scaling latency.

Set the soft delete policy of newly created Dataproc staging and temp Cloud Storage buckets to 0 days.

Fixed a bug where Dataproc Serverless sessions that live longer than 48 hours are underbilled.

Dataproc Serverless for Spark: The preview release of Advanced troubleshooting, including Gemini-assisted troubleshooting, is now available for Spark workloads submitted with the following or later-released runtime versions:

• 1.1.55
• 1.2.0-RC1
• 2.0.63
• 2.1.42
• 2.2.0-RC15

Dataproc Serverless for Spark: Announcing the preview release of Autotuning Spark workloads.

New Dataproc Serverless for Spark runtime versions:

• 1.1.57
• 1.2.1
• 2.0.65
• 2.1.44
• 2.2.1

Added bigframes Python package by default in the Dataproc Serverless for Spark runtime versions 1.2 and 2.2

The following previously released sub-minor versions of Dataproc on Compute Engine images have been rolled back and can only be used when updating existing clusters that already use them:

• 2.0.97-debian10, 2.0.97-rocky8, 2.0.97-ubuntu18
• 2.1.45-debian11, 2.1.45-rocky8, 2.1.45-ubuntu20, 2.1.45-ubuntu20-arm
• 2.2.11-debian12, 2.2.11-rocky9, 2.2.11-ubuntu22

Dataproc Serverless for Spark: runtime version 2.2 will become the default Dataproc Serverless for Spark runtime version on May 3, 2024.

Note: This announcement was updated in the April 19, 2024 release note.

• New Dataproc on Compute Engine subminor image versions:

  • 2.0.97-debian10, 2.0.97-rocky8, 2.0.97-ubuntu18
  • 2.1.45-debian11, 2.1.45-rocky8, 2.1.45-ubuntu20, 2.1.45-ubuntu20-arm
  • 2.2.11-debian12, 2.2.11-rocky9, 2.2.11-ubuntu22

Note: the above subminor image versions were rolled back on April 2, 2024

Dataproc on Compute Engine: New Hadoop Google Secret Manager Credential Provider feature introduced in latest Dataproc on Compute Engine 2.0 image versions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.56
• 1.2.0
• 2.0.64
• 2.1.43
• 2.2.0

Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime versions 1.2 and 2.2, which include the following components:

• Spark 3.5.1
• BigQuery Spark Connector 0.36.1
• Cloud Storage Connector 3.0.0
• Conda 24.1
• Java 17
• Python 3.12
• R 4.3
• Scala 2.12 (1.2 runtime) and Scala 2.13 (2.2 runtime)

Dataproc Serverless for Spark:

• Upgraded Spark to version 3.5.1 in the latest 1.2 and 2.2 runtimes.
• Upgraded Conda to version 24.1 in the latest 1.2 and 2.2 runtimes.
• Upgraded Spark BigQuery connector to version 0.36.1 in the latest 1.2 and 2.2 runtimes.

New Dataproc on Compute Engine subminor image versions:

• 2.0.96-debian10, 2.0.96-rocky8, 2.0.96-ubuntu18
• 2.1.44-debian11, 2.1.44-rocky8, 2.1.44-ubuntu20, 2.1.44-ubuntu20-arm
• 2.2.10-debian12, 2.2.10-rocky9, 2.2.10-ubuntu22

Announcing the Preview release of Dataproc Serverless for Spark 1.2 runtime:

• Spark 3.5.0
• BigQuery Spark Connector 0.35.1
• Cloud Storage Connector 3.0.0
• Conda 23.11
• Java 17
• Python 3.12
• R 4.3
• Scala 2.12

New Dataproc Serverless for Spark runtime versions:

• 1.1.55
• 1.2.0-RC1
• 2.0.63
• 2.1.42
• 2.2.0-RC15

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS plugin to version 24.2.0 in the latest runtimes.
• Upgraded Spark to version 3.3.4 in the latest 1.1 and 2.0 runtimes.
• Backported SPARK-44198 in the latest 1.2 and 2.2 runtimes.

New Dataproc on Compute Engine subminor image versions:

• 2.0.95-debian10, 2.0.95-rocky8, 2.0.95-ubuntu18
• 2.1.43-debian11, 2.1.43-rocky8, 2.1.43-ubuntu20, 2.1.43-ubuntu20-arm
• 2.2.9-debian12, 2.2.9-rocky9, 2.2.9-ubuntu22

Added the bigframes (BigQuery DataFrames) Python package in the Dataproc Serverless for Spark 2.1 runtime.

New Dataproc Serverless for Spark runtime versions:

• 1.1.54
• 2.0.62
• 2.1.41
• 2.2.0-RC14

New Dataproc Serverless for Spark runtime versions:

• 1.1.53
• 2.0.61
• 2.1.40
• 2.2.0-RC13

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.20 version in the latest 1.1, 2.0, and 2.1 runtimes.

New Dataproc on Compute Engine subminor image versions:

• 2.0.94-debian10, 2.0.94-rocky8, 2.0.94-ubuntu18
• 2.1.42-debian11, 2.1.42-rocky8, 2.1.42-ubuntu20, 2.1.42-ubuntu20-arm
• 2.2.8-debian12, 2.2.8-rocky9, 2.2.8-ubuntu22

Dataproc on Compute Engine: Upgraded Cloud Storage connector version to 2.2.20 for 2.0 and 2.1 images.

Dataproc on Compute Engine: Mounted Java cacerts into containers by default when the Docker-on-YARN feature is enabled.

Dataproc Serverless for Spark: Extended Spark metrics collected for a batch now include executor:resultSize, executor:shuffleBytesWritten, and executor:shuffleTotalBytesRead.

New Dataproc Serverless for Spark runtime versions:

• 1.1.52
• 2.0.60
• 2.1.39
• 2.2.0-RC12

New Dataproc on Compute Engine subminor image versions:

• 2.0.93-debian10, 2.0.93-rocky8, 2.0.93-ubuntu18
• 2.1.41-debian11, 2.1.41-rocky8, 2.1.41-ubuntu20, 2.1.41-ubuntu20-arm
• 2.2.7-debian12, 2.2.7-rocky9, 2.2.7-ubuntu22

Dataproc on Compute Engine: The new Secret Manager credential provider feature is available in the latest 2.1 image versions.

Dataproc on Compute Engine:

• Upgraded Zookeeper to 3.8.3 for Dataproc 2.2.
• Upgraded ORC for Hive to 1.15.13 for Dataproc 2.1.
• Upgraded ORC for Spark to 1.7.10 for Dataproc 2.1.
• Extended expiry for the internal Knox Gateway certificate from one year to five years from cluster creation for Dataproc images 2.0, 2.1, and 2.2.

Dataproc on Compute Engine: Fixed ZooKeeper startup failures in image 2.2 HA (High Availability) clusters that use fully qualified hostnames.

New Dataproc Serverless for Spark runtime versions:

• 1.1.51
• 2.0.59
• 2.1.38
• 2.2.0-RC11

Dataproc on Compute Engine: The internalIpOnly cluster configuration setting now defaults to true for clusters created with 2.2 image versions. Also see Create a Dataproc cluster with internal IP addresses only.

Dataproc Serverless for Spark: Spark Lineage is available for Dataproc Serverless for Spark 1.1 runtime.

New Dataproc Serverless for Spark runtime versions:

• 1.1.50
• 2.0.58
• 2.1.37
• 2.2.0-RC10

Dataproc on Compute Engine: The new Secret Manager credential provider feature is available in the latest 2.2 image versions.

Dataproc on Compute Engine Ranger Cloud Storage enhancement:

• Enabled downscoping
• Added caching of tokens in local cache

Both settings are configurable and can be enabled by customers: see Use Ranger with caching and downscoping .

New Dataproc on Compute Engine subminor image versions:

• 2.0.92-debian10, 2.0.92-rocky8, 2.0.92-ubuntu18
• 2.1.40-debian11, 2.1.40-rocky8, 2.1.40-ubuntu20, 2.1.40-ubuntu20-arm
• 2.2.6-debian12, 2.2.6-rocky9, 2.2.6-ubuntu22

Dataproc on Compute Engine: Backported patch for HADOOP-18652.

New Dataproc Serverless for Spark runtime versions:

• 1.1.49
• 2.0.57
• 2.1.36
• 2.2.0-RC9

Dataproc Serverless for Spark: Backported patch for HADOOP-18652.

Dataproc on Compute Engine: Bucket ttl validation now also runs for buckets created by Dataproc.

Dataproc Serverless for Spark: When dynamic allocation is enabled, the initial executor number is determined by max of spark.dynamicAllocation.initialExecutors and spark.executor.instances.

Dataproc on Compute Engine: Added a warning during cluster creation if the cluster Cloud Storage staging bucket is using the legacy fine-grained/ACL IAM configuration instead of the recommended Uniform bucket-level access controls.

New Dataproc on Compute Engine subminor image versions:

• 2.0.91-debian10, 2.0.91-rocky8, 2.0.91-ubuntu18
• 2.1.39-debian11, 2.1.39-rocky8, 2.1.39-ubuntu20, 2.1.39-ubuntu20-arm
• 2.2.5-debian12, 2.2.5-rocky9, 2.2.5-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.48
• 2.0.56
• 2.1.35
• 2.2.0-RC8

Dataproc on Compute Engine: Backported patches for HIVE-21214, HIVE-23154, HIVE-23354 and HIVE-23614.

Dataproc is now available in the africa-south1 region (Johannesburg, South Africa).

The GitHub Ops Agent initialization action installs the Ops Agent on a Dataproc cluster, and provides metrics similar to the metrics that were enabled with the --metric-sources=monitoring-agent-defaults setting available for use with Dataproc images versions prior to version 2.2.

New Dataproc Serverless for Spark runtime versions:

• 1.1.47
• 2.0.55
• 2.1.34
• 2.2.0-RC7

New Dataproc on Compute Engine subminor image versions:

• 2.0.90-debian10, 2.0.90-rocky8, 2.0.90-ubuntu18
• 2.1.38-debian11, 2.1.38-rocky8, 2.1.38-ubuntu20, 2.1.38-ubuntu20-arm
• 2.2.4-debian12, 2.2.4-rocky9, 2.2.4-ubuntu22

Backport HIVE-19568: Active/Passive HiveServer2 HA: Disallow direct connection to passive instance.

Backport HIVE-27715: Remove ThreadPoolExecutorWithOomHook.

Dataproc on Compute Engine: The default yarn.nm.liveness-monitor.expiry-interval-ms Hadoop YARN setting has been changed in the latest image versions from 15000 (15 seconds) to 120000 (2 minutes).

New Dataproc on Compute Engine subminor image versions:

• 2.0.89-debian10, 2.0.89-rocky8, 2.0.89-ubuntu18
• 2.1.37-debian11, 2.1.37-rocky8, 2.1.37-ubuntu20, 2.1.37-ubuntu20-arm
• 2.2.3-debian12, 2.2.3-rocky9, 2.2.3-ubuntu22

Dataproc on Compute Engine: Upgraded Miniconda to 23.11, Python to 3.11, and curl to 8.5 to fix CVE-2023-38545 in the latest 2.2 images.

Dataproc on Compute Engine: Fixed the gsutil: command not found error in the latest Ubuntu images.

Dataproc on Compute Engine: Fixed Trino startup issue in the latest 2.2 images.

Dataproc on Compute Engine: Upgraded Cloud Storage connector version to 2.2.19 in the latest 2.0 and 2.1 images.

Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.19 version in the latest 1.1, 2.0, and 2.1 runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.46
• 2.0.54
• 2.1.33
• 2.2.0-RC6

Beginning March 31, 2024, when you submit a Dataproc Serverless Batch with a CMEK key:

• In addition to encrypting disk and Cloud Storage data, Dataproc Serverless will use your CMEK to also encrypt batch job arguments. This change will require that you assign the Cloud KMS CryptoKey Encrypter/Decrypter and the Service Usage Consumer role to the Dataproc Service Agent service account.
• batches.list will return an unreachable field that lists any batches with job arguments that couldn't be decrypted. You can issue a batches.get request to obtain more information on an unreachable batch.
• Multi-regional and cross-regional CMEKs will no longer be permitted. The key (CMEK) must be located in the same location as the encrypted resource. For example, the CMEK used to encrypt a batch that runs in the us-central1 region must also be located in the us-central1 region.

Dataproc Serverless for Spark:

• Upgraded Spark RAPIDS to version 23.12.1

• Upgraded the following components to the following versions in the latest 2.2 runtime:

  • Spark BigQuery connector version 0.35.0
  • Cloud Storage connector version 3.0.0
  • Conda version 23.11

New Dataproc Serverless for Spark runtime versions:

• 1.1.45
• 2.0.53
• 2.1.32
• 2.2.0-RC5

• Dataproc Spark Enhancements are now available in the Google Cloud console Dataproc cluster and job creation pages.

New Dataproc Serverless for Spark runtime versions:

• 1.1.44
• 2.0.52
• 2.1.31
• 2.2.0-RC4

The following previously released sub-minor versions of Dataproc images have been rolled back and can only be used when updating existing clusters that already use them:

• 2.0.88-debian10, 2.0.88-rocky8, 2.0.88-ubuntu18
• 2.1.36-debian11, 2.1.36-rocky8, 2.1.36-ubuntu20, 2.1.36-ubuntu20-arm
• 2.2.2-debian12, 2.2.2-rocky9, 2.2.2-ubuntu22

New Dataproc on Compute Engine subminor image versions:

• 2.0.88-debian10, 2.0.88-rocky8, 2.0.88-ubuntu18
• 2.1.36-debian11, 2.1.36-rocky8, 2.1.36-ubuntu20, 2.1.36-ubuntu20-arm

• 2.2.2-debian12, 2.2.2-rocky9, 2.2.2-ubuntu22

• Rollback Notice: See the January 4, 2024 release note rollback notice.

Dataproc on Compute Engine: Backported the patch for YARN-10975 in the latest 2.0 images.

Dataproc on Compute Engine: Changed the Hive Server2 and MetaStore maximum default JVM heap size to 32GiB. Previously, the limit was set to 1/4 of total node memory, which could be too large on large-memory machines.

New Dataproc Serverless for Spark runtime versions:

• 1.1.43
• 2.0.51
• 2.1.30
• 2.2.0-RC3

New Dataproc on Compute Engine subminor image versions:

• 2.0.87-debian10, 2.0.87-rocky8, 2.0.87-ubuntu18
• 2.1.35-debian11, 2.1.35-rocky8, 2.1.35-ubuntu20, 2.1.35-ubuntu20-arm
• 2.2.1-debian12, 2.2.1-rocky9, 2.2.1-ubuntu22

New Dataproc Serverless for Spark runtime versions:

• 1.1.42
• 2.0.50
• 2.1.29
• 2.2.0-RC2

Added the google-cloud-secret-manager Python package in the latest Dataproc Serverless for Spark runtimes.

Announcing the GA release of Dataproc on Compute Engine image version 2.2 :

• 2.2.0-debian12, 2.2.0-rocky9, 2.2.0-ubuntu22

The 2.2.0 release includes the following components:

• Debian-12 / Ubuntu-2204 / RockyLinux 9
• Apache Hadoop 3.3.6
• Apache Spark 3.5.0
• Spark-BigQuery Connector 0.34.0
• Cloud Storage Connector 3.0.0
• Trino 432
• Apache Flink 1.17.0
• Apache Ranger 2.4.0
• Apache Solr 9.2.1
• R 4.2
• Hue 4.11.0
• JupyterLab Notebook 3.6

Monitoring-agent-defaults metrics are not available in Dataproc on Compute Engine image version 2.2 clusters unless the Ops Agent is installed. Other metrics for Dataproc provided components will continue to work.

Blocklisted the following Dataproc on Compute Engine Images due to issue with increase in startup time:

• 2.0.86-debian10, 2.0.86-rocky8, 2.0.86-ubuntu18
• 2.1.34-debian11, 2.1.34-rocky8, 2.1.34-ubuntu20, 2.1.34-ubuntu20-arm

New Dataproc Serverless for Spark runtime versions:

• 1.1.41
• 2.0.49
• 2.1.28
• 2.2.0-RC1

Announcing the Preview release of Dataproc Serverless for Spark 2.2 runtime:

• Spark 3.5.0
• BigQuery Spark Connector 0.34.0
• Cloud Storage Connector 3.0.0-RC1
• Conda 23.10
• Java 17
• Python 3.12
• R 4.3
• Scala 2.13

New Dataproc on Compute Engine subminor image versions:

• 2.0.85-debian10, 2.0.85-rocky8, 2.0.85-ubuntu18
• 2.1.33-debian11, 2.1.33-rocky8, 2.1.33-ubuntu20, 2.1.33-ubuntu20-arm

Added the Confidential Computing option on the "Manage Security" panel on the "Create a Dataproc cluster on Compute Engine" page in the Google Cloud console.

Updated the Zookeeper component version from 3.8.0 to 3.8.3 in the latest Dataproc on Compute Engine 2.1 image version.

Fixed Dataproc Hub issue in latest Dataproc on Compute Engine 2.1 image.

Backported HIVE-21698 in Hive 3.1.3 component in latest Dataproc on Compute Engine image versions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.40
• 2.0.48
• 2.1.27

The Cloud Storage connector has been upgraded to version 2.2.18 in all Dataproc Serverless for Spark runtimes.

In the Flink component in the latest Dataproc on Compute Engine 2.1 image version, added the following java-storage client properties:

• gs.retry.max.attempts property to set the max number of retry attempts
• gs.retry.total.timeout property to set the total retry timeout

Upgraded the Cloud Storage connector version to 2.2.18 in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.

New Dataproc on Compute Engine subminor image versions:

• 2.0.84-debian10, 2.0.84-rocky8, 2.0.84-ubuntu18
• 2.1.32-debian11, 2.1.32-rocky8, 2.1.32-ubuntu20, 2.1.32-ubuntu20-arm
• 2.2.0-RC3-debian11/-ubuntu22/-rocky9

Fixed a regression in the Zeppelin websocket rules that caused a websocket error in Zeppelin notebooks.

The Python kernel does not work in Zeppelin on the Dataproc on Compute Engine 2.1 image version. Other kernels are not impacted.

The Zeppelin REST API does not work (drops query parameters) on Dataproc on Compute Engine 2.0 and 2.1 image versions via the Component Gateway. Other Zeppelin interactions can also break as a result of dropped query parameters.

You can use CMEK (Customer Managed Encrytion Keys) with encrypted Dataproc cluster data, incuding persistent disk data, job arguments and queries submitted with Dataproc jobs, and cluster data saved in the cluster Dataproc staging bucket. See Use CMEK with cluster data for more information.

Announcing the General Availability (GA) release of Dataproc Jupyter Plugin and its availability in Vertex AI Workbench instance notebooks.

New Dataproc on Compute Engine subminor image versions:

• 2.0.83-debian10, 2.0.83-rocky8, 2.0.83-ubuntu18
• 2.1.31-debian11, 2.1.31-rocky8, 2.1.31-ubuntu20, 2.1.31-ubuntu20-arm

Announcing the release of Workflow Template CMEK (Customer Managed Encryption Key) encryption. Use this feature to apply CMEK encryption to workflow template job arguments. For example, when this feature is enabled, the query string of a workflow template SparkSQL job is encrypted using CMEK.

You can now use Dataproc Serverless autoscaling V2 to help you manage Dataproc Serverless workloads, improve workload performance, and save costs.

Set spark.shuffle.mapOutput.minSizeForBroadcast=128m to fix SPARK-38101 when Dataproc Serverless Spark dynamic allocation is enabled.

Announcing the Preview release of Dataproc Flexible VMs. This feature lets you specify prioritized lists of secondary worker VM types that Dataproc will select from when creating your cluster. Dataproc will select the VM type with sufficient available capacity while taking quotas and reservations into account.

New Dataproc on Compute Engine subminor image versions:

• 2.0.82-debian10, 2.0.82-rocky8, 2.0.82-ubuntu18
• 2.1.30-debian11, 2.1.30-rocky8, 2.1.30-ubuntu20, 2.1.30-ubuntu20-arm

Added spark.dataproc.scaling.version=2 config to let customers control the Dataproc Serverless for Spark autoscaling version.

Increased the TTL for Dataproc on Compute Engine custom images from 60 days to 365 days.

Fixed Knox rewrite rules for Zeppelin URLs in some cases in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.37
• 2.0.45
• 2.1.24

Announcing the General Availability (GA) release of Dataproc Serverless GPU accelerators.

New Dataproc Serverless for Spark runtime versions:

• 1.1.36
• 2.0.44
• 2.1.23

Dataproc on Compute Engine: Dataproc now collects the dataproc.googleapis.com/job/yarn/vcore_seconds and dataproc.googleapis.com/job/yarn/memory_seconds job-level resource attribution metrics to track YARN application vcore and memory usage during job execution. These metrics are collected by default and are not chargeable to customers.

Dataproc on Compute Engine: Properties dataproc:agent.ha.enabled and dataproc:componentgateway.ha.enabled now default to true to provide high availability for the Dataproc Agent and Component Gateway.

Dataproc on Compute Engine: Dataproc now collects a dataproc.googleapis.com/node/yarn/nodemanager/health health metric to track the health of individual YARN node managers running on VMs. This metric is written against the gce_instance monitored resource to help you find suspect nodes. It is collected by default and is not chargeable to customers.

New Dataproc Serverless for Spark runtime versions:

• 1.1.35
• 2.0.43
• 2.1.22

New Dataproc on Compute Engine subminor image versions:

• 2.0.80-debian10, 2.0.80-rocky8, 2.0.80-ubuntu18
• 2.1.28-debian11, 2.1.28-rocky8, 2.1.28-ubuntu20, 2.1.28-ubuntu20-arm

Announcing the General Availability (GA) release of Dataproc Serverless for Spark Interactive sessions.

New Dataproc on Compute Engine image version 2.2 is available for preview with upgraded components.

Upgraded Hadoop version from 3.3.3 to 3.3.6 in the latest Dataproc on Compute Engine 2.1 image version.

New Dataproc on Compute Engine subminor image versions:

• 2.0.79-debian10, 2.0.79-rocky8, 2.0.79-ubuntu18
  
• 2.1.27-debian11, 2.1.27-rocky8, 2.1.27-ubuntu20, 2.1.27-ubuntu20-arm
• 2.2.0-RC2-debian11, 2.2.0-RC2-rocky9, 2.2.0-RC2-ubuntu22

Upgraded the Cloud Storage connector version to 2.2.17 in the latest Dataproc Serverless for Spark runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.34
• 2.0.42
• 2.1.21

Added the gs.filesink.entropy.enabled property in Flink to enable entropy injection in filesink Cloud Storage path in the latest Dataproc on Compute Engine 2.1 image version.

Added the gs.http.connect-timeout and gs.http.read-timeout properties in Flink to set the connection timeout and read timeout for java-storage client in the latest Dataproc on Compute Engine 2.1 image version.

New Dataproc on Compute Engine subminor image versions:

• 2.0.78-debian10, 2.0.78-rocky8, 2.0.78-ubuntu18
  
• 2.1.26-debian11, 2.1.26-rocky8, 2.1.26-ubuntu20, 2.1.26-ubuntu20-arm

Upgraded the Cloud Storage connector version to 2.2.17 in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.

Upgraded Hive version from 3.1.2 to 3.1.3 in the latest Dataproc on Compute Engine 2.0 image version.

New Dataproc on Compute Engine subminor image versions:

• 2.0.77-debian10, 2.0.77-rocky8, 2.0.77-ubuntu18
  
• 2.1.25-debian11, 2.1.25-rocky8, 2.1.25-ubuntu20, 2.1.25-ubuntu20-arm

In the latest Dataproc on Compute Engine 2.0 and 2.1 image versions, unset the CLOUDSDK_PYTHON variable to allow the gcloud command-line tool to use its bundled Python interpreter.

Fixed Jupyter notebooks bug that made Scala compilation errors invisible with the Toree kernel in Dataproc on Compute Engine 2.1 images.

New Dataproc Serverless for Spark runtime versions:

• 1.1.32
• 2.0.40
• 2.1.19

Dataproc is now available in the me-central2 region (Dammam, Saudi Arabia).

New Dataproc on Compute Engine subminor image versions:

• 2.0.76-debian10, 2.0.76-rocky8, 2.0.76-ubuntu18
• 2.1.24-debian11, 2.1.24-rocky8, 2.1.24-ubuntu20, 2.1.24-ubuntu20-arm

New Dataproc Serverless for Spark runtime versions:

• 1.1.31
• 2.0.39
• 2.1.18

Scala has been upgraded to version 2.12.18 and Apache Tez has been upgraded to version 0.10.2 in Dataproc on Compute Engine 2.1 images.

Announcing the Private Preview release of the Dataproc on Compute Engine Flink Jobs resource. During Private Preview, you can contact your Google Cloud Sales representative to have your project(s) added to an allowlist to allow you to submit Flink jobs to the Dataproc on Compute Engine service.

The dataproc.diagnostics.enabled property is now avaiable to enable running diagnostics on Dataproc Serverless for Spark. The existing spark.dataproc.diagnostics.enabled property will be deprecated for use with newer runtimes.

Dataproc Auto zone placement for clusters is now available in the Google Cloud console by selecting the "Any" option for the cluster zone.

New Dataproc Serverless for Spark runtime versions:

• 1.1.30
• 2.0.38
• 2.1.17

The Apache Spark version has been upgraded from 3.3.0 to 3.3.2 in Dataproc on Compute Engine 2.1 images.

New Dataproc on Compute Engine subminor image versions:

• 2.0.75-debian10, 2.0.75-rocky8, 2.0.75-ubuntu18
• 2.1.23-debian11, 2.1.23-rocky8, 2.1.23-ubuntu20, 2.1.23-ubuntu20-arm

Announcing the General Availability (GA) release of Data Lineage for Dataproc, which captures data transformations (lineage events) in Dataproc Spark jobs, and publishes them to Dataplex Lineage.

Dataproc Serverless Interactive sessions detail and list pages are now available in the Google Cloud console.

Announcing the Preview release of Dataproc Serverless for Spark Interactive sessions and the Dataproc Jupyter Plugin.

New Dataproc Serverless for Spark runtime versions:

• 1.1.29
• 2.0.37
• 2.1.16

Fixed a Dataproc Serverless issue where Spark batches failed with unhelpful error messages.

Dataproc is now available in the europe-west10 region (Berlin).

New Dataproc Serverless for Spark runtime versions:

• 1.1.28
• 2.0.36
• 2.1.15

New Dataproc on Compute Engine subminor image versions:

• 2.0.74-debian10, 2.0.74-rocky8, 2.0.74-ubuntu18
• 2.1.22-debian11, 2.1.22-rocky8, 2.1.22-ubuntu20, 2.1.22-ubuntu20-arm

Backported the patches for HIVE-20618 in the new Dataproc on Compute Engine 2.0 and 2.1 images.

New Dataproc on Compute Engine subminor image versions:

• 2.0.73-debian10, 2.0.73-rocky8, 2.0.73-ubuntu18
• 2.1.21-debian11, 2.1.21-rocky8, 2.1.21-ubuntu20, 2.1.21-ubuntu20-arm

New Dataproc Serverless for Spark runtime versions:

• 1.1.27
• 2.0.35
• 2.1.14

Added new Dataproc Serverless Templates for batch workload creation:

• Cloud Spanner to Cloud Storage
• Cloud Storage to JDBC
• Cloud Storage to Cloud Storage
• Hive to BigQuery
• JDBC to Cloud Spanner
• JDBC to JDBC
• Pub/Sub to Cloud Storage

Improved the reliability of Dataproc Serverless compute node initialization with a Premium disk tier option.

Added a dataproc:dataproc.cluster.caching.enabled flag to enable and disable Dataproc on Compute Engine cluster caching. The flag is false by default. Use this feature with the latest Dataproc on Compute Engine images.

The following previously released sub-minor versions of Dataproc on Compute Engine images unintentionally reverted several dependency library versions. This caused a risk of backward-incompatibility for some workloads.

These sub-minor versions have been rolled back, and can only be used when updating existing clusters that already use them:

• 2.0.71-debian10, 2.0.71-rocky8, 2.0.71-ubuntu18
• 2.1.19-debian11, 2.1.19-rocky8, 2.1.19-ubuntu20, 2.1.19-ubuntu20-arm

Backported ZEPPELIN-5434 to image 2.1 to fix CVE-2022-2048.

Upgraded Hudi to 0.12.3 and added the BigQuery Sync tool as part of the Hudi optional component.

New Dataproc on Compute Engine image versions:

• 2.0.72-debian10, 2.0.72-rocky8, 2.0.72-ubuntu18
• 2.1.20-debian11, 2.1.20-rocky8, 2.1.20-ubuntu20, 2.1.20-ubuntu20-arm

Downgraded Cloud Storage connector version to 2.2.15 in all Dataproc on Compute Engine image versions to prevent potential performance regression.

Backported the patches for HIVE-22170 and HIVE-22331.

New Dataproc Serverless for Spark runtime versions:

• 1.1.26
• 2.0.34
• 2.1.13

Downgraded Cloud Storage connector to 2.2.15 version in all Dataproc Serverless for Spark runtimes to prevent potential performance regression.

New Dataproc on Compute Engine image versions:

• 2.0.71-debian10, 2.0.72-rocky8, 2.0.72-ubuntu18
• 2.1.19-debian11, 2.1.20-rocky8, 2.1.20-ubuntu20, 2.1.20-ubuntu20-arm

Note: The above image versions were rolled back. See the August 6, 2023 release note

The Maximum total memory per core for Dataproc Serverless Premium compute tiers has increased to 24576m (7424m for Standard compute tiers unchanged). See Dataproc Serverless Resource allocation properties.

New Dataproc Serverless for Spark runtime versions:

• 1.1.25
• 2.0.33
• 2.1.12

Clusters cannot be created with a driver node group if the cluster image version is older than 2.0.57 or 2.1.5, or if the permissions for the staging bucket are missing.

Added recommendation details in Autoscaler Stackdriver logs for the CANCEL and DO_NOT_CANCEL recommendations.

New Dataproc Serverless for Spark runtime versions:

• 1.1.24
• 2.0.32
• 2.1.11

Fixed a race condition in Spark startup that could lead to nodes failing to initialize when using premium disk tier.

New Dataproc on Compute Engine image versions, which includes a 2.1.18-ubuntu20-arm image that supports ARM machine types:

• 2.0.70-debian10, 2.0.70-rocky8, 2.0.70-ubuntu18
• 2.1.18-debian11, 2.1.18-rocky8, 2.1.18-ubuntu20, 2.1.18-ubuntu20-arm

New Dataproc Serverless for Spark runtime versions:

• 1.1.23
• 2.0.31
• 2.1.10

Upgraded the Cloud Storage connector version to 2.2.16 in Dataproc Serverless for Spark runtimes.

Clusters that use a driver node group now configure YARN queues with user-limit-factor set to 2, allowing for a single user to burst to 2x utilization of capacity, which is set to 50. This achieves better resource utilization for workloads submitted by a single user.

New Dataproc on Compute Engine image versions:

• 2.0.69-debian10, 2.0.69-rocky8, 2.0.69-ubuntu18
• 2.1.17-debian11, 2.1.17-rocky8, 2.1.17-ubuntu20

Upgraded the Cloud Storage connector version to 2.2.16 for Dataproc on Compute Engine 2.0 and 2.1 images.

Dataproc Serverless Spark 1.1 and 2.0 runtime subminor versions can now be used 365 days after their release (instead of 90 days).

The goog-dataproc-batch-id, goog-dataproc-batch-uuid and goog-dataproc-location labels are now automatically applied to Dataproc Serverless batch resources.

Dataproc Serverless for Spark now supports updating the BigQuery connector using the dataproc.sparkBqConnector.version and dataproc.sparkBqConnector.uri properties see Use the BigQuery connector with Dataproc Serverless for Spark.

New Dataproc Serverless for Spark runtime versions:

• 1.1.22
• 2.0.30
• 2.1.9

Added support for Premium compute and storage pricing tiers for Dataproc Serverless Spark workloads. Premium compute offers higher performance per core, and Premium storage offers higher throughput and IOPs. To use Premium compute and storage, set the following Spark runtime environment properties:

• spark.dataproc.(driver|executor).compute.tier=premium
• spark.dataproc.(driver|executor).storage.tier=premium.

New Dataproc Serverless for Spark runtime versions:

• 1.1.21
• 2.0.29
• 2.1.8

Backported ZEPPELIN-5755 to Zeppelin 0.10 in 2.1 images for Spark 3.3 support.

New Dataproc on Compute Engine subminor image versions:

• 2.0.68-debian10, 2.0.68-rocky8, 2.0.68-ubuntu18
• 2.1.16-debian11, 2.1.16-rocky8, 2.1.16-ubuntu20

Added Dataproc Serverless Templates for batch creation:

• Cloud Storage to BigQuery
• Cloud Storage to Cloud Spanner
• Hive to Cloud Storage
• JDBC to BigQuery
• JDBC to Cloud Storage

New Dataproc Serverless for Spark runtime versions:

• 1.1.20
• 2.0.28
• 2.1.7

New Dataproc on Compute Engine subminor image versions:

• 2.0.67-debian10, 2.0.67-rocky8, 2.0.67-ubuntu18
• 2.1.15-debian11, 2.1.15-rocky8, 2.1.15-ubuntu20

Fixed a bug that caused cluster creation to fail when ATSv2 is enabled for tables that have a garbage collection policy setup other than maxversions.

New Dataproc Serverless for Spark runtime versions:

• 1.1.19
• 2.0.27
• 2.1.6

New Dataproc Serverless for Spark runtime versions:

• 1.1.18
• 2.0.26
• 2.1.5

Upgrade Cloud Storage connector to 2.2.14 version in Dataproc Serverless for Spark runtimes.

New Dataproc Serverless for Spark runtime versions:

• 1.1.17
• 2.0.25
• 2.1.4

Upgrade Cloud Storage connector version to 2.2.14 for 2.0 and 2.1 images

New sub-minor versions of Dataproc images:

• 2.0.66-debian10, 2.0.66-rocky8, 2.0.66-ubuntu18
• 2.1.14-debian11, 2.1.14-rocky8, 2.1.14-ubuntu20

Backport HIVE-22891, HIVE-21660, HIVE-21915 to 2.0 images.

Backport HIVE-22891, HIVE-21660, HIVE-25520, HIVE-25521 to 2.1 images.

New sub-minor versions of Dataproc images:

• 2.0.65-debian10, 2.0.65-rocky8, 2.0.65-ubuntu18
• 2.1.13-debian11, 2.1.13-rocky8, 2.1.13-ubuntu20

New Dataproc Serverless for Spark runtime versions:

• 1.1.16
• 2.0.24
• 2.1.3

Unauthorized callers attempting to get, delete, or terminate non-existent Sessions will now receive a 403 response code instead of a 404 response code. This does not impact authorized callers.

Upgraded the Cloud Storage connector to 2.2.13 version in Dataproc on Compute Engine 2.0 and 2.1 image versions.

Fixed Serverless history server endpoint URL when Persistent History Server (PHS) was setup without using a wildcard.

New Dataproc Serverless for Spark runtime versions:

• 1.1.15
• 2.0.23
• 2.1.2

Upgraded the Cloud Storage connector to 2.2.13 version in Dataproc Serverless for Spark runtimes.

Backported HIVE-22891 to 2.0 images.

Fixed the NoClassDefFoundError for log4j class in Zeppelin BigQuery interpreter in 2.0 images.

New sub-minor versions of Dataproc images:

• 2.0.64-debian10, 2.0.64-rocky8, 2.0.64-ubuntu18
• 2.1.12-debian11, 2.1.12-rocky8, 2.1.12-ubuntu20

You can now use --properties=dataproc:componentgateway.ha.enabled=true to enable the Dataproc Component Gateway and Knox along with the Spark History Server (SHS) UI in HA mode.

New Dataproc Serverless for Spark runtime versions:

• 1.1.14
• 2.0.22
• 2.1.1

Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime version 2.1, which includes the following components:

• Spark 3.4.0
• BigQuery Spark Connector 0.28.1
• Cloud Storage Connector 2.2.11
• Conda 23.3
• Java 17
• Python 3.11
• R 4.2
• Scala 2.13

Upgraded Conda to 23.3 in Dataproc Serverless for Spark runtime 2.1.

New Dataproc Serverless for Spark runtime versions:

• 1.1.13
• 2.0.21
• 2.1.0

New Dataproc Serverless for Spark runtime versions:

• 1.1.12
• 2.0.20
• 2.1.0-RC8

hive principal will be used for Hive catalog queries via presto in kerberos cluster.

New sub-minor versions of Dataproc images:

• 1.5.89-debian10, 1.5.89-rocky8, 1.5.89-ubuntu18
• 2.0.63-debian10, 2.0.63-rocky8, 2.0.63-ubuntu18
• 2.1.11-debian11, 2.1.11-rocky8, 2.1.11-ubuntu20

Upgrade Spark to 3.4.0 and its dependencies in Dataproc Serverless for Spark 2.1 runtime:

• Jetty to 9.4.51.v20230217
• ORC to 1.8.3
• Parquet to 1.13.0
• Protobuf to 3.22.3

Dataproc now supports the usage of cross-project service account.

Autoscaler recommendation reasoning details are available now in Cloud Logging logs.

Default batch TTL is set to 4 hours for Dataproc Serverless for Spark runtime version 2.1.

New sub-minor versions of Dataproc images:

• 1.5.88-debian10, 1.5.88-rocky8, 1.5.88-ubuntu18
• 2.0.62-debian10, 2.0.62-rocky8, 2.0.62-ubuntu18
• 2.1.10-debian11, 2.1.10-rocky8, 2.1.10-ubuntu20

Running Spark jobs with the DataprocFileOutoutputCommitter is now supported. Enable the committer for Spark applications that write to a Cloud Storage destination concurrently.

Add Autoscaler recommendation reasoning details in Cloud Logging.

Dataproc on GKE SLM force delete timeout exception converted to DataprocIoException.

Announcing Dataproc General Availability (GA) support for CMEK organization policy.

New Dataproc Serverless for Spark runtime versions:

• 1.1.11
• 2.0.19
• 2.1.0-RC7

Make spark user an owner for all items in the driver working directory for Dataproc Serverless for Spark workloads to fix permissions issues after Hadoop upgrade to 3.3.5.

New Dataproc Serverless for Spark runtime versions:

• 1.1.10
• 2.0.18
• 2.1.0-RC6

Upgrade Hadoop to 3.3.5 in Dataproc Serverless for Spark runtimes

Announcing the General Availability (GA) release of Key Access Justifications for Dataproc.

Dataproc is now available in the me-central1 region (Doha).

New sub-minor versions of Dataproc images:

• 1.5.87-debian10, 1.5.87-rocky8, 1.5.87-ubuntu18
• 2.0.61-debian10, 2.0.61-rocky8, 2.0.61-ubuntu18
• 2.1.9-debian11, 2.1.9-rocky8, 2.1.9-ubuntu20

Dataproc cluster creation now supports the pd-extreme disk type.

Dataproc on GKE now disallows update operations.

Dataproc on GKE diagnose operation now verifies that the master agent is running.

New sub-minor versions of Dataproc images:

• 1.5.86-debian10, 1.5.86-rocky8, 1.5.86-ubuntu18
• 2.0.60-debian10, 2.0.60-rocky8, 2.0.60-ubuntu18
• 2.1.8-debian11, 2.1.8-rocky8, 2.1.8-ubuntu20

New Dataproc Serverless for Spark runtime versions:

• 1.1.9
• 2.0.17
• 2.1.0-RC5

New Dataproc Serverless for Spark runtime versions:

• 1.1.8
• 2.0.16
• 2.1.0-RC4

Upgrade Python to 3.11 and Conda to 23.1 in Dataproc Serverless for Spark runtime 2.1

Dataproc is now available in the europe-west12 region (Turin).

New Dataproc Serverless for Spark runtime versions:

• 1.1.7
• 2.0.15
• 2.1.0-RC3

• Upgrade Flink to 1.15.3 from 1.15.0 in 2.1 images

New sub-minor versions of Dataproc images:

• 1.5.85-debian10, 1.5.85-rocky8, 1.5.85-ubuntu18
• 2.0.59-debian10, 2.0.59-rocky8, 2.0.59-ubuntu18
• 2.1.7-debian11, 2.1.7-rocky8, 2.1.7-ubuntu20

New Dataproc Serverless for Spark runtime versions:

• 1.1.6
• 2.0.14
• 2.1.0-RC2

Upgraded Spark BigQuery connector version to 0.28.1 in 1.1 and 2.1 Dataproc Serverless for Spark runtimes.

Added stronger validations to disallow upper-case characters in template IDs per Resource Names guidance, which allows Workflow template creation to fail fast instead of failing at workflow template instantiation.

Added decision metric field in Stackdriver autoscaler logs.

New Dataproc Serverless for Spark runtime versions:

• 1.1.5
• 2.0.13
• 2.1.0-RC1

Release Dataproc Serverless for Spark runtime 2.1 preview:

• Spark 3.4.0-rc1
• BigQuery Spark Connector 0.28.0
• Cloud Storage Connector 2.2.11
• Conda 22.11
• Java 17
• Python 3.10
• R 4.2
• Scala 2.13

--properties=dataproc:agent.ha.enabled=true can now be used to enable the Dataproc Agent in high availability mode. This property is supported by Dataproc Image versions 2.0 and above.

Upgrade Spark to 3.3.2 and its dependencies in 1.1 and 2.0 Dataproc Serverless for Spark runtimes:

• Jackson to 2.13.5
• Jetty to 9.4.50.v20221201
• ORC to 1.8.2
• Protobuf to 3.21.12
• RoaringBitmap to 0.9.39

New Dataproc Serverless for Spark runtime versions:

• 1.1.4
• 2.0.12

New Dataproc Serverless for Spark runtime versions:

• 1.1.3
• 2.0.11

New sub-minor versions of Dataproc images:

• 1.5.82-debian10, 1.5.82-rocky8, 1.5.82-ubuntu18
• 2.0.56-debian10, 2.0.56-rocky8, 2.0.56-ubuntu18
• 2.1.4-debian11, 2.1.4-rocky8, 2.1.4-ubuntu20

New Dataproc Serverless for Spark runtime versions:

• 1.1.2
• 2.0.10

Add support for filters when listing batches. Batches may be filtered on one or more of batch_id, batch_uuid, state, or create_time (for example, state = RUNNING AND create_time < "2023-01-01T00:00:0Z"). See Filter expressions for more information.

Dataproc Serverless for Spark now supports unconditional TTL to batches. The workload will be terminated after the TTL without waiting for work to complete.

Dataproc Serverless for Spark now supports statically-sized Dataproc Serverless for Spark batch workloads with more than 500 executors.

Generate a warning when initialization actions are used in a cluster created with a driver node group.

The default Dataproc Serverless for Spark runtime version has changed to 2.0.

New Dataproc Serverless for Spark runtime versions:

• 1.0.29
• 1.1.1
• 2.0.9

1.0.29 is the last release of the Dataproc Serverless for Spark runtime version 1.0, it will no longer be supported and will not receive new releases.

Upgrade Cloud Storage connector to 2.2.11 version in Dataproc Serverless for Spark runtimes.

Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime version 1.1, which includes the following components:

• Spark 3.3.1
• BigQuery Spark Connector 0.28.0
• Cloud Storage Connector 2.2.9
• Conda 22.11
• Java 11
• Python 3.10
• R 4.2
• Scala 2.12

New Dataproc Serverless for Spark runtime versions:

• 1.0.28
• 1.1.0
• 2.0.8

Dataproc Serverless for Spark runtime version 1.0 changed to non-LTS because of the release of backward-compatible Dataproc Serverless for Spark runtime version 1.1 LTS.

Announcing the General Availability (GA) release of the Dataproc Hudi Optional Component.

Announcing the General Availability (GA) release of the Dataproc driver node groups.

New Dataproc Serverless for Spark runtime versions:

• 1.0.27
• 2.0.7

Added support for enabling Hive Metastore OSS metrics by passing hivemetastore to --metric-sources property during cluster creation.

Added support for Dataproc Metastore integration with Trino.

New sub-minor versions of Dataproc images:

• 1.5.80-debian10, 1.5.80-rocky8, 1.5.80-ubuntu18
• 2.0.54-debian10, 2.0.54-rocky8, 2.0.54-ubuntu18
• 2.1.2-debian11, 2.1.2-rocky8, 2.1.2-ubuntu20

Upgraded Parquet to 1.12.2 for 2.1 images.

The value of hive.server2.builtin.udf.blacklist is now set by default to reflect,reflect2 in hive-site.xml to prevent arbitrary code execution.

New Dataproc Serverless for Spark runtime versions:

• 1.0.26
• 2.0.6

Backported Spark patch in Dataproc Serverless for Spark runtime 1.0 and 2.0:

• SPARK-40481: Ignore stage fetch failure caused by decommissioned executor.

New sub-minor versions of Dataproc images:

• 1.5.79-debian10, 1.5.79-rocky8, 1.5.79-ubuntu18
• 2.0.53-debian10, 2.0.53-rocky8, 2.0.53-ubuntu18
• 2.1.1-debian11, 2.1.1-rocky8, 2.1.1-ubuntu20

New Dataproc Serverless for Spark runtime versions:

• 1.0.25
• 2.0.5

General Availability (GA) release of Dataproc 2.1 images.

Upgrade Cloud Storage connector version to 2.1.9 for 1.5 images.

Upgrade Cloud Storage connector version to 2.2.9 for 2.1 images.

New sub-minor versions of Dataproc images:

• 1.5.78-debian10, 1.5.78-rocky8, 1.5.78-ubuntu18
• 2.0.52-debian10, 2.0.52-rocky8, 2.0.52-ubuntu18
• 2.1.0-debian11, 2.1.0-rocky8, 2.1.0-ubuntu20

New Dataproc Serverless for Spark runtime versions: 1.0.24 and 2.0.4

Dataproc Serverless for Spark runtime 1.0:

• Upgrade to Spark to 3.2.3
• Upgrade Cloud Storage connector to 2.2.9
• Upgrade Spark dependencies:
  • Jetty to 9.4.49.v20220914
  • ORC to 1.7.7
  • Protobuf to 3.19.6
  • RoaringBitmap to 0.9.35
  • Scala to 2.12.17

Dataproc Serverless for Spark runtime 2.0:

• Upgrade Cloud Storage connector to 2.2.9
• Upgrade Spark dependencies:
  • Protobuf to 3.21.9
  • RoaringBitmap to 0.9.35

Use jemalloc as a default OS memory allocator in Dataproc Serverless for Spark runtime.

Backport Spark patches in Dataproc Serverless for Spark runtime 1.0 and 2.0:

• SPARK-39324: Log ExecutorDecommission as INFO level in TaskSchedulerImpl
• SPARK-40168: Handle SparkException during shuffle block migration
• SPARK-40269: Randomize the orders of peer in BlockManagerDecommissioner
• SPARK-40778: Make HeartbeatReceiver as an IsolatedRpcEndpoint

Added the dataproc.googleapis.com/job/state metric to track the status of Dataproc Jobs states (such as, RUNNING or PENDING). This metric is collected by default and is not chargeable to customers.

Dataproc job IDs are now queryable and viewable from MQL(Monitoring Query Language), and the metric can be used for long-running job monitoring and alerting.

Dataproc Serverless for Spark runtime version 2.0 will become the default Dataproc Serverless for Spark runtime version on January 24, 2023 (instead of December 13, 2022, as previously announced).

Dataproc Serverless for Spark supports Spark and System metrics. These metrics are enabled by default. Spark driver and executor metrics can be customised using overrides.

Nodemanagers in DECOMMISSIONING, NEW, and SHUTDOWN state are now included in the /cluster/yarn/nodemanagers metric.

Secure Boot, Virtual trusted platform module (vTPM), and Integrity monitoring Shielded VM features are enabled by default for Dataproc on Compute Engine clusters that use 2.1 preview images.

Added support for Dataproc to attach to a gRPC Dataproc Metastore in any region.

Dataproc Serverless for Spark now shows the subminor runtime version used in the runtimeConfig.version field,

Fixed a bug that caused a Dataproc cluster with a Dataproc Metastore service to fail the creation process, if the cluster was in the same network but different subnetworks.

New sub-minor versions of Dataproc images:

1.5.77-debian10, 1.5.77-rocky8, 1.5.77-ubuntu18,

2.0.51-debian10, 2.0.51-rocky8, 2.0.51-ubuntu18,

preview 2.1.0-RC4-debian11, preview 2.1.0-RC4-rocky8, preview 2.1.0-RC4-ubuntu20.

Dataproc Serverless for Spark now now uses runtime version 1.0.23 and 2.0.3.

Backported HIVE-17317 in the latest 2.0 and 2.1 images.

Downgraded google-auth-oauthlib Python package to fix gcsfs Python package for 2.0 and 2.1 images.

Dataproc Serverless for Spark runtime version 1.0.23 and 2.0.3 downgrades google-auth-oauthlib Python package to fix gcsfs Python package.

Upgraded Apache Commons Text to 1.10.0 for Knox in 1.5 images, and for Spark, Pig, Knox in 2.0 images, addressing CVE-2022-42889.

Dataproc Serverless for Spark runtime version 1.0.23 and 2.0.3 adds PyMongo Python library.

Dataproc images 2.0.50 and preview 2.1.0-RC3 are deprecated and cluster creations based on these images will fail starting 11/11/2022. This is due to an update to the google auth library which breaks running Pyspark batch workloads having dependency on gcsfs. Upcoming image versions will have a fix to address this issue.

Dataproc Serverless for Spark runtime versions 1.0.22 and 2.0.2 will be deprecated on 11/11/2022. New batch submissions that use these runtime versions will fail starting 11/11/2022. This is due to an update to the google auth library which breaks running Pyspark batch workloads having dependency on gcsfs. Upcoming runtime versions will address this issue.

New sub-minor versions of Dataproc images:

1.5.76-debian10, 1.5.76-rocky8, 1.5.76-ubuntu18

2.0.50-debian10, 2.0.50-rocky8, 2.0.50-ubuntu18

preview 2.1.0-RC3-debian11, preview 2.1.0-RC3-rocky8, preview 2.1.0-RC3-ubuntu20,

Dataproc Serverless for Spark now now uses runtime version 1.0.22 and 2.0.2.

If a Dataproc Metastore service uses the gRPC endpoint protocol, a Dataproc or self-managed cluster located in any region can attach to the service.

Dataproc Serverless for Spark now allows the customization of driver and executor memory using the following properties:

• spark.driver.memory
• spark.driver.memoryOverhead
• spark.executor.memory
• spark.executor.memoryOverhead

Dataproc Serverless for Spark now outputs approximate_usage after a workload finishes that shows the approximate DCU and shuffle storage resource consumption by the workload.

Removed the Auto Zone placement check for supported machine types.

The following preview Dataproc image versions are available:

• 2.1.0-RC2-debian11
• 2.1.0-RC2-rocky8
• 2.1.0-RC2-ubuntu20

The following component versions are available for use with the 2.1.0-RC2 images (the HBase and Druid components are not supported in 2.1 image versions):

• Apache Atlas 2.2.0

• Apache Flink 1.15.0

• Apache Hadoop 3.3.3

• Apache Hive 3.1.3

• Apache Hive WebHCat 3.1.3

• Apache Kafka 3.1.0

• Apache Pig 0.18.0-SNAPSHOT

• Apache Spark 3.3.0

• Apache Sqoop v1 1.5.0-SNAPSHOT

• Apache Sqoop v2 1.99.6

• Apache Tez 0.10.1

• Cloud Storage Connector hadoop3-2.2.8

• Conscrypt 2.5.2

• Docker 20.10

• Hue 4.10.0

• Java temurin-11-jdk

• JupyterLab Notebook 3.4

• Oozie 5.2.1

• Presto 376

• Python 3.10

• R 4.1

• Ranger 2.2.0

• Scala 2.12.14

• Solr 9.0.0

• Zeppelin Notebook 0.10.1

• Zookeeper 3.8.0

Dataproc Serverless for Spark runtime version 2.0.1 upgrades Apache Commons Text to 1.10.0, addressing CVE-2022-42889

Dataproc Serverless for Spark runtime version 2.0.1 upgrades the following components:

• Spark to 3.3.1
• SLF4J to 2.0.3

Dataproc Serverless for Spark now now uses runtime version 1.0.21 and 2.0.1.

All Dataproc Serverless for Spark runtime versions prior to 1.0.21 and 2.0.1 will be deprecated on November 2, 2022.

Dataproc Serverless for Spark runtime version 2.0 will become the default Dataproc Serverless for Spark runtime version on December 13, 2022.

Dataproc Serverless for Spark now supports spark.dataproc.diagnostics.enabled property that enables auto diagnostics on Batch failure. Note that enabling auto diagnostics will hold compute and storage quota after Batch is complete and until diagnostics is finished.

Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime 2.0.

New sub-minor versions of Dataproc images:

1.5.75-debian10, 1.5.75-rocky8, 1.5.75-ubuntu18

2.0.49-debian10, 2.0.49-rocky8, 2.0.49-ubuntu18

Dataproc Serverless for Spark runtime version 2.0.0 upgrades the following components:

• Conda to 22.9
• Jetty to 9.4.49.v20220914
• ORC to 1.8.0
• Protobuf to 3.21.7
• RoaringBitmap to 0.9.32

Upgraded Cloud Storage connector version to 2.2.8 in the latest 2.0 images.

Dataproc Serverless for Spark now uses runtime version 1.0.20 and 2.0.0.

Upgraded the Conscrypt library to 2.5.2 in the latest 1.5 and 2.0 images.

Disabled auto deletion of files under /tmp in the latest Rocky images. Previous Rocky images have files in the /tmp folder deleted every 10 days due to default OS system setting in /usr/lib/tmpfiles.d/tmp.conf.

Set yarn:spark.yarn.shuffle.stop_on_failure to true by default in the latest 1.5 and 2.0 images. This change causes YARN node manager startup to fail if the Spark external shuffle service startup fails. On VM boot, Dataproc will continuously restart the YARN node manager until it is able to start. This change reduces Spark executor errors, such as: org.apache.spark.SparkException: Unable to register with external shuffle server due to : Failed to connect to <worker host>:7337, particularly when starting a stopped cluster. See Spark external shuffle service documentation for details.

Changed Hive TokenStoreDelegationTokenSecretManager in the latest 1.5 and 2.0 images so that it updates the base class's current key ID after generating a new master key. This is important for users of DBTokenStore, which generates key IDs based on a monotonically increasing sequence from the database. Prior to this fix, there was a race condition during master key rollover that could cause it to attempt updating the prior master key using an incorrect ID value. This would fail and then quickly retry, sometimes multiple times, causing too many rows in the database.

Addressing CVE-2022-23305, CVE-2022-23302, CVE-2021-4104, CVE-2019-17571, migrated log4j 1.2 to reload4j for Hadoop, Spark in the latest 1.5 images and Hadoop, Spark, ZooKeeper, Oozie, Knox in the latest 2.0 images.

Enabled Spark authentication and encryption for Kerberos enabled clusters created with the latest 1.5 and 2.0 images.

Backported the patch for HADOOP-18316 in the latest 2.0 images, addressing CVE-2022-25168.

Set HDFS /user/<name> directory permission with owner=<name> and mode=700 for Kerberos enabled clusters created with the latest 1.5 and 2.0 images.

Backported the patch for HIVE-25468 in the latest 1.5 and 2.0 images, addressing CVE-2021-34538.

Backported the patch for HIVE-20607 in the latest 2.0 images.

Backported the patch for HIVE-19310 in the latest 1.5 images.

Fixed an issue in which Presto queries might fail when submitted to HA clusters in the latest 1.5 and 2.0 images.

Backported the patch for HIVE-26447 in the latest 2.0 images.

Backported the patch for SPARK-36383 in the latest 2.0 images.

Fixed a "gsutil not found" issue in the latest 1.5 and 2.0 Ubuntu images.

Fixed a bug where metrics created based on the yarn:yarn.resourcemanager.metrics.runtime.buckets property were not exported to Cloud Monitoring, even though listed in --metric-overrides during cluster creation.

Backported the patch for HIVE-20004 in the latest 2.0 images.

Dataproc is now available in the me-west1 region (Tel Aviv, Israel).

Preemptible SPOT VMs can be used as secondary workers in a Dataproc cluster. Unlike legacy preemptible VMs with a 24-hour maximum lifetime, Spot VMs have no maximum lifetime.

Dataproc Serverless for Spark now supports Artifact Registry with image streaming.

Dataproc Metastore: Fixed an endpoint resolution issue that caused 500 type errors for valid setups. The service was overly aggressive in describing networks and subnetworks attached to the service via the NetworkConfig field.

Dataproc Auto Zone Placement now takes ANY reservation into account by default.

Dataproc Serverless for Spark now uses runtime version 1.0.19 and 2.0.0-RC4, which also upgrades both runtimes to Cloud Storage Connector to 2.2.8.

Dataproc Serverless for Spark: You can now use the spark.dynamicAllocation.executorAllocationRatio property to configure how aggressively to scale up Serverless workloads. A value of 1.0 provides maximum scale up.

Dataproc Serverless for Spark: Increased initial and maximum Spark executor limits to 500 and 2,000, respectively.

Dataproc Serverless for Spark: Sets a maximum limit of 500 workers per scale up or scale down operation.

Dataproc Serverless for Spark: Reduced the latency between batch workload completion and when a batch is marked SUCCEEDED.

Dataproc on Compute Engine: Stop all master and worker VMs when starting a cluster fails due to stockout or insufficient quota.

Dataproc Serverless for Spark now uses runtime version 1.0.18 and 2.0.0-RC3.

Dataproc Serverless for Spark now uses runtime version 1.0.17 and 2.0.0-RC2.

Avoid using the following image versions when creating new clusters:

• 2.0.31-debian10, 2.0.31-ubuntu18, 2.0.31-rocky8
• 2.0.32-debian10, 2.0.32-ubuntu18, 2.0.32-rocky8
• 2.0.33-debian10, 2.0.33-ubuntu18, 2.0.33-rocky8
• 1.5.57-debian10, 1.5.57-ubuntu18, 1.5.57-rocky8
• 1.5.58-debian10, 1.5.58-ubuntu18, 1.5.58-rocky8
• 1.5.59-debian10, 1.5.59-ubuntu18, 1.5.59-rocky8

If your cluster uses one of these image versions, there is a small chance that the cluster might enter an ERROR_DUE_TO_UPDATE state while being updated, either manually or as a result of autoscaling. If that happens, contact support. You can avoid future occurrences by creating new clusters with a newer image version.

Fixed issue where gcloud dataproc batches list hangs when a large number of batches is present.

Announcing the Preview release of Dataproc custom constraints, which can be used to allow or deny specific operations on Dataproc clusters.

Announcing Dataproc Serverless for Spark preview runtime version 2.0.0-RC1, which includes the following components:

• Spark 3.3.0
• Cloud Storage Connector 2.2.7
• Java 17
• Conda 4.13
• Python 3.10
• R 4.1
• Scala 2.13

Dataproc Serverless for Spark now uses runtime version 1.0.16, which upgrades the following components to the following versions:

• Spark 3.2.2
• Avro 1.11.1
• Hadoop 3.3.4
• Jetty 9.4.48.v20220622
• ORC 1.7.5
• RoaringBitmap 0.9.31
• Scala 2.12.16

New sub-minor versions of Dataproc images:

1.5.73-debian10, 1.5.73-rocky8, 1.5.73-ubuntu18

2.0.47-debian10, 2.0.47-rocky8, 2.0.47-ubuntu18

Dataproc on Compute Engine images now have master VM memory protection enabled by default. Jobs may be terminated to prevent the master VM running out of memory.

Enabled Spark authentication and encryption for Kerberos clusters in 1.5 and 2.0 images.

Dataproc Serverless for Spark now uses runtime version 1.0.15, which upgrades the following Spark dependencies to the following versions:

• Jackson 2.13.3
• Jetty 9.4.46.v20220331
• ORC 1.7.4
• Parquet 1.12.3
• Protobuf 3.19.4
• RoaringBitmap 0.9.28

FallbackHiveAuthorizerFactory is now set by default on newly created 1.5 and 2.0 image clusters that have the any of the following features enabled:

• Kerberos security
• Personal Cluster Authentication
• Secure Multi-Tenancy

If you encounter a Cannot modify <PARAM> or similar runtime error when running a SET statement in a Hive query, this means the parameter is not in list of allowable runtime parameters. You can allow the parameter using hive.security.authorization.sqlstd.confwhitelist.append as a cluster property when you create a cluster.

Example:

--properties="hive:hive.security.authorization.sqlstd.confwhitelist.append=tez.application.tags,<ADDITIONAL-1>,<ADDITIONAL-2>"

Default MySQL instance root password changed to a random value in 1.5 and 2.0 images. New password is now stored in MySQL configuration file accessible only by the OS level root user.

New sub-minor versions of Dataproc images:

1.5.72-debian10, 1.5.72-rocky8, 1.5.72-ubuntu18

2.0.46-debian10, 2.0.46-rocky8, 2.0.46-ubuntu18

Upgraded Hadoop to version 2.10.2 version 2.10.2 in 1.5 images.

Upgraded Hadoop to version 3.2.3 in 2.0 images.

Backported the patches for HIVE-19047 and HIVE-19048 in 1.5 images.

Backported the patch for HIVE-19048 in 2.0 images.

Backported the patch for KNOX-1997 in 2.0 images.

Dataproc support for the following images has been extended to the following dates:

• Dataproc image version 1.5: March 31, 2023
• Dataproc image version 2.0: June 30, 2023

New sub-minor versions of Dataproc images:

1.5.71-debian10, 1.5.71-rocky8, 1.5.71-ubuntu18

2.0.45-debian10, 2.0.45-rocky8, 2.0.45-ubuntu18

For 1.5 images and the 2.0.45-ubuntu18 image, backported the upstream fix for KNOX-1997.

New sub-minor versions of Dataproc images:

1.5.70-debian10, 1.5.70-rocky8, 1.5.70-ubuntu18

2.0.44-debian10, 2.0.44-rocky8, 2.0.44-ubuntu18

Dataproc Metastore: For 1.5 images, added a spark.hadoop.hive.eager.fetch.functions.enabled Spark Hive client property to control whether the client fetches all functions from Hive Metastore during initialization. The default setting is true, which preserves the existing behavior of fetching all functions. If set to false, the client will not fetch all functions during initialization, which can help reduce high latency during initialization, particularly when there are many functions and the Metastore is not located in the client's region.

For 1.5 and 2.0 images, backported YARN-9608 to fix the issue in graceful decommissioning.

New sub-minor versions of Dataproc images:

1.5.69-debian10, 1.5.69-rocky8, 1.5.69-ubuntu18

2.0.43-debian10, 2.0.43-rocky8, 2.0.43-ubuntu18

Backported the patch for HBASE-23287 to HBase 1.5.0 in 1.5 image

Announcing the General Availability (GA) release of Dataproc Custom OSS Metrics GA, which collects then integrates Dataproc cluster OSS component metrics into Cloud Monitoring.

Announcing the General Availability (GA) release of Ranger Cloud Storage plugin. This plugin activates an authorization service on each Dataproc cluster VM, which evaluates requests from the Cloud Storage connector against Ranger policies and, if the request is allowed, returns an access token for the cluster VM service account

Dataproc is now available in the us-south1 region (Dallas, Texas).

Announcing the General Availability (GA) release of Dataproc Persistent History Server, which provides web interfaces to view job history for jobs run on active or deleted Dataproc clusters.

Dataproc Serverless for Spark now uses runtime version 1.0.13.

New sub-minor versions of Dataproc images:

1.5.68-debian10, 1.5.68-rocky8, 1.5.68-ubuntu18

2.0.42-debian10, 2.0.42-rocky8, 2.0.42-ubuntu18

Dataproc Serverless for Spark runtime versions 1.0.2, 1.0.3 and 1.0.4 are unavailable for new batch submissions.

Upgrade Cloud Storage connector version 2.1.8 for 1.5 images only.

Dataproc on GKE Spark 3.1 images upgraded to Spark version 3.1.3.

Fixed a bug where HDFS directories initialization could fail when user names in a project contain special characters.

Fix a Dataproc on GKE bug that caused upload of driver logs to Cloud Storage to fail.

Dataproc is now available in the us-east5 region (Columbus, Ohio).

Dataproc is now available in the europe-west9 region (Paris, France).

Dataproc is now available in the europe-southwest1 region (Madrid, Spain).

New sub-minor versions of Dataproc images:

1.5.67-debian10, 1.5.67-ubuntu18, 1.5.67-rocky8

2.0.41-debian10, 2.0.41-ubuntu18, 2.0.41-rocky8

Dataproc on GKE error messages now provide additional information.

Backported fixes for HIVE-22098, HIVE-23809, HIVE-20462, HIVE-21675 to Hive 3.1 in Dataproc 2.0 images.

Fixed a bug where older-image (for example, 1.3.95) cluster create operations failed with the error message : "does not support specifying local SSD interface other than 'SCSI'".

Fix a bug where properties related to Kerberos cross realm trust were not properly set.

New sub-minor versions of Dataproc images:

1.5.66-debian10, 1.5.66-ubuntu18, 1.5.66-rocky8

2.0.40-debian10, 2.0.40-ubuntu18, 2.0.40-rocky8

Upgraded Spark to 3.1.3 in Dataproc image version 2.0.

Fixed a bug where job was not being marked as terminated after master node reboot.

Backported the fix for HIVE-20514 to Hive 2.3 in Dataproc image version 1.5.

Fixed a bug with HDFS directories initialization when core:fs.defaultFS is set to an external HDFS.

Fixed a bug where Flink was not able to run on HA clusters.

Dataproc Serverless for Spark now uses runtime version 1.0.12.

New sub-minor versions of Dataproc images:

1.5.65-debian10, 1.5.65-ubuntu18, 1.5.65-rocky8

2.0.39-debian10, 2.0.39-ubuntu18, 2.0.39-rocky8

Fixed an issue where chronyd systemd service failed to start due to a race condition between systemd-timesyncd and chronyd.

Dataproc Serverless for Spark runtime version 1.0.1 is unavailable for new batch submissions.

Dataproc Serverless for Spark now uses runtime version 1.0.11.

New sub-minor versions of Dataproc images:

1.5.64-debian10, 1.5.64-ubuntu18, 1.5.64-rocky8

2.0.38-debian10, 2.0.38-ubuntu18, 2.0.38-rocky8

If you request to cancel a job in one of the following states, Dataproc will return the job, but not initiate cancellation, since it is already in progress: CANCEL_PENDING, CANCEL_STARTED, or CANCELLED.

Added Dataproc Serverless support for updating the Cloud Storage connector using the dataproc.gcsConnector.version and dataproc.gcsConnector.uri properties.

When submitting a Dataproc job or workflow that selects a cluster that matches the specified labels, Dataproc will avoid choosing clusters that are in a state that disallows running jobs. Specifically, Dataproc will only choose among clusters in one of the following states: RUNNING, UPDATING, CREATING, or ERROR_DUE_TO_UPDATE.

Hive: Upgrade to Apache ORC 1.5.13 in image version 2.0. Notable in this release are 2 bug fixes: ORC-598 and ORC-672, related to handling ORC files with arrays larger than 1024 elements.

Dataproc correctly defaults NodePool locations when the GKE cluster is in us-east1 and europe-west1.

Dataproc Serverless for Spark runtime version 1.0.0 is unavailable for new batch submissions.

Cloud Storage connector version upgraded to 2.2.6 in image version 2.0.

New sub-minor versions of Dataproc images:

1.5.63-debian10, 1.5.63-ubuntu18, 1.5.63-rocky8

2.0.37-debian10, 2.0.37-ubuntu18, 2.0.37-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.10.

Hive: Bundle threeten classes in hive-exec.jar in image version 2.0. ORC now requires date handling classes in the org.threeten package, which are not present in hive-exec.jar at query time.

HIVE-22589 fixed this bug upstream, but it was part of a large new feature. Instead, this change applies a small targeted fix to address the bug.

Dataproc is now available in the europe-west8 region (Milan, Italy).

Announcing the General Availability (GA) release of Dataproc on GKE, which allows you to execute Big Data applications using the Dataproc jobs API on GKE clusters.

The dataproc:dataproc.performance.metrics.listener.enabled cluster property, which is enabled by default, listens on port 8791 on all master nodes to extract performance-related telemetry Spark metrics. The metrics are published to the Dataproc service for it to use to set better defaults and improve the service. To opt-out of this feature, set dataproc:dataproc.performance.metrics.listener.enabled=false when creating a Dataproc cluster.

New sub-minor versions of Dataproc images:

1.5.62-debian10, 1.5.62-ubuntu18, and 1.5.62-rocky8

2.0.36-debian10, 2.0.36-ubuntu18, and 2.0.36-rocky8

Fixed and issue in which the Dataproc autoscaler would sometimes try to scale down a cluster by more than one thousand secondary worker nodes at one time. Now, the autoscaler will scale down at most one thousand nodes at one time. In cases where the autoscaler previously would have scaled down more than one thousand nodes at one time, it will scale down the nodes by at most one thousand nodes, and a log will be written to the autoscaler log noting this occurrence.

Changed the owner of /usr/lib/knox/conf/gateway-site.xml from root:root to knox:knox.

Dataproc Serverless for Spark now uses runtime version 1.0.9.

Fixed bugs that could cause Dataproc to delay marking a job cancelled.

New sub-minor versions of Dataproc images:

1.5.61-debian10, 1.5.61-ubuntu18, and 1.5.61-rocky8

2.0.35-debian10, 2.0.35-ubuntu18, and 2.0.35-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.8.

Changed the owner of /var/lib/zookeeper/myid from root to zookeeper.

Dataproc Serverless for Spark now uses runtime version 1.0.7.

New sub-minor versions of Dataproc images:

1.5.60-debian10, 1.5.60-ubuntu18, and 1.5.60-rocky8

2.0.34-debian10, 2.0.34-ubuntu18, and 2.0.34-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.6.

New sub-minor versions of Dataproc images:

1.5.59-debian10, 1.5.59-ubuntu18, and 1.5.59-rocky8

2.0.33-debian10, 2.0.33-ubuntu18, and 2.0.33-rocky8

New sub-minor versions of Dataproc images:

1.5.58-debian10, 1.5.58-ubuntu18, and 1.5.58-rocky8

2.0.32-debian10, 2.0.32-ubuntu18, and 2.0.32-rocky8

Dataproc Serverless for Spark now uses runtime version 1.0.5.

Fixed bug where clusters created with Kerberos failed with SSL Certificate string is too long error.

Fixed bug where clusters created via Dataproc Hub failed with Unit file jupyter.service does not exist error.

Added support for Enhanced Flexibility Mode (EFM) with primary worker shuffle mode on Spark for image version 2.0.

General Availability (GA) release of new Rocky Linux based images: 1.5.57-rocky8 and 2.0.31-rocky8. These images are replacing CentOS images which are EOL.

Upgraded Cloud Storage connector version to 2.1.7 in image version 1.5.

CentOS images are EOL. 1.5.56-centos8 and 2.0.30-centos8 are the final CentOS based images. CentOS images are no longer supported and will not receive new releases.

Dataproc Serverless for Spark now uses runtime version 1.0.4, which updates GCS connector to 2.2.5 version.

New sub-minor versions of Dataproc images:

1.5.57-debian10, 1.5.57-ubuntu18, and 1.5.57-rocky8

2.0.31-debian10, 2.0.31-ubuntu18, and 2.0.31-rocky8

Upgraded Cloud Storage connector version to 2.2.5 in image version 2.0.

A script that checks if a project or organization is using an unsupported Dataproc image is available for downloading (see Unsupported Dataproc versions).

Dataproc images prior to 1.3.95, 1.4.77, 1.5.53, and 2.0.27 are deprecated and cluster creations based on these images will fail starting 2/28/2022.

Dataproc Serverless for Spark now uses runtime version 1.0.3.

Added cluster_type field to job and operation metrics in Cloud Monitoring.

Enabled the Resource Manager UI and HA capable UIs in HA cluster mode.

1.4.80-debian10 and 1.4.80-ubuntu18 are the last releases for the 1.4 images. Dataproc 1.4 images will no longer be supported and will not receive new releases.

Configured Zeppelin Spark interpreter to run in YARN client mode by default for image version 2.0.

New sub-minor versions of Dataproc images:

1.4.80-debian10 and 1.4.80-ubuntu18

1.5.56-debian10, 1.5.56-ubuntu18, and 1.5.56-centos8

2.0.30-debian10, 2.0.30-ubuntu18, and 2.0.30-centos8

Dataproc Serverless for Spark now uses runtime version 1.0.2, which updates Spark to 3.2.1 version.

Dataproc Serverless for Spark now uses runtime version 1.0.1, which includes improved error messaging for network connectivity issues.

Announcing the General Availability (GA) release of Dataproc Serverless for Spark, which allows you to run your Spark jobs on Dataproc without having to spin up and manage your own cluster.

Added support for Dataproc Metastore's beta NetworkConfig field. Beta services using this field can now be used in conjunction with v1 Dataproc clusters.

Dataproc extracts the warehouse directory from the Dataproc Metastore service for the cluster-local warehouse directory.

New sub-minor versions of Dataproc images:

1.4.79-debian10 and 1.4.79-ubuntu18

1.5.55-debian10, 1.5.55-ubuntu18, and 1.5.55-centos8

2.0.29-debian10, 2.0.29-ubuntu18, and 2.0.29-centos8

The Cloud Storage connector jar is installed on the Solr server (even if dataproc:solr.gcs.path property is not set). Applies to image versions 1.4, 1.5, and 2.0.

Dataproc images 1.4.79, 1.5.55, and 2.0.29, listed above, were updated with log4j version 2.17.1. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.

Fixed a bug where cluster restart disabled Solr and Ranger services even if the components are selected. Applies to image versions 1.4, 1.5, and 2.0.

Migrated to Eclipse Temurin JDK in image versions 1.4, 1.5, and 2.0.

RANGER-3324: Make optimized db schema script idempotent for MySQL DB. Applies to 2.0 images.

YARN-8865: RMStateStore contains large number of expired RMDelegationToken. Applies to 1.5 images.

New sub-minor versions of Dataproc images:

1.4.78-debian10, and 1.4.78-ubuntu18

1.5.54-centos8, 1.5.54-debian10, and 1.5.54-ubuntu18

2.0.28-centos8, 2.0.28-debian10, and 2.0.28-ubuntu18

Dataproc images 1.4.78, 1.5.54, and 2.0.28, listed above, were updated with log4j version 2.17.0. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.

Fixed the problem that jars added with the --jars flag in gcloud dataproc jobs submit spark-sql are missing at runtime.

Upgraded Cloud Storage connector version to 2.2.4 in image version 2.0.

Dataproc has released 1.3.95-debian10/-ubuntu18 images with a one-time patch that addresses the Apache Log4j 2 CVE-2021-44228 and CVE-2021-45046 vulnerabilities, but note that that all 1.3 images remain unsupported, and Dataproc will not provide upgrades to 1.3 images .

Removed the Geode interpreter from Zeppelin notebook, which is affected by https://nvd.nist.gov/vuln/detail/CVE-2021-45046.

Dataproc has released the following sub-minor image versions to address an Apache Log4j 2 vulnerability (also see Create a cluster and Recreate and update a cluster for more information). Note: These images supersede the 1.5 and 2.0 images listed in the December 16, 2021 release note:

1.5.53-centos8, 1.5.53-debian10, 1.5.53-ubuntu18,

2.0.27-centos8, 2.0.27-debian10, 2.0.27-ubuntu18

Upgraded log4j version to 2.16.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-44228.

Dataproc has released the following [sub-minor image versions](https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions:

1.4.77-debian10, 1.4.77-ubuntu18,

1.5.52-centos8, 1.5.52-debian10, 1.5.52-ubuntu18,

2.0.26-centos8, 2.0.26-debian10, 2.0.26-ubuntu18

Dataproc has added new images, listed in this release note, to address an Apache Log4j 2 vulnerability.

Note: these images have been superseded. by the 12/16/21 images (see the December 16, 2021 release note). Also see Create a cluster and Recreate and update a cluster for more information.

Upgraded log4j version to 2.15.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-45046.

New sub-minor versions of Dataproc images:

1.4.76-debian10, 1.4.76-ubuntu18,

1.5.51-centos8, 1.5.51-debian10, 1.5.51-ubuntu18,

2.0.25-centos8, 2.0.25-debian10, 2.0.25-ubuntu18

HIVE-21040: msck does unnecessary file listing at last level of directory tree. Applies to 1.5 and 2.0 images.

Fixed a bug where driver log links on PHS Web UI stop working once the job cluster is deleted. Applies to 1.4 and 1.5 images.

Fixed executor log links on Spark History Server Web UI for running and completed applications. Applies to 1.4 and 1.5 images.

SPARK-35817: Queries against wide Avro tables can be slow. Applies to 2.0 images.

SPARK-7768: Make user-defined type (UDT) API public. Applies to 2.0 images.

YARN-8990: Fixed a Fairscheduler race condition. Applies to 2.0 images.

Dataproc is now available in the southamerica-west1 region (Santiago, Chile).

Added the following new Apache Spark properties to control Cloud Storage flush behavior for event logs for 1.4 and later images:

• spark.history.fs.gs.outputstream.type (default: BASIC)
• spark.history.fs.gs.outputstream.sync.min.interval.ms (default: 5000ms).

Note: The default configuration of these properties enables the display of running jobs in the Spark History Server UI for clusters using Cloud Storage to store spark event logs.

Added support in 1.5 and 2.0 images to filter Spark Applications on the Spark History Server Web UI based on Cloud Storage path. Filtering is accomplished using the eventLogDirFilter parameter, which accepts any Cloud Storage path substring and will return applications that match the Cloud Storage path.

New sub-minor versions of Dataproc images:

1.4.75-debian10, 1.4.75-ubuntu18,

1.5.50-centos8, 1.5.50-debian10, 1.5.50-ubuntu18,

2.0.24-centos8, 2.0.24-debian10, 2.0.24-ubuntu18

The previous Dataproc on GKE beta documentation has been replaced with a Dataproc on GKE private preview sign up form. Existing beta customers can continue using the beta release, but note that the beta release is planned to be deprecated and removed.