Managed Service for Apache Spark release notes

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.119-debian11, 2.1.119-rocky8, 2.1.119-ubuntu20, 2.1.119-ubuntu20-arm
• 2.2.87-debian12, 2.2.87-rocky9, 2.2.87-ubuntu22, 2.2.87-ubuntu22-arm
• 2.3.36-debian12, 2.3.36-ml-ubuntu22, 2.3.36-rocky9, 2.3.36-ubuntu22, 2.3.36-ubuntu22-arm
• 3.0.2-debian13, 3.0.2-ml-ubuntu24, 3.0.2-rocky9, 3.0.2-ubuntu24

Key updates in these image versions include:

• Feature updates:
  • Apache Hudi: Added support for the Apache Hudi optional component in 3.0 images. Version 1.2.0 is available in 3.0 images.
  • Parquet footer caching: Enabled Parquet footer caching by default for Lightning Engine (Velox). If executor out-of-memory (OOM) or task failure spikes on tiny files are observed, these can be mitigated by setting spark.gluten.sql.columnar.backend.velox.cacheParquetFooters=false.
  • Apache Iceberg 1.10: Added support for Apache Iceberg 1.10 in 2.2 images. Users can opt-in by setting the cluster property, dataproc:dataproc.iceberg.version=1.10.
  • Lakehouse catalog: Lakehouse catalog auto-loading is supported for image versions 2.2 and later.
• Library updates:
  • Cloud Storage connector: Upgraded the Cloud Storage connector to 4.0.4 in 3.0 images.
  • OpenLineage: Upgraded OpenLineage to 1.49 in 3.0 images to support lineage for tables created using the Lakehouse Runtime catalog.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

• Preconfigured Conda channels removed: Preconfigured conda channels (such as conda-forge) have been removed from configurations. All image version aliases now point to the latest image without conda channel configuration. Support for earlier images with conda channels configuration to be announced in upcoming release notes. Recommendation: Migrate to the latest image versions as soon as possible.
  • Affected image versions: 1.3.96+, 1.4.81+, 1.5.92+, 2.0.161+, 2.1.119+, 2.2.87+, and 2.3.36+.
  • Impact: Additional conda package installation using the dataproc:conda.packages cluster property or direct conda install <package> command will fail.
  • Workaround:
    • Use the <channel>::<package>==<version> specification for the dataproc:conda.packages property (for example: dataproc:conda.packages=conda-forge::pip==24.0).
    • Specify the channel name on the command line when running conda install (for example: conda install <packages> -c conda-forge).
• google-guest-agent upgraded: Upgraded google-guest-agent on Debian and Rocky Linux images (remediating CVE-2026-33186).
• SSH metadata restriction: Stricter SSH metadata restrictions are enforced by default; metadata SSH keys mapped directly to root are ignored. Connect using a standard non-root user (such as dataproc) with sudo, or use Google Cloud OS Login.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings with the Lakehouse Runtime catalog.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on August 19, 2026 have been rolled back:

• 2.2.86-debian12, 2.2.86-rocky9, 2.2.86-ubuntu22, 2.2.86-ubuntu22-arm

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.118-debian11, 2.1.118-rocky8, 2.1.118-ubuntu20, 2.1.118-ubuntu20-arm
• 2.2.86-debian12, 2.2.86-rocky9, 2.2.86-ubuntu22, 2.2.86-ubuntu22-arm
• 2.3.35-debian12, 2.3.35-ml-ubuntu22, 2.3.35-rocky9, 2.3.35-ubuntu22, 2.3.35-ubuntu22-arm
• 3.0.1-debian13, 3.0.1-ml-ubuntu24, 3.0.1-rocky9, 3.0.1-ubuntu24

Key updates in these image versions include:

• Iceberg updates: In the 2.3 image version, 2.3 clusters with Lightning Engine now use Iceberg version 1.10 by default.
• OpenLineage updates: In the 2.2 and 2.3 image versions:
  • Upgraded OpenLineage to version 1.49 to support lineage for tables created using the Lakehouse Runtime catalog.

Rollback Notice: The 2.2.86 image versions were rolled back on August 31, 2026.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.86
• 2.2.86
• 2.3.39

Key updates in these runtime versions include:

• OpenLineage updates: In the 2.3 runtime:
  • Upgraded OpenLineage to version 1.49 to support lineage for tables created using the Lakehouse Runtime catalog.
  • Fixed a segmentation fault when OpenLineage parses complex SQL query strings.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

A critical bug related to Conda channels has been fixed in-place in image versions 1.4.81, 1.5.92, 2.1.117, and 2.2.85. These image versions were released without pre-configured Conda channels.

Required customer actions: To comply with Google requirements, recreate the following resources if they were created using these image versions on or before August 10, 2026:

• Custom images
• Clusters

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.85
• 2.2.85
• 2.3.38

Notes:

• Apache Spark upgraded to 3.5.3 in 2.2 runtime.

• Apache Gluten upgraded to 1.6 in 2.3 runtime.

Managed Service for Apache Spark latest image and runtime versions:

• Configured spark.scheduler.listenerbus.exitTimeout to 30s.

New Managed Service for Apache Spark on Google Kubernetes Engine (formerly Dataproc on Google Kubernetes Engine) subminor image version:

• 3.5-dataproc-28

Key updates in this image version include:

• Conda channels: The new 3.5-dataproc-28 subminor image version doesn't have preconfigured Conda channels, and is mapped to default aliases (such as 3.5 and latest).
  • Impact: When creating clusters with 3.5-dataproc-28 or using default aliases (3.5, latest), packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels, pin your clusters to the previous image versions before August 25, 2026.
  • Default change schedule: All workloads must transition to image versions without preconfigured Conda channels after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Starting with Managed Service for Apache Spark image version 3.0, clusters that are created without a specified machine type for a node are created with a Flex VM configuration for the node.

Announcing the General Availability (GA) release of Managed Service for Apache Spark cluster image version 3.0, as follows:

• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, and 3.0.0-ubuntu24.

  • The 3.0.0-ml-ubuntu24 image extends the 3.0 base image with ML-specific libraries.

Image version 3.0 is a lightweight image that contains the following pre-installed core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs):

• Apache Hadoop 3.5.0.
• Apache Hive 4.2.0
• Apache Spark 4.1.2
• Apache Tez 0.10.5
• Cloud Storage Connector 3.1.13
• Conscrypt 2.6
• Java 21
• Python 3.12
• R 4.5
• Scala 2.13.17
• Spark-BigQuery Connector 0.44.1-preview

You can add listed optional components when you create a 3.0 image version cluster.

Recommendation: Use the 3.0 (and later) image versions to meet security compliance requirements.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.34-debian12, 2.3.34-ml-ubuntu22, 2.3.34-rocky9, 2.3.34-ubuntu22, 2.3.34-ubuntu22-arm
• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, 3.0.0-ubuntu24

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

• The 2.1, 2.2 and 2.3 cluster image versions now support Confidential Compute for the g4-standard-48 GPU machine type.

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):

• The 3.0 runtime now uses fewer executors, as follows:

  • 0 min executors for spark.dynamicAllocation.minExecutors property
  • 1 min executor for spark.executor.instances and spark.dynamicAllocation.initialExecutors properties

• All runtimes now configure spark.scheduler.listenerbus.exitTimeout to 30 seconds.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.117-debian11, 2.1.117-rocky8, 2.1.117-ubuntu20, 2.1.117-ubuntu20-arm
• 2.2.85-debian12, 2.2.85-rocky9, 2.2.85-ubuntu22, 2.2.85-ubuntu22-arm
• 2.3.33-debian12, 2.3.33-ml-ubuntu22, 2.3.33-rocky9, 2.3.33-ubuntu22, 2.3.33-ubuntu22-arm

Key updates in these image versions include:

• Conda channels: The new 2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
  • Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions (for example, 2.1.116-debian11 or 2.2.84-debian12).
  • Default change schedule: After August 25, 2026, the default aliases for 2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32, don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026. The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels, but note the following Default change schedule:, which will impact all supported image versions.

• Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
• Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions.
• Default change schedule: The subminor versions 1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026. Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases. All workloads must use new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 1.3.96-debian10, 1.3.96-ubuntu18
• 1.4.81-debian10, 1.4.81-ubuntu18
• 1.5.92-debian10, 1.5.92-rocky8, 1.5.92-ubuntu18
• 2.0.161-debian10, 2.0.161-rocky8, 2.0.161-ubuntu18
• 2.1.115-debian11, 2.1.115-rocky8, 2.1.115-ubuntu20, 2.1.115-ubuntu20-arm
• 2.1.116-debian11, 2.1.116-rocky8, 2.1.116-ubuntu20, 2.1.116-ubuntu20-arm
• 2.2.83-debian12, 2.2.83-rocky9, 2.2.83-ubuntu22, 2.2.83-ubuntu22-arm
• 2.2.84-debian12, 2.2.84-rocky9, 2.2.84-ubuntu22, 2.2.84-ubuntu22-arm
• 2.3.31-debian12, 2.3.31-ml-ubuntu22, 2.3.31-rocky9, 2.3.31-ubuntu22, 2.3.31-ubuntu22-arm
• 2.3.32-debian12, 2.3.32-ml-ubuntu22, 2.3.32-rocky9, 2.3.32-ubuntu22, 2.3.32-ubuntu22-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:

• Iceberg support: Added support for Iceberg 1.10 in Dataproc 2.3 images. This change only applies to 2.3 clusters created with the ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.
• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default in new GCE image versions.
• Scheduler exitTimeout: Configured spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.82
• 2.2.82
• 2.3.35

Key updates in these runtime versions include:

• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.