View a markdown version of this page

Scheduling section - AWS ParallelCluster
Scheduling propertiesAwsBatchQueuesSlurmQueuesSlurmSettings

Scheduling section

(Required) Defines the job scheduler that's used in the cluster and the compute instances that the job scheduler manages. You can either use the Slurm or AWS Batch scheduler. Each supports a different set of settings and properties. Starting with AWS ParallelCluster version 3.16.0, AWS Batch as a scheduler is no longer supported.

Scheduling:
  Scheduler: slurm
  ScalingStrategy: string    
  SlurmSettings:
    MungeKeySecretArn: string        
    ScaledownIdletime: integer    
    QueueUpdateStrategy: string
    EnableMemoryBasedScheduling: boolean
    CustomSlurmSettings: [dict]
    CustomSlurmSettingsIncludeFile: string
    Database:
      Uri: string
      UserName: string
      PasswordSecretArn: string
      DatabaseName: string    
    ExternalSlurmdbd: boolean
      Host: string
      Port: integer  
    Dns:
      DisableManagedDns: boolean
      HostedZoneId: string
      UseEc2Hostnames: boolean  
  SlurmQueues:
    - Name: string  
      ComputeSettings:
        LocalStorage:
          RootVolume:
            Size: integer
            Encrypted: boolean
            VolumeType: string
            Iops: integer
            Throughput: integer
          EphemeralVolume:
            MountDir: string
      CapacityReservationTarget:
        CapacityReservationId: string
        CapacityReservationResourceGroupArn: string
      CapacityType: string
      AllocationStrategy: string
      JobExclusiveAllocation: boolean
      CustomSlurmSettings: dict
      Tags:
        - Key: string
          Value: string
      HealthChecks:
        Gpu:
          Enabled: boolean
      Networking:
        SubnetIds:
          - string
        AssignPublicIp: boolean
        SecurityGroups:
          - string
        AdditionalSecurityGroups:
          - string
        PlacementGroup:
          Enabled: boolean
          Id: string
          Name: string
        Proxy:
          HttpProxyAddress: string
      ComputeResources:
        - Name: string
          InstanceType: string
          Instances:
            - InstanceType: string
          MinCount: integer
          MaxCount: integer
          DynamicNodePriority: integer
          StaticNodePriority: integer
          SpotPrice: float
          DisableSimultaneousMultithreading: boolean
          SchedulableMemory: integer
          HealthChecks:
            Gpu:
              Enabled: boolean
          Efa:
            Enabled: boolean
            GdrSupport: boolean          
          CapacityReservationTarget:
            CapacityReservationId: string
            CapacityReservationResourceGroupArn: string
          Networking:   
            PlacementGroup:
              Enabled: boolean
              Name: string
          CustomSlurmSettings: dict
          Tags:
            - Key: string
              Value: string
          LaunchTemplateOverrides:
            LaunchTemplateId: string
            Version: string
      CustomActions:
        OnNodeStart:
          Sequence:
            - Script: string
              Args:
                - string
          Script: string
          Args:
            - string
        OnNodeConfigured:
          Sequence:
            - Script: string
              Args:
                - string
          Script: string
          Args:
            - string
      Iam:
        InstanceProfile: string
        InstanceRole: string
        S3Access:
          - BucketName: string
            EnableWriteAccess: boolean
            KeyName: string
        AdditionalIamPolicies:
          - Policy: string
      Image:
        CustomAmi: string
Scheduling:
  Scheduler: awsbatch
  AwsBatchQueues:
    - Name: string
      CapacityType: string
      Networking:
        SubnetIds:
          - string
        AssignPublicIp: boolean
        SecurityGroups:
          - string
        AdditionalSecurityGroups:
          - string
      ComputeResources:  # this maps to a Batch compute environment (initially we support only 1)
        - Name: string
          InstanceTypes:
            - string
          MinvCpus: integer
          DesiredvCpus: integer
          MaxvCpus: integer
          SpotBidPercentage: float

Scheduling properties

Scheduler (Required, String)

Specifies the type of scheduler that's used. Supported values are slurm and awsbatch. Starting with AWS ParallelCluster version 3.16.0, AWS Batch as a scheduler is no longer supported.

Update policy: If this setting is changed, the update is not allowed.

Note

awsbatch only supports the alinux2 operating system and x86_64 platform.

ScalingStrategy (Optional, String)

Allows you to choose how dynamic Slurm nodes scale up. Supported values are all-or-nothing, greedy-all-or-nothing and best-effort The default value is all-or-nothing.

Update policy: This setting can be changed during an update.

Note

The scaling strategy applies only to nodes to be resumed by Slurm, not to nodes that are eventually already running.

  • all-or-nothingThis strategy strictly follows an all-or-nothing-approach, aimed at avoiding idle instances at the end of the scaling process. It operates on an all-or-nothing basis, meaning it either scales up completely or not at all. Be aware that there may be additional costs due to temporarily launched instances, when jobs require over 500 nodes or span multiple compute resources. This strategy has the lowest throughput among the three possible Scaling Strategies. The scaling time depends on the number of jobs submitted per Slurm resume program execution. Also, you can't scale far beyond the default RunInstances resource account limit per execution, which is 1000 instances by defaults. More details can be found at the Amazon EC2 API throttling documentation

  • greedy-all-or-nothing Similar to the all-or-nothing strategy, it aims to avoid idle instances post-scaling. This strategy allows for temporary over-scaling during the scaling process in order to achieve higher throughput than the all-or-nothing approach but also comes with the same scaling limit of 1000 instances as per the RunInstances resource account limit.

  • best-effort This strategy prioritizes high throughput, even if it means that some instances might be idle at the end of the scaling process. It attempts to allocate as many nodes as requested by the jobs, but there's a possibility of not fulfilling the entire request. Unlike the other strategies, the best-effort approach can accumulate more instances than the standard RunInstances limit, at the cost of having idle resources along the multiple scaling process executions.

Each strategy is designed to cater to different scaling needs, allowing you to select one that meets your specific requirements and constraints.

AwsBatchQueues

(Optional) The AWS Batch queue settings. Only one queue is supported. If Scheduler is set to awsbatch, this section is required. For more information about the awsbatch scheduler, see networking setup and Using AWS Batch (awsbatch) scheduler with AWS ParallelCluster.

AwsBatchQueues:
  - Name: string
    CapacityType: string
    Networking:
      SubnetIds:
        - string
      AssignPublicIp: boolean
      SecurityGroups:
        - string
      AdditionalSecurityGroups:
        - string
    ComputeResources:  # this maps to a Batch compute environment (initially we support only 1)
      - Name: string
        InstanceTypes:
          - string
        MinvCpus: integer
        DesiredvCpus: integer
        MaxvCpus: integer
        SpotBidPercentage: float

Update policy: This setting can be changed during an update.

AwsBatchQueues properties

Name (Required, String)

The name of the AWS Batch queue.

Update policy: If this setting is changed, the update is not allowed.

CapacityType (Optional, String)

The type of the compute resources that the AWS Batch queue uses. Supported values are ONDEMAND , SPOT or CAPACITY_BLOCK. The default value is ONDEMAND.

Note

If you set CapacityType to SPOT, your account must contain an AWSServiceRoleForEC2Spot service-linked role. You can create this role using the following AWS CLI command.

$ aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

For more information, see Service-linked role for Spot Instance requests in the Amazon Amazon EC2 User Guide for Linux Instances.

Update policy: The compute fleet must be stopped for this setting to be changed for an update.

Networking

(Required) Defines the networking configuration for the AWS Batch queue.

Networking:
  SubnetIds:
    - string
  AssignPublicIp: boolean
  SecurityGroups:
    - string
  AdditionalSecurityGroups:
    - string
Networking properties
SubnetIds (Required, [String])

Specifies the ID of an existing subnet to provision the AWS Batch queue in. Currently only one subnet is supported.

Update policy: The compute fleet must be stopped for this setting to be changed for an update.

AssignPublicIp (Optional, String)

Creates or assigns a public IP address to the nodes in the AWS Batch queue. Supported values are true and false. The default depends on the subnet that you specified.

Update policy: If this setting is changed, the update is not allowed.

SecurityGroups (Optional, [String])

List of security groups that the AWS Batch queue uses. If you don't specify security groups, AWS ParallelCluster creates new security groups.

Update policy: This setting can be changed during an update.

AdditionalSecurityGroups (Optional, [String])

List of security groups that the AWS Batch queue uses.

Update policy: This setting can be changed during an update.

ComputeResources

(Required) Defines the ComputeResources configuration for the AWS Batch queue.

ComputeResources:  # this maps to a Batch compute environment (initially we support only 1)
  - Name: string
    InstanceTypes:
      - string
    MinvCpus: integer
    DesiredvCpus: integer
    MaxvCpus: integer
    SpotBidPercentage: float
ComputeResources properties
Name (Required, String)

The name of the AWS Batch queue compute environment.

Update policy: The compute fleet must be stopped for this setting to be changed for an update.

InstanceTypes (Required, [String])

The AWS Batch compute environment array of instance types. All of the instance types must use the x86_64 architecture.

Update policy: The compute fleet must be stopped for this setting to be changed for an update.

MinvCpus (Optional, Integer)

The minimum number of VCPUs that an AWS Batch compute environment can use.

Update policy: This setting can be changed during an update.

DesiredVcpus (Optional, Integer)

The desired number of VCPUs in the AWS Batch compute environment. AWS Batch adjusts this value between MinvCpus and MaxvCpus based on the demand in the job queue.

Update policy: This setting is not analyzed during an update.

MaxvCpus (Optional, Integer)

The maximum number of VCPUs for the AWS Batch compute environment. You can't set this to a value that's lower than DesiredVcpus.

Update policy: This setting can't be decreased during an update.

SpotBidPercentage (Optional, Float)

The maximum percentage of the On-Demand price for the instance type that an Amazon EC2 Spot Instance price can reach before instances are launched. The default value is 100 (100%). The supported range is 1-100.

Update policy: This setting can be changed during an update.

SlurmQueues

(Optional) Settings for the Slurm queue. If Scheduler is set to slurm, this section is required.

SlurmQueues:
  - Name: string
    ComputeSettings:
      LocalStorage:
        RootVolume:
          Size: integer
          Encrypted: boolean
          VolumeType: string
          Iops: integer
          Throughput: integer
        EphemeralVolume:
          MountDir: string
    CapacityReservationTarget:
      CapacityReservationId: string
      CapacityReservationResourceGroupArn: string
    CapacityType: string
    AllocationStrategy: string
    JobExclusiveAllocation: boolean
    CustomSlurmSettings: dict
    Tags:
      - Key: string
        Value: string
    HealthChecks:
      Gpu:
        Enabled: boolean
    Networking:
      SubnetIds:
        - string
      AssignPublicIp: boolean
      SecurityGroups:
        - string
      AdditionalSecurityGroups:
        - string
      PlacementGroup:
        Enabled: boolean
        Id: string
        Name: string
      Proxy:
        HttpProxyAddress: string
    ComputeResources:
      - Name: string
        InstanceType: string
        Instances:
          - InstanceType: string        
        MinCount: integer
        MaxCount: integer
        DynamicNodePriority: integer
        StaticNodePriority: integer
        SpotPrice: float
        DisableSimultaneousMultithreading: boolean
        SchedulableMemory: integer
        HealthChecks:
          Gpu:
            Enabled: boolean
        Efa:
          Enabled: boolean
          GdrSupport: boolean    
        CapacityReservationTarget:
          CapacityReservationId: string
          CapacityReservationResourceGroupArn: string     
        Networking:   
          PlacementGroup:
            Enabled: boolean
            Name: string
        CustomSlurmSettings: dict
        Tags:
          - Key: string
            Value: string
        LaunchTemplateOverrides:
          LaunchTemplateId: string
          Version: string
    CustomActions:
      OnNodeStart:
        Sequence:
          - Script: string
            Args:
              - string
        Script: string
        Args:
          - string
      OnNodeConfigured:
        Sequence:
          - Script: string
            Args:
              - string        
        Script: string
        Args:
          - string
    Iam:
      InstanceProfile: string
      InstanceRole: string
      S3Access:
        - BucketName: string
          EnableWriteAccess: boolean
          KeyName: string
      AdditionalIamPolicies:
        - Policy: string
    Image:
      CustomAmi: string

Update policy: For this list values setting, a new value can be added during an update or the compute fleet must be stopped when removing an existing value.

SlurmQueues properties

Name (Required, String)

The name of the Slurm queue.

Note

Cluster size may change during an update. For more information, see Cluster capacity size and update

Update policy: If this setting is changed, the update is not allowed.

CapacityReservationTarget
Note

CapacityReservationTarget is added with AWS ParallelCluster version 3.3.0.

CapacityReservationTarget:
   CapacityReservationId: string
   CapacityReservationResourceGroupArn: string

Specifies the On-Demand capacity reservation for the queue's compute resources.

CapacityReservationId (Optional, String)

The ID of the existing capacity reservation to target for the queue's compute resources. The ID can refer to an ODCR or a Capacity Block for ML.

The reservation must use the same platform that the instance uses. For example, if your instances run on rhel8, your capacity reservation must run on the Red Hat Enterprise Linux platform. For more information, see Supported platforms in the Amazon EC2 User Guide for Linux Instances.

Note

If you include Instances in the cluster configuration, you must exclude this queue level CapacityReservationId setting from the configuration.

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

CapacityReservationResourceGroupArn (Optional, String)

The Amazon Resource Name (ARN) of the resource group that serves as the service-linked group of capacity reservations for the queue's compute resources. AWS ParallelCluster identifies and uses the most appropriate capacity reservation from the resource group based on the following conditions:

The resource group must have at least one ODCR for each instance type reserved in an Availability Zone across all of the queue's compute resources and Availability Zones. For more information, see Launch instances with On-Demand Capacity Reservations (ODCR).

For more information on multiple subnet configuration requirements, see Networking / SubnetIds.

Note

Multiple Availability Zones is added in AWS ParallelCluster version 3.4.0.

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

CapacityType (Optional, String)

The type of the compute resources that the Slurm queue uses. Supported values are ONDEMAND , SPOT or CAPACITY_BLOCK. The default value is ONDEMAND.

Note

If you set the CapacityType to SPOT, your account must have an AWSServiceRoleForEC2Spot service-linked role. You can use the following AWS CLI command to create this role.

$ aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

For more information, see Service-linked role for Spot Instance requests in the Amazon Amazon EC2 User Guide for Linux Instances.

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

AllocationStrategy (Optional, String)

Specify the allocation strategy for all the compute resources defined in Instances.

Valid values: lowest-price | capacity-optimized | price-capacity-optimized | prioritized | capacity-optimized-prioritized

CapacityType Allowed strategies
ONDEMAND lowest-price, prioritized
SPOT lowest-price, capacity-optimized, price-capacity-optimized, capacity-optimized-prioritized
CAPACITY_BLOCK Not supported — AllocationStrategy cannot be configured

Default: lowest-price

lowest-price

  • If you use CapacityType = ONDEMAND, Amazon EC2 Fleet uses price to determine the order and launches the lowest price instances first.

  • If you use CapacityType = SPOT, Amazon EC2 Fleet launches instances from the lowest price Spot Instance pool that has available capacity. If a pool runs out of capacity before it fulfills your required capacity, Amazon EC2 Fleet fulfills your request by launching instances for you. In particular, Amazon EC2 Fleet launches instances from the lowest price Spot Instance pool that has available capacity. Amazon EC2 Fleet might launch Spot Instances from several different pools.

  • If you set CapacityType = CAPACITY_BLOCK, there are no allocation strategies, thus AllocationStrategy parameter cannot be configured.

capacity-optimized

  • If you set CapacityType = ONDEMAND, capacity-optimized isn't available.

  • If you set CapacityType = SPOT, Amazon EC2 Fleet launches instances from Spot Instance pools with optimal capacity for the number of instances to be launched.

price-capacity-optimized

  • If you set CapacityType = ONDEMAND, capacity-optimized isn't available.

  • If you set CapacityType = SPOT, Amazon EC2 Fleet identifies the pools with the highest capacity availability for the number of instances that are launching. This means that we will request Spot Instances from the pools that we believe have the lowest chance of interruption in the near term. Amazon EC2 Fleet then requests Spot Instances from the lowest priced of these pools.

prioritized

  • If you set CapacityType = ONDEMAND, Amazon EC2 Fleet honors the priority order that AWS ParallelCluster applies to the LaunchTemplate overrides when multiple subnets are specified. AWS ParallelCluster derives the override priority from the position of the target subnet in SlurmQueues/Networking/SubnetIds with the first Subnet getting the highest priority. The priorities are drived by AWS ParallelCluster in descending order from SlurmQueues/Networking/SubnetIds, with the first SubnetId having the highest priority and the last SubnetID having the lowest priority.

  • If you set CapacityType = SPOT, prioritized isn't available.

capacity-optimized-prioritized

  • If you set CapacityType = ONDEMAND, capacity-optimized-prioritized isn't available.

  • If you set CapacityType = SPOT, Amazon EC2 Fleet optimizes for capacity first and then applies, on a best-effort basis, the priority order that AWS ParallelCluster assigns to LaunchTemplate overrides. The priorities are drived by AWS ParallelCluster in descending order from SlurmQueues/Networking/SubnetIds, with the first SubnetId having the highest priority and the last SubnetID having the lowest priority. All overrides that target the same subnet receive the same priority value.

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

Note

AllocationStrategy is supported starting in AWS ParallelCluster version 3.3.0.

New in 3.14.0: prioritized (for On-Demand) and capacity-optimized-prioritized (for Spot).

JobExclusiveAllocation (Optional, String)

If set to true, the Slurm partition OverSubscribe flag is set to EXCLUSIVE. When OverSubscribe=EXCLUSIVE, jobs in the partition have exclusive access to all allocated nodes. For more information, see EXCLUSIVE in the Slurm documentation.

Valid values: true | false

Default: false

Update policy: This setting can be changed during an update.

Note

JobExclusiveAllocation is supported starting in AWS ParallelCluster version 3.7.0.

CustomSlurmSettings (Optional, Dict)

Defines the custom Slurm partition (queue) configuration settings.

Specifies a dictionary of custom Slurm configuration parameter key-value pairs that apply to queues (partitions).

Each separate key-value pair, such as Param1: Value1, is added separately to the end of the Slurm partition configuration line in the format Param1=Value1.

You can only specify Slurm configuration parameters that aren't deny-listed in CustomSlurmSettings. For information about deny-listed Slurm configuration parameters, see Deny-listed Slurm configuration parameters for CustomSlurmSettings.

AWS ParallelCluster only checks whether a parameter is in a deny list. AWS ParallelCluster doesn't validate your custom Slurm configuration parameter syntax or semantics. It is your responsibility to validate your custom Slurm configuration parameters. Invalid custom Slurm configuration parameters can cause Slurm daemon failures that can lead to cluster create and update failures.

For more information about how to specify custom Slurm configuration parameters with AWS ParallelCluster, see Slurm configuration customization.

For more information about Slurm configuration parameters, see slurm.conf in the Slurm documentation.

Update policy: This setting can be changed during an update.

Note

CustomSlurmSettings is supported starting with AWS ParallelCluster version 3.6.0.

Tags (Optional, [String])

A list of tag key-value pairs. ComputeResource tags override duplicate tags specified in the Tags section or in SlurmQueues / Tags.

Key (Optional, String)

The tag key.

Value (Optional, String)

The tag value.

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

HealthChecks (Optional)

Specify compute node health checks on all compute resources in the queue.

Gpu (Optional)

Specify GPU health checks on all compute resources in a queue.

Note

AWS ParallelCluster doesn't support HealthChecks / Gpu in nodes that use alinux2 ARM operating systems. These platforms don't support the NVIDIA Data Center GPU Manager (DCGM).

Important

The built-in GPU health check runs an NVIDIA DCGM level-2 diagnostic in the Slurm prolog. Enable it only with job-exclusive allocation (one job per node; set JobExclusiveAllocation: true). Do not enable it on P6, P6e, or later P-family GPU instances. For details, see Best practices: GPU health checks.

Enabled (Optional, Boolean)

Whether AWS ParallelCluster performs GPU health checks on compute nodes. The default is false.

Gpu health check behavior

  • If Gpu / Enabled is set to true, AWS ParallelCluster performs GPU health checks on compute resources in the queue.

  • The Gpu health check performs GPU health checks on compute resources to prevent the submission of jobs on nodes with a degraded GPU.

  • If a compute node fails a Gpu health check, the compute node state changes to DRAIN. New jobs don't start on this node. Existing jobs run to completion. After all running jobs complete, the compute node terminates if it's a dynamic node, and it's replaced if it's a static node.

  • The duration of the Gpu health check depends on the selected instance type, the number of GPUs in the instance, the total GPU memory and the number of Gpu health check targets (equivalent to the number of job GPU targets). For example, on a p4d.24xlarge, the typical duration is 3 minutes.

  • If the Gpu health check runs on an instance that's not supported, it exits and the job runs on the compute node. For example, if an instance doesn't have a GPU, or, if an instance has a GPU, but it isn't an NVIDIA GPU, the health check exits and the job runs on the compute node. Only NVIDIA GPUs are supported.

  • The Gpu health check uses the dcgmi tool to perform health checks on a node and takes the following steps:

    When the Gpu health check begins in a node:

    1. It detects whether the nvidia-dcgm and nvidia-fabricmanager services are running.

    2. If these services aren't running, the Gpu health check starts them.

    3. It detects whether the persistence mode is enabled.

    4. If the persistence mode isn't enabled, the Gpu health check enables it.

    At the end of the health check, the Gpu health check restores these services and resources to their initial state.

  • If the job is assigned to a specific set of node GPUs, the Gpu health check runs only on that specific set. Otherwise, the Gpu health check runs on all GPUs in the node.

  • If a compute node receives 2 or more Gpu health check requests at the same time, only the first health check runs and the others are skipped. This is also the case for health checks that target node GPUs. You can check the log files for additional information regarding this situation.

  • The health check log for a specific compute node is available in the /var/log/parallelcluster/slurm_health_check.log file. The file is available in Amazon CloudWatch, in the cluster CloudWatch log group, where you can find:

    • Details on the action run by the Gpu health check, including enabling and disabling services and persistence mode.

    • The GPU identifier, serial ID, and the UUID.

    • The health check output.

Update policy: This setting can be changed during an update.

Note

HealthChecks is supported starting in AWS ParallelCluster version 3.6.0.

Networking

(Required) Defines the networking configuration for the Slurm queue.

Networking:
  SubnetIds:
    - string
  AssignPublicIp: boolean
  SecurityGroups:
    - string
  AdditionalSecurityGroups:
    - string
  PlacementGroup:
    Enabled: boolean
    Id: string
    Name: string
  Proxy:
    HttpProxyAddress: string

Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

Networking properties
SubnetIds (Required, [String])

The IDs of existing subnets that you provision the Slurm queue in.

If you configure instance types in SlurmQueues / ComputeResources / InstanceType, you can only define one subnet.

If you configure instance types in SlurmQueues / ComputeResources / Instances, you can define a single subnet or multiple subnets.

If you use multiple subnets, all subnets defined for a queue must be in the same VPC, with each subnet in a separate Availability Zone (AZ).

For example, suppose you define subnet-1 and subnet-2 for your queue.

subnet-1 and subnet-2 can't both be in AZ-1.

subnet-1 can be in AZ-1 and subnet-2 can be in AZ-2.

If you configure only one instance type and want to use multiple subnets, define your instance type in Instances rather than InstanceType.

For example, define ComputeResources / Instances / InstanceType=instance.type instead of ComputeResources / InstanceType=instance.type.

Note

Elastic Fabric Adapter (EFA) isn't supported over different availability zones.

The use of multiple Availability Zones might cause increases in storage networking latency and added inter-AZ data transfer costs. For example, this could occur when an instance accesses file storage that's located in a different AZ. For more information, see Data Transfer within the same AWS Region.

Cluster updates to change from the use of a single subnet to multiple subnets:

  • Suppose the subnet definition of a cluster is defined with a single subnet and an AWS ParallelCluster managed FSx for Lustre file system. Then, you can't update this cluster with an updated subnet ID definition directly. To make the cluster update, you must first change the managed file system to an external file system. For more information, see Convert AWS ParallelCluster managed storage to external storage.

  • Suppose the subnet definition of a cluster is defined with a single subnet and an external Amazon EFS file system if EFS mount targets don't exist for all of the AZs for the multiple subnets defined to be added. Then, you can't update this cluster with an updated subnet ID definition directly. To make the cluster update or to create a cluster, you must first create all of the mount targets for all of the AZs for the defined multiple subnets.

Availability Zones and cluster capacity reservations defined in CapacityReservationResourceGroupArn:

  • You can't create a cluster if there is no overlap between the set of instance types and availability zones covered by the defined capacity reservation resource group and the set of instance types and availability zones defined for the queue.

  • You can create a cluster if there is a partial overlap between the set of instance types and availability zones covered by the defined capacity reservation resource group and the set of instance types and availability zones defined for the queue. AWS ParallelCluster sends a warning message about the partial overlap for this case.

  • For more information, see Launch instances with On-Demand Capacity Reservations (ODCR).

Note

Multiple Availability Zones is added in AWS ParallelCluster version 3.4.0.

Warning

This warning applies to all 3.x.y AWS ParallelCluster versions prior to version 3.3.1. AWS ParallelCluster version 3.3.1 isn't impacted if this parameter is changed.

For AWS ParallelCluster 3 versions prior to version 3.3.1:

If you change this parameter and update a cluster this creates a new managed FSx for Lustre file system and deletes the existing managed FSx for Lustre file system without preserving the existing data. This results in data loss. Before you proceed, make sure you back up the data from the existing FSx for Lustre file system if you want to preserve data. For more information, see Working with backups in the FSx for Lustre User Guide.

If a new subnet value is added, Update policy: This setting can be changed during an update.

If a subnet value is removed, Update policy: The compute fleet must be stopped or QueueUpdateStrategy must be set for this setting to be changed for an update.

AssignPublicIp (Optional, String)

Creates or assigns a public IP address to the nodes in the Slurm queue. Supported values are true and false. The subnet that you specify determines the default value. A subnet with public IPs default to assigning public IP addresses.

If you define a p4d or hpc6id instance type, or another instance type that has multiple network interfaces or a network interface card, you must set HeadNode / Networking / ElasticIp to true to provide public access. AWS public IPs can only be assigned to instances launched with a single network interface. For this case, we recommend that you use a NAT gateway to provide public access to the cluster compute nodes. In this case, set AssignPublicIp to false. For more information on IP addresses, see