RunJobFlow
RunJobFlow creates and starts running a new cluster (job flow). The cluster runs the
steps specified. After the steps complete, the cluster stops and the HDFS partition is
lost. To prevent loss of data, configure the last step of the job flow to store results in
Amazon S3. If the JobFlowInstancesConfig
KeepJobFlowAliveWhenNoSteps parameter is set to TRUE, the cluster
transitions to the WAITING state rather than shutting down after the steps have completed.
For additional protection, you can set the JobFlowInstancesConfig
TerminationProtected parameter to TRUE to lock the cluster and
prevent it from being terminated by API call, user intervention, or in the event of a job
flow error.
A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop.
For long-running clusters, we recommend that you periodically store your results.
Note
The instance fleets configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x versions. The RunJobFlow request can contain InstanceFleets parameters or InstanceGroups parameters, but not both.
Request Syntax
{
"AdditionalInfo": "string",
"AmiVersion": "string",
"Applications": [
{
"AdditionalInfo": {
"string" : "string"
},
"Args": [ "string" ],
"Name": "string",
"Version": "string"
}
],
"AutoScalingRole": "string",
"AutoTerminationPolicy": {
"IdleTimeout": number
},
"BootstrapActions": [
{
"Name": "string",
"ScriptBootstrapAction": {
"Args": [ "string" ],
"Path": "string"
}
}
],
"Configurations": [
{
"Classification": "string",
"Configurations": [
"Configuration"
],
"Properties": {
"string" : "string"
}
}
],
"CustomAmiId": "string",
"EbsRootVolumeIops": number,
"EbsRootVolumeSize": number,
"EbsRootVolumeThroughput": number,
"ExtendedSupport": boolean,
"Instances": {
"AdditionalMasterSecurityGroups": [ "string" ],
"AdditionalSlaveSecurityGroups": [ "string" ],
"Ec2KeyName": "string",
"Ec2SubnetId": "string",
"Ec2SubnetIds": [ "string" ],
"EmrManagedMasterSecurityGroup": "string",
"EmrManagedSlaveSecurityGroup": "string",
"HadoopVersion": "string",
"InstanceCount": number,
"InstanceFleets": [
{
"Context": "string",
"InstanceFleetType": "string",
"InstanceTypeConfigs": [
{
"BidPrice": "string",
"BidPriceAsPercentageOfOnDemandPrice": number,
"Configurations": [
{
"Classification": "string",
"Configurations": [
"Configuration"
],
"Properties": {
"string" : "string"
}
}
],
"CustomAmiId": "string",
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"Iops": number,
"SizeInGB": number,
"Throughput": number,
"VolumeType": "string"
},
"VolumesPerInstance": number
}
],
"EbsOptimized": boolean
},
"InstanceType": "string",
"Priority": number,
"WeightedCapacity": number
}
],
"LaunchSpecifications": {
"OnDemandSpecification": {
"AllocationStrategy": "string",
"CapacityReservationOptions": {
"CapacityReservationPreference": "string",
"CapacityReservationResourceGroupArn": "string",
"UsageStrategy": "string"
}
},
"SpotSpecification": {
"AllocationStrategy": "string",
"BlockDurationMinutes": number,
"TimeoutAction": "string",
"TimeoutDurationMinutes": number
}
},
"Name": "string",
"ResizeSpecifications": {
"OnDemandResizeSpecification": {
"AllocationStrategy": "string",
"CapacityReservationOptions": {
"CapacityReservationPreference": "string",
"CapacityReservationResourceGroupArn": "string",
"UsageStrategy": "string"
},
"TimeoutDurationMinutes": number
},
"SpotResizeSpecification": {
"AllocationStrategy": "string",
"TimeoutDurationMinutes": number
}
},
"TargetOnDemandCapacity": number,
"TargetSpotCapacity": number
}
],
"InstanceGroups": [
{
"AutoScalingPolicy": {
"Constraints": {
"MaxCapacity": number,
"MinCapacity": number
},
"Rules": [
{
"Action": {
"Market": "string",
"SimpleScalingPolicyConfiguration": {
"AdjustmentType": "string",
"CoolDown": number,
"ScalingAdjustment": number
}
},
"Description": "string",
"Name": "string",
"Trigger": {
"CloudWatchAlarmDefinition": {
"ComparisonOperator": "string",
"Dimensions": [
{
"Key": "string",
"Value": "string"
}
],
"EvaluationPeriods": number,
"MetricName": "string",
"Namespace": "string",
"Period": number,
"Statistic": "string",
"Threshold": number,
"Unit": "string"
}
}
}
]
},
"BidPrice": "string",
"Configurations": [
{
"Classification": "string",
"Configurations": [
"Configuration"
],
"Properties": {
"string" : "string"
}
}
],
"CustomAmiId": "string",
"EbsConfiguration": {
"EbsBlockDeviceConfigs": [
{
"VolumeSpecification": {
"Iops": number,
"SizeInGB": number,
"Throughput": number,
"VolumeType": "string"
},
"VolumesPerInstance": number
}
],
"EbsOptimized": boolean
},
"InstanceCount": number,
"InstanceRole": "string",
"InstanceType": "string",
"Market": "string",
"Name": "string"
}
],
"KeepJobFlowAliveWhenNoSteps": boolean,
"MasterInstanceType": "string",
"Placement": {
"AvailabilityZone": "string",
"AvailabilityZones": [ "string" ]
},
"ServiceAccessSecurityGroup": "string",
"SlaveInstanceType": "string",
"TerminationProtected": boolean,
"UnhealthyNodeReplacement": boolean
},
"JobFlowRole": "string",
"KerberosAttributes": {
"ADDomainJoinPassword": "string",
"ADDomainJoinUser": "string",
"CrossRealmTrustPrincipalPassword": "string",
"KdcAdminPassword": "string",
"Realm": "string"
},
"LogEncryptionKmsKeyId": "string",
"LogUri": "string",
"ManagedScalingPolicy": {
"ComputeLimits": {
"MaximumCapacityUnits": number,
"MaximumCoreCapacityUnits": number,
"MaximumOnDemandCapacityUnits": number,
"MinimumCapacityUnits": number,
"UnitType": "string"
},
"ScalingStrategy": "string",
"UtilizationPerformanceIndex": number
},
"MonitoringConfiguration": {
"CloudWatchLogConfiguration": {
"Enabled": boolean,
"EncryptionKeyArn": "string",
"LogGroupName": "string",
"LogStreamNamePrefix": "string",
"LogTypes": {
"string" : [ "string" ]
}
},
"S3LoggingConfiguration": {
"LogTypeUploadPolicy": {
"string" : "string"
}
}
},
"Name": "string",
"NewSupportedProducts": [
{
"Args": [ "string" ],
"Name": "string"
}
],
"OSReleaseLabel": "string",
"PlacementGroupConfigs": [
{
"InstanceRole": "string",
"PlacementStrategy": "string"
}
],
"ReleaseLabel": "string",
"RepoUpgradeOnBoot": "string",
"ScaleDownBehavior": "string",
"SecurityConfiguration": "string",
"ServiceRole": "string",
"SessionEnabled": boolean,
"StepConcurrencyLevel": number,
"StepExecutionRoleArn": "string",
"Steps": [
{
"ActionOnFailure": "string",
"HadoopJarStep": {
"Args": [ "string" ],
"Jar": "string",
"MainClass": "string",
"Properties": [
{
"Key": "string",
"Value": "string"
}
]
},
"Name": "string",
"StepMonitoringConfiguration": {
"S3MonitoringConfiguration": {
"EncryptionKeyArn": "string",
"LogUri": "string"
}
}
}
],
"SupportedProducts": [ "string" ],
"Tags": [
{
"Key": "string",
"Value": "string"
}
],
"VisibleToAllUsers": boolean
}Request Parameters
For information about the parameters that are common to all actions, see Common Parameters.
The request accepts the following data in JSON format.
- AdditionalInfo
-
A JSON string for selecting additional features.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 10280.
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*Required: No
- AmiVersion
-
Applies only to Amazon EMR AMI versions 3.x and 2.x. For Amazon EMR releases 4.0 and later,
ReleaseLabelis used. To specify a custom AMI, useCustomAmiID.Type: String
Length Constraints: Minimum length of 0. Maximum length of 256.
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*Required: No
- Applications
-
Applies to Amazon EMR releases 4.0 and later. A case-insensitive list of applications for Amazon EMR to install and configure when launching the cluster. For a list of applications available for each Amazon EMR release version, see the Amazon EMRRelease Guide.
Type: Array of Application objects
Required: No
- AutoScalingRole
-
An IAM role for automatic scaling policies. The default role is
EMR_AutoScaling_DefaultRole. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate Amazon EC2 instances in an instance group.Type: String
Length Constraints: Minimum length of 0. Maximum length of 10280.
Pattern:
[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*Required: No
- AutoTerminationPolicy
-
An auto-termination policy for an Amazon EMR cluster. An auto-termination policy defines the amount of idle time in seconds after which a cluster automatically terminates. For alternative cluster termination options, see Control cluster termination.
Type: