Troubleshooting tools Configuration errors External key store connection errors Latency and timeout errors Authentication credential errors Key state errors Decryption errors External key errors Proxy issues Proxy authorization issues

Troubleshooting external key stores

The resolution for most problems with external key stores are indicated by the error message that AWS KMS displays with each exception, or by the connection error code that AWS KMS returns when an attempt to connect the external key store to its external key store proxy fails. However, some issues are a bit more complex.

When diagnosing an issue with an external key store, first locate the cause. This will narrow the range of remedies and make your troubleshooting more efficient.

AWS KMS — The problem might be within AWS KMS, such as an incorrect value in your external key store configuration.
External — The problem might originate outside of AWS KMS, including problems with the configuration or operation of the external key store proxy, external key manager, external keys, or VPC endpoint service.
Networking — It might be a problem with connectivity or networking, such as a problem with your proxy endpoint, port, IP stack, or your private DNS name or domain.

Note

When management operations on external key stores fail, they generate several different exceptions. But AWS KMS cryptographic operations return KMSInvalidStateException for all failures related to the external configuration or connection state of the external key store. To identify the problem, use the accompanying error message text.

The ConnectCustomKeyStore operation succeeds quickly before the connection process is complete. To determine whether the connection process is successful, view the connection state of the external key store. If the connection process fails, AWS KMS returns a connection error code that explains the cause and suggests a remedy.

Topics

Troubleshooting tools for external key stores
Configuration errors
External key store connection errors
Latency and timeout errors
Authentication credential errors
Key state errors
Decryption errors
External key errors
Proxy issues
Proxy authorization issues

Troubleshooting tools for external key stores

AWS KMS provides several tools to help you identify and resolve problems with your external key store and its keys. Use these tools in conjunction with the tools provided with your external key store proxy and external key manager.

Note

Your external key store proxy and external key manager might provide easier methods of creating and maintaining your external key store and its KMS keys. For details, see the documentation for your external tools.

AWS KMS exceptions and error messages

AWS KMS provides a detailed error message about any problem it encounters. You can find additional information about AWS KMS exceptions in the AWS Key Management Service API Reference and AWS SDKs. Even if you are using the AWS KMS console, you might find these references to be helpful. For example, see the Errors list for the CreateCustomKeyStores operation.

To optimize the performance of your external key store proxy, AWS KMS returns exceptions based on your proxy's reliability within a given aggregation period of 5 minutes. In the event of a 500 Internal Server Error, 503 Service Unavailable, or connection timeout, a proxy with high reliability returns KMSInternalException and triggers an automatic retry to ensure that requests eventually succeed. However, a proxy with low reliability returns KMSInvalidStateException. For more information, see Monitoring an external key store.

If the problem surfaces in a different AWS service, such as when you use a KMS key in your external key store to protect a resource in another AWS service, the AWS service might provide additional information to help you identify the problem. If the AWS service doesn't provide the message, you can view the error message in the CloudTrail logs that record the use of your KMS key.

CloudTrail logs

Every AWS KMS API operation, including actions in the AWS KMS console, is recorded in AWS CloudTrail logs. AWS KMS records a log entry for successful and failed operations. For failed operations, the log entry includes the AWS KMS exception name (errorCode) and the error message (errorMessage). You can use this information to help you identify and resolve the error. For an example, see Decrypt failure with a KMS key in an external key store.

The log entry also includes the request ID. If the request reached your external key store proxy, you can use the request ID in the log entry to find the corresponding request in your proxy logs, if your proxy provides them.

CloudWatch metrics

AWS KMS records detailed Amazon CloudWatch metrics about the operation and performance of your external key store, including latency, throttling, proxy errors, external key manager status, the number of days until your TLS certificate expires, and the reported age of your proxy authentication credentials. You can use these metrics to develop data models for the operation of your external key store and CloudWatch alarms that alert you to impending problems before they occur.

Important

AWS KMS recommends that you create CloudWatch alarms to monitor the external key store metrics. These alarms will alert you to early signs of problems before they develop.

Monitoring graphs

AWS KMS displays graphs of the external key store CloudWatch metrics on the detail page for each external key store in the AWS KMS console. You can use the data in the graphs to help locate the source of errors, detect impending problems, establish baselines, and refine your CloudWatch alarm thresholds. For details about interpreting the monitoring graphs and using their data, see Monitor external key stores.

Displays of external key stores and KMS keys

AWS KMS displays detailed information about your external key stores and the KMS keys in the external key store in the AWS KMS console, and in the response to the DescribeCustomKeyStores and DescribeKey operations. These displays include special fields for external key stores and KMS keys with information that you can use for troubleshooting, such as the connection state of the external key store and the ID of the external key that is associated with the KMS key. For details, see View external key stores.

XKS Proxy Test Client

AWS KMS provides an open source test client that verifies that your external key store proxy conforms to the AWS KMS External Key Store Proxy API Specification. You can use this test client to identify and resolve problems with your external key store proxy.

Configuration errors

When you create an external key store, you specify property values that comprise the configuration of your external key store, such as the proxy authentication credential, proxy URI endpoint, proxy URI path, and VPC endpoint service name. When AWS KMS detects an error in a property value, the operation fails and returns an error that indicates the faulty value.

Many configuration issues can be resolved by fixing the incorrect value. You can fix an invalid proxy URI path or proxy authentication credential without disconnecting the external key store. For definitions of these values, including uniqueness requirements, see Assemble the prerequisites. For instructions about updating these values, see Edit external key store properties.

To avoid errors with your proxy URI path and proxy authentication credential values, when creating or updating your external key store, upload a proxy configuration file to the AWS KMS console. This is a JSON-based file with proxy URI path and proxy authentication credential values that is provided by your external key store proxy or external key manager. You can't use a proxy configuration file with AWS KMS API operations, but you can use the values in the file to help you provide parameter values for your API requests that match the values in your proxy.

General configuration errors

Exceptions: CustomKeyStoreInvalidStateException (CreateKey), KMSInvalidStateException (cryptographic operations), XksProxyInvalidConfigurationException (management operations, except for CreateKey)

Connection error codes: XKS_PROXY_INVALID_CONFIGURATION, XKS_PROXY_INVALID_TLS_CONFIGURATION

For external key stores with public endpoint connectivity, AWS KMS tests the property values when you create and update the external key store. For external key stores with VPC endpoint service connectivity, AWS KMS tests the property values when you connect and update the external key store.

Note

The ConnectCustomKeyStore operation, which is asynchronous, might succeed even though the attempt to connect the external key store to its external key store proxy fails. In that case, there is no exception, but the connection state of the external key store is Failed, and a connection error code explains the error message. For more information, see External key store connection errors.

If AWS KMS detects an error in a property value, the operation fails and returns XksProxyInvalidConfigurationException with one of the following error messages.

The external key store proxy rejected the request because of an invalid URI path. Verify the URI path for your external key store and update if necessary.

The proxy URI path is the base path for AWS KMS requests to the proxy APIs. If this path is incorrect, all requests to the proxy fail. To view the current proxy URI path for your external key store, use the AWS KMS console or the DescribeCustomKeyStores operation. To find the correct proxy URI path, see your external key store proxy documentation. For help correcting your proxy URI path value, see Edit external key store properties.
The proxy URI path for your external key store proxy can change with updates to your external key store proxy or external key manager. For information about these changes, see the documentation for your external key store proxy or external key manager.

XKS_PROXY_INVALID_TLS_CONFIGURATION

AWS KMS cannot establish a TLS connection to the external key store proxy. Verify the TLS configuration, including its certificate.

All external key store proxies require a TLS certificate. The TLS certificate must be issued by a public certificate authority (CA) that is supported for external key stores. For list of supported CAs, see Trusted Certificate Authorities in the AWS KMS External Key Store Proxy API Specification.
For public endpoint connectivity, the subject common name (CN) on the TLS certificate must match the domain name in the proxy URI endpoint for the external key store proxy. For example, if the public endpoint is https://myproxy.xks.example.com, the TLS, the CN on the TLS certificate must be myproxy.xks.example.com or *.xks.example.com.
For VPC endpoint service connectivity, the subject common name (CN) on the TLS certificate must match the private DNS name for your VPC endpoint service. For example, if the private DNS name is myproxy-private.xks.example.com, the CN on the TLS certificate must be myproxy-private.xks.example.com or *.xks.example.com.
The TLS certificate cannot be expired. To get the expiration date of a TLS certificate, use SSL tools, such as OpenSSL. To monitor the expiration date of a TLS certificate associated with an external key store, use the XksProxyCertificateDaysToExpire CloudWatch metric. The number of days to your TLS certification expiration date also appears in the Monitoring section of the AWS KMS console.
If you are using public endpoint connectivity, use SSL test tools to test your SSL configuration. TLS connection errors can result from incorrect certificate chaining.

VPC endpoint service connectivity configuration errors

Exceptions: XksProxyVpcEndpointServiceNotFoundException, XksProxyVpcEndpointServiceInvalidConfigurationException

In addition to general connectivity issues, you might encounter the following issues while creating, connecting, or updating an external key store with VPC endpoint service connectivity. AWS KMS tests the property values of an external key store with VPC endpoint service connectivity while creating, connecting, and updating the external key store. When management operations fail due to configuration errors, they generate the following exceptions:

XksProxyVpcEndpointServiceNotFoundException

The cause might be one of the following:

An incorrect VPC endpoint service name. Verify that the VPC endpoint service name for the external key store is correct and matches the proxy URI endpoint value for the external key store. To find the VPC endpoint service name, use the Amazon VPC console or the DescribeVpcEndpointServices operation. To find the VPC endpoint service name and proxy URI endpoint of an existing external key store, use the AWS KMS console or the DescribeCustomKeyStores operation. For details, see View external key stores.
The VPC endpoint service might be in a different AWS Region than the external key store. Verify that the VPC endpoint service and external key store are in same Region. (The external name of the Region name, such as us-east-1, is part of the VPC endpoint service name, such as com.amazonaws.vpce.us-east-1.vpce-svc-example.) For a list of requirements for the VPC endpoint service for an external key store, see VPC endpoint service. You cannot move a VPC endpoint service or an external key store to a different Region. However, you can create a new external key store in the same Region as the VPC endpoint service. For details, see Configure VPC endpoint service connectivity and Create an external key store.
AWS KMS is not an allowed principal for the VPC endpoint service. The Allow principals list for the VPC endpoint service must include the cks.kms.<region>.amazonaws.com value, such as cks.kms.eu-west-3.amazonaws.com. For instructions about adding this value, see Manage permissions in the AWS PrivateLink Guide.

XksProxyVpcEndpointServiceInvalidConfigurationException

This error occurs when the VPC endpoint service fails to meet one of the following requirements:

The VPC requires at least two private subnets, each in a different Availability Zone. For help adding a subnet to your VPC, see Create a subnet in your VPC in the Amazon VPC User Guide.
Your VPC endpoint service type must use a network load balancer, not a gateway load balancer.
Acceptance must not be required for the VPC endpoint service (Acceptance required must be false.). If manual acceptance of each connection request is required, AWS KMS cannot use the VPC endpoint service to connect to the external key store proxy. For details, see Accept or reject connection requests in the AWS PrivateLink Guide.
The VPC endpoint service must have a private DNS name that is a subdomain of a public domain. For example, if the private DNS name is https://myproxy-private.xks.example.com, the xks.example.com or example.com domains must have a public DNS server. To view or change the private DNS name for your VPC endpoint service, see Manage DNS names for VPC endpoint services in the AWS PrivateLink Guide.
The Domain verification status of the domain for your private DNS name must be verified. To view and update the verification status of the private DNS name domain, see Step 5: Verify your private DNS name domain. It might take a few minutes for the updated verification status to appear after you've added the required text record.

Note
A private DNS domain can be verified only if it is the subdomain of a public domain. Otherwise, the verification status of the private DNS domain does not change, even after you add the required TXT record.
Ensure that any firewalls between AWS KMS and the external key store proxy allow traffic to and from port 443 on the proxy. AWS KMS communicates on port 443 over IPv4. This value is not configurable.
The private DNS name of the VPC endpoint service must match the proxy URI endpoint value for the external key store. For an external key store with VPC endpoint service connectivity, the proxy URI endpoint must be https:// followed by the private DNS name of the VPC endpoint service. To view the proxy URI endpoint value, see View external key stores. To change the proxy URI endpoint value, see Edit external key store properties.

External key store connection errors

The process of connecting an external key store to its external key store proxy takes about five minutes to complete. Unless it fails quickly, the ConnectCustomKeyStore operation returns an HTTP 200 response and a JSON object with no properties. However, this initial response does not indicate that the connection was successful. To determine whether the external key store is connected, see its connection state. If the connection fails, the connection state of the external key store changes to FAILED and AWS KMS returns a connection error code that explains the cause of the failure.