View a markdown version of this page

Amazon DocumentDB migration runbook - Amazon DocumentDB
CompatibilityWorkload discoveryCreate new target clusterIndex migrationUser migrationData migrationMonitoringValidation

Amazon DocumentDB migration runbook

This runbook provides a comprehensive guide for migrating a MongoDB database to Amazon DocumentDB using AWS Database Migration Service (DMS). It is designed to support database administrators, cloud engineers, and developers throughout the end-to-end migration journey—from initial discovery to post-migration validation.

Given the differences in implementation and supported features between MongoDB and Amazon DocumentDB, this runbook emphasizes a structured and systematic approach. It outlines essential pre-migration assessments, highlights compatibility considerations, and details the key tasks required to ensure a successful migration with minimal disruption.

The runbook is organized into the following topics:

  • Compatibility — Understand the supported MongoDB features and data types in Amazon DocumentDB, and identify potential incompatibilities.

  • Workload discovery — Analyze existing MongoDB workloads, including read/write patterns, data volumes, and performance baselines.

  • Index migration — Analyze strategies for extracting and transforming MongoDB indexes for optimal performance in Amazon DocumentDB.

  • User migration — Detail the approach for migrating database users, roles, and access controls to Amazon DocumentDB.

  • Data migration — Cover various methods for data migration using AWS DMS, including full load and change data capture (CDC).

  • Monitoring — Detail various monitoring approaches when migrating using DMS or native tools.

  • Validation — Provide procedures for data integrity checks, functional validation, and performance comparison post-migration.

By following the guidance in this runbook, teams can ensure a smooth, secure, and efficient transition to Amazon DocumentDB, while preserving application functionality and minimizing risk.

Compatibility

When migrating from MongoDB to Amazon DocumentDB, a thorough initial assessment and feature compatibility check is essential for a successful migration. This process begins with a comprehensive inventory of your MongoDB features, including aggregation pipeline operators, query patterns, indexes, and data models.

Since Amazon DocumentDB is compatible with MongoDB 3.6, 4.0, 5.0, and 8.0 API's, applications using newer MongoDB-specific features may require refactoring. Critical areas to evaluate include sharding mechanisms(Amazon DocumentDB uses a different approach), transaction implementations, change streams functionality, and index types (particularly sparse and partial indexes).

Performance characteristics also differ, with Amazon DocumentDB optimized for enterprise workloads with predictable performance. Testing should involve running representative workloads against both systems to identify query patterns that might need optimization.

Monitoring execution plans to detect potential performance gaps is important during the assessment phase. This helps create a clear migration roadmap, identifying necessary application changes and establishing realistic timelines for a smooth transition.

Core feature compatibility

Comprehensive feature support

  • CRUD operations — Enjoy full support for all basic create, read, update, and delete operations, including bulk and query operators - providing seamless application compatibility.

  • Rich indexing capabilities — Leverage comprehensive support for single field, compound, TTL, partial, sparse, and 2dsphere indexes, to optimize your query performance and text indexes (version 5) for text-based lookups.

  • Enterprise-grade replication — Benefit from a robust automatic failover mechanism with read replicas for superior high availability without operational overhead.

  • Advanced backup solutions — Rest easy with automated backup system featuring Point-in-Time Recovery (PITR) and on-demand manual snapshots for data protection.

Enhanced AWS-integrated features

  • Streamlined aggregation — Take advantage of the most commonly used aggregation stages ($match, $group, $sort, $project, etc.) with optimized performance for enterprise workloads.

  • Transaction support — Implement multi-document and multi-collection transactions, perfect for most business application needs.

  • Real-time data tracking — Enable change streams by a simple command and increase change stream retention period through a simple parameter group setting for real-time data change monitoring.

  • Location-based services — Implement geospatial applications with support for $geoNear operator and 2dsphere indexes.

  • Text search capabilities — Utilize built-in text search functionality for content discovery needs.

Modern architecture advantages

  • Cloud-native design — Enjoy AWS-optimized architecture that replaces legacy features like MapReduce with more efficient aggregation pipeline operations.

  • Enhanced security — Benefit from AWS Identity and Access Management (IAM), SCRAM-SHA-1, SCRAM-SHA-256, X.509 certificate authentication, and password-based authentication.

  • Predictable performance — Experience consistent performance optimized specifically for enterprise workloads.

For a comprehensive overview of Amazon DocumentDB's capabilities, refer to the Supported MongoDB APIs, operations, and data types in Amazon DocumentDB and Functional differences: Amazon DocumentDB and MongoDB to maximize your database's potential.

Amazon DocumentDB does not support all the indexes offered by MongoDB. We provide a free index tool on GitHub to check the compatibility. Run the index tool to assess incompatibility and plan workarounds accordingly.

Amazon DocumentDB compatibility assessment tool

The Amazon DocumentDB Compatibility Tool on GitHub analyzes your MongoDB workload and reports operators, commands, and features that are unsupported or require changes in Amazon DocumentDB. It supports target versions from 3.6 through 8.0.1 (default: 8.0.1).

Recommended: URI mode (MongoDB 5.0+)

URI mode connects directly to your MongoDB instance, samples operations in real time, and requires no profiling or log access.

Alternative methods

  • Log-based analysis — Parses MongoDB profiler logs. Captures actual runtime behavior but requires profiling to be enabled and only covers the logging period.

  • Source code analysis — Scans application source code for MongoDB API usage. Provides comprehensive coverage but may flag unused code paths and cannot detect dynamically constructed queries.

Prerequisites

  • Python 3.7+

  • Clone the tool: git clone https://github.com/awslabs/amazon-documentdb-tools.git

  • Install dependencies: pip install -r compat-tool/requirements.txt

For full usage details and all available options, see the README on GitHub.

Workload discovery

Migrating from MongoDB to Amazon DocumentDB requires a thorough understanding of the existing database workload. Workload discovery is the process of analyzing your database usage patterns, data structures, query performance, and operational dependencies to ensure a seamless transition with minimal disruption. This section outlines the key steps involved in workload discovery to facilitate an effective migration from MongoDB to Amazon DocumentDB.

Assessing the existing MongoDB deployment

Before migration, it is crucial to evaluate the current MongoDB environment, including:

  • Cluster architecture — Identify the number of nodes, replica sets, and sharding configurations. When migrating from MongoDB to Amazon DocumentDB, understanding your MongoDB sharding configuration is important because Amazon DocumentDB does not support user-controlled sharding. Applications designed for a sharded MongoDB environment will need architectural changes, as Amazon DocumentDB uses a different scaling approach with its storage-based architecture. You'll need to adapt your data distribution strategy and possibly consolidate sharded collections when moving to Amazon DocumentDB.

  • Storage and data volume — Measure the total data size and index size of your cluster. Complement this with the Oplog review tool to understand write patterns and data growth velocity. For more information about sizing your cluster, see Instance sizing.

  • Workload patterns — Analyze read and write throughput, query execution frequency, and indexing efficiency.

  • Operational dependencies — Document all applications, services, and integrations relying on MongoDB.

Identifying data model differences

Although Amazon DocumentDB is MongoDB-compatible, there are differences in supported features, such as:

Query and performance analysis

Understanding query behavior helps optimize migration and post-migration performance. Key areas to analyze include:

  • Slow queries — Identify queries with high execution time using MongoDB’s profiling tools.

  • Query patterns — Categorize common query types, including CRUD operations and aggregations.

  • Index usage — Assess whether indexes are effectively utilized or need optimization in Amazon DocumentDB. To assess index usage and optimize performance in Amazon DocumentDB, use the $indexStats aggregation pipeline stage combined with the explain() method on your critical queries. Start by running db.collection.aggregate([{$indexStats{}}]) to identify which indexes are being used. You can do more detailed analysis by executing you most frequent queries with explainPlan.

  • Concurrency & workload distribution — Evaluate read and write ratios, connection pooling, and performance bottlenecks.

Security and access control review

Authentication and authorization

  • MongoDB RBAC to Amazon DocumentDB IAM and RBAC — Map MongoDB's role-based access control users and roles to AWS Identity and Access Management (IAM) policies and Amazon DocumentDB SCRAM authentication users.

  • User migration strategy — Plan for migrating database users, custom roles, and privileges to Amazon DocumentDB's supported authentication mechanisms.

  • Privilege differences — Identify MongoDB privileges without direct Amazon DocumentDB equivalents (for example, cluster administration roles).

  • Application authentication — Update connection strings and credential management for Amazon DocumentDB's password policies. You can use secrets manager to store your credentials and rotate passwords.

  • Service account management — Establish processes for managing service account credentials in AWS Secrets Manager.

  • Least privilege implementation — Review and refine access controls to implement least privilege principles in the new environment.

Encryption

Ensure encryption at rest and in transit aligns with compliance requirements.

Network configuration

Plan for Virtual Private Cloud (VPC) setup and security group rules.

Operational and monitoring considerations

To maintain system reliability, workload discovery should also include:

  • Backup and restore strategy — Evaluate existing backup methods and Amazon DocumentDB’s backup capabilities.

  • AWS Backup integration — Leverage AWS Backup for centralized backup management across AWS services including Amazon DocumentDB.

  • CloudWatch metrics — Map MongoDB monitoring metrics to Amazon DocumentDB CloudWatch metrics for CPU, memory, connections, and storage.

  • Performance Insights — Implement Amazon DocumentDB Performance Insights to visualize database load and analyze performance issues with detailed query analytics.

  • Profiler — Configure Amazon DocumentDB profiler to capture slow-running operations (similar to MongoDB's profiler but with Amazon DocumentDB-specific settings).

    • Enable through parameter groups with appropriate thresholds.

    • Analyze profiler data to identify optimization opportunities

  • CloudWatch Events — Set up event-driven monitoring for Amazon DocumentDB cluster events.

    • Configure notifications for backup events, maintenance windows, and failovers.

    • Integrate with Amazon SNS for alerting and AWS Lambda for automated responses.

  • Audit logging — Plan for audit logging configuration to track user activity and security-relevant events.

  • Enhanced monitoring — Enable enhanced monitoring for granular OS-level metrics at 1-second intervals.

Create new target cluster

Create an Amazon DocumentDB cluster following the steps in Creating an Amazon DocumentDB cluster. When configuring the cluster for migration, apply these recommendations:

  • Instance class: Choose R8G for best performance. If R8G is unavailable in your target Region, use R6G. Choose as large an instance as possible for best full-load throughput. Scale down after migration is complete.

  • Engine version: latest (8.0.1)

  • Number of instances: Choose 1 instance to minimize costs during migration. Scale to 3 instances for high availability after the full-load migration is complete.

  • Network settings: Ensure your Amazon DocumentDB security group allows inbound connections from the DMS replication instance's security group (for online migrations) or from the EC2 migration host (for offline migrations using mongorestore).

Index migration

Migrating from MongoDB to Amazon DocumentDB involves transferring not just data but also indexes to maintain query performance and optimize database operations. This section outlines the detailed step-by-step process for migrating indexes from MongoDB to Amazon DocumentDB while ensuring compatibility and efficiency.

Using the Amazon DocumentDB index tool

Clone the index tool on GitHub

git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/index-tool
pip install -r requirements.txt

Export indexes from MongoDB (if migrating from MongoDB)

python3 migrationtools/documentdb_index_tool.py \
  --dump-indexes \
  --dir index_export \
  --uri 'mongodb://<username>:<password>@<source_endpoint>:27017'

Verify indexes

python3 migrationtools/documentdb_index_tool.py \
  --show-issues \
  --dir index_export

Import indexes

python3 migrationtools/documentdb_index_tool.py \
  --restore-indexes \
  --skip-incompatible \
  --dir index_export \
  --uri 'mongodb://<username>:<password>@<target_endpoint>:27017/?tls=true&tlsCAFile=global-bundle.pem&replicaSet=rs0&retryWrites=false'

User migration

Migrating users from MongoDB to Amazon DocumentDB is essential for maintaining access control, authentication, and database security. This section outlines detailed steps to successfully migrate MongoDB users while preserving their roles and permissions using the Amazon DocumentDB export user tool.

Using Amazon DocumentDB export users tool

The Export Users tool exports users and roles from MongoDB or Amazon DocumentDB to JavaScript files, which can then be used to recreate them in another cluster.

Prerequisites

# Clone the repository
git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/migration/export-users
# Install required dependencies
pip install pymongo

Step 1: Export users and roles

# Export users and roles to JavaScript files
python3 docdbExportUsers.py \
  --users-file mongodb-users.js \
  --roles-file mongodb-roles.js \
  --uri "mongodb://<username>:<password>@<source_endpoint>:27017"

Step 2: Edit the Users File

Passwords are not exported for security reasons. Open the mongodb-users.js file and add a password for each user by replacing the pwd value in each createUser statement:

db.getSiblingDB("admin").createUser({
  user: "appuser",
  pwd: "REPLACE_THIS_PASS",
  roles: [
    { role: "readWrite", db: "mydb" }
  ]
})

Step 3: Restore Custom Roles to Amazon DocumentDB

# Import roles first
mongosh \
  --tls \
  --host <target_endpoint>:27017 \
  --tlsCAFile global-bundle.pem \
  --username <username> \
  --password <password> \
  mongodb-roles.js

Step 4: Restore Users to Amazon DocumentDB

# Import users after roles are created
mongosh \
  --tls \
  --host <target_endpoint>:27017 \
  --tlsCAFile global-bundle.pem \
  --username <username> \
  --password <password> \
  mongodb-users.js

Important notes

  • Passwords are not exported for security reasons and must be manually added to the users.js file.

  • Roles must be imported before users to ensure proper role assignments.

  • The tool generates JavaScript files that can be directly executed with the mongosh shell.

  • Custom roles and their privileges are preserved during migration.

  • This approach allows for review and modification of user permissions before importing.

This method provides a secure and flexible approach to migrating users and roles from MongoDB to Amazon DocumentDB while allowing for password resets during the migration process.

Data migration

Online migration

This section provides detailed steps to perform an online migration from MongoDB to Amazon DocumentDB using AWS DMS to enable minimal downtime and continuous replication. To begin, you set up an Amazon DocumentDB cluster as the target and ensure your MongoDB instance is properly configured as the source, typically requiring replica set mode for change data capture. Next, you create a DMS replication instance and define source and target endpoints with the necessary connection details. After validating the endpoints, you configure and start a migration task that can include full data load, ongoing replication, or both.

Create new target cluster

See Create new target cluster.

Configure source

MongoDB and Amazon DocumentDB can both serve as migration sources, depending on your scenario:

  • MongoDB as source — Common when migrating from an on-premises or a self-managed MongoDB to an Amazon DocumentDB or other AWS database services. Requires running in replica set mode with an adequately sized oplog (make sure it is sized to hold all operations during Full Load) to support change data capture during migration.

  • Amazon DocumentDB as source — Typically used for cross-Region replication, version upgrades, or migrating to other database services like MongoDB Atlas. Requires Enabling change streams by setting the change_stream_log_retention_duration parameter in the cluster parameter group to capture ongoing changes during migration. Make sure your change_stream_log_retention_duration setting is large enough to cover the time needed to complete the Full Load.

Before starting migration, configure your source to allow AWS DMS access.

Create a MongoDB user with proper permissions:

db.createUser({
  user: "dmsUser",
  pwd: "<password>",
  roles: [{ role: "readAnyDatabase", db: "admin" }]
})

Configure network and authentication.

When configuring network connectivity for MongoDB to DMS migration:

EC2-hosted MongoDB source

  • Modify the EC2 security group to allow inbound traffic from the DMS replication instance security group.

  • Add a rule for TCP port 27017 (or your custom MongoDB port).

  • Use the DMS replication instance's security group ID as the source for precise access control.

  • Ensure the EC2 instance's subnet has a route to the DMS replication instance's subnet.

On-premises MongoDB source

  • Configure your firewall to allow inbound connections from the DMS replication instance's public IP addresses.

  • If using Direct Connect or a VPN, ensure proper routing between your network and the VPC containing the DMS instance.

  • Test connectivity using telnet or nc commands from the DMS subnet to your MongoDB server.

MongoDB Atlas source

  • Add a DMS replication instance IP addresses to the MongoDB Atlas IP allowlist.

  • Configure VPC peering between AWS VPC and MongoDB Atlas VPC if Atlas is running on AWS.

  • Set up AWS PrivateLink for private connectivity (Enterprise tier), if running on another cloud provider.

  • Create a dedicated user with appropriate read/write permissions.

  • Use a MongoDB Atlas connection string with SSL Mode set to "verify-full".

  • Ensure sufficient oplog size for migration duration.

Amazon DocumentDB source

Configure your source Amazon DocumentDB security group to allow inbound traffic from the DMS replication instance security group.

Create DMS replication instance

Use DMS Buddy on GitHub to create optimal migration infrastructure with optimal DMS settings and instance sizes. If you prefer to configure manually, follow these steps:

  1. In the AWS DMS console, in the navigation pane, choose Migrate or replicate, then choose Provisioned instances.

  2. Enter replication instance details:

    • Instance name: Choose a unique name.

    • Instance class: Select based on workload. Example: dms.r7i.large (small workloads), dms.r7i.4xlarge (large workloads).

    • Engine version: 3.5.4

    • Allocated storage: Default is 50 GB (increase if needed). This is determined by document size, updates/second and full load duration.

    • Multi-AZ Deployment: Enable for high availability, if needed.

    • Choose the same VPC as Amazon DocumentDB.

    • Ensure Security groups allow inbound traffic from source and Amazon DocumentDB.

  3. Choose Create replication instance and wait for the status to be available.

Create DMS endpoints

Create a source endpoint

For a MongoDB source

  1. In the DMS console, in the navigation pane, choose Migrate or replicate, then choose Endpoints.

  2. Choose Create endpoint.

  3. On the Create endpoint page, choose Source endpoint.

  4. In the Endpoint configuration section:

    • Enter a unique and meaningful Endpoint identifier (for example, "mongodb-source").

    • Choose MongoDB as the Source engine.

    • For Access to endpoint database, choose Provide access information manually.

    • For Server name, enter your MongoDB server DNS name/IP address.

    • For Port, enter 27017 (default MongoDB port).

    • For Authentication mode, choose the appropriate mode for your application (password/SSL) (default is secrets manager).

    • If Authentication mode is Password, provide:

      • Username and Password: Enter MongoDB credentials.

      • Database name: Your source database name.

      • Authentication mechanism: SCRAM-SHA-1 (default) or appropriate mechanism

  5. For Metadata mode, leave the default setting of document.

  6. Additional connection attributes:

    • authSource=admin (if authentication database is different)

    • replicaSet=<your-replica-set-name> (required for CDC)

For an Amazon DocumentDB source

  1. In the DMS console, in the navigation pane, choose Migrate or replicate, then choose Endpoints.

  2. Choose Create endpoint.

  3. On the Create endpoint page, choose Source endpoint.

  4. In the Endpoint configuration section:

    • Enter a unique and meaningful Endpoint identifier (for example, "docdb-source").

    • Choose Amazon DocumentDB as the Source engine.

    • For Access to endpoint database, choose Provide access information manually.

    • For Server name, enter your source Amazon DocumentDB cluster endpoint.

    • For Port, enter 27017 (default Amazon DocumentDB port).

    • For SSL mode, choose verify-full (recommended for Amazon DocumentDB).

    • For CA Certificate, choose the Amazon RDS root CA certificate.

    • For Authentication mode, choose the appropriate mode for your application (password/SSL) (default is secrets manager).

    • If Authentication mode is Password, provide:

      • Username and Password: Enter Amazon DocumentDB credentials.

      • Database name: Your source database name.

      • Authentication mechanism: SCRAM-SHA-1 (default) or appropriate mechanism

  5. For Metadata mode, leave the default setting of document.

Create a target endpoint (Amazon DocumentDB)

  1. In the DMS console, in the navigation pane, choose Migrate or replicate, then choose Endpoints.

  2. Choose Create endpoint.

  3. On the Create endpoint page, choose Target endpoint.

  4. In the Endpoint configuration section:

    • Enter a unique and meaningful Endpoint identifier (for example, "docdb-target").

    • Choose Amazon DocumentDB as the Target engine.

    • For Access to endpoint database, choose the method you want to use to authenticate access to the database:

      • If you choose AWS Secrets Manager, choose the secret where you store your Amazon DocumentDB credentials in the