Upgrading your Amazon DocumentDB cluster from 3.6 to 4.0 using AWS Database Migration Service

You can upgrade your Amazon DocumentDB 3.6 cluster to 4.0 with minimal downtime using AWS DMS. AWS DMS is a fully managed service that makes it easy to migrate relational databases and non-relational databases to Amazon DocumentDB.

Step 1: Enable Change Streams

To perform a minimal downtime migration, AWS DMS requires access to the cluster’s change streams. Amazon DocumentDB change streams provide a time-ordered sequence of update events that occur within your cluster’s collections and databases. Reading from the change stream enables AWS DMS to perform change data capture (CDC) and apply incremental updates to the target Amazon DocumentDB cluster.

To enable change streams for all collections on a specific database, authenticate to your Amazon DocumentDB cluster using the mongo shell and execute the following commands:

db.adminCommand({modifyChangeStreams: 1,
    database: "db_name",
    collection: "", 
    enable: true});

Step 2: Modify the Change Streams Retention Duration

Next, modify the change stream retention period based on how long you would like to retain change events in the change stream. For example, if you expect your AWS DMS migration from Amazon DocumentDB v3.6 to v4.0 to take 12 hours, you should set the change stream retention to a value greater than 12 hours. The default retention period for your Amazon DocumentDB cluster is three hours. You can modify the change stream log retention duration for your Amazon DocumentDB cluster to be between one hour and seven days using the AWS Management Console or the AWS CLI. For more details, refer to Modifying the Change Stream Log Retention Duration.

Step 3: Migrate Your Indexes

Create the same indexes on your Amazon DocumentDB 4.0 cluster that you have on your Amazon DocumentDB 3.6 cluster. Although AWS DMS handles the migration of data, it does not migrate indexes. To migrate the indexes, use the Amazon DocumentDB Index Tool to export indexes from the Amazon DocumentDB 3.6 cluster. You can get the tool by creating a clone of the Amazon DocumentDB tools GitHub repo and following the instructions in README.md. You can run the tool from an Amazon EC2 instance or an AWS Cloud9 environment running in the same Amazon VPC as your Amazon DocumentDB cluster.

The following code dumps indexes from your Amazon DocumentDB v3.6 cluster:

python migrationtools/documentdb_index_tool.py --dump-indexes 
--dir ~/index.js/ 
--host docdb-36-xx.cluster-xxxxxxxx.us-west-2.docdb.amazonaws.com:27017
--tls --tls-ca-file ~/rds-ca-2019-root.pem 
--username user 
--password <password>
2020-02-11 21:51:23,245: Successfully authenticated to database: admin
2020-02-11 21:46:50,432: Successfully connected to instance docdb-36-xx.cluster-xxxxxxxx.us-west-2.docdb.amazonaws.com:27017
2020-02-11 21:46:50,432: Retrieving indexes from server...
2020-02-11 21:46:50,440: Completed writing index metadata to local folder: /home/ec2-user/index.js/

Once your indexes are successfully exported, restore those indexes in your Amazon DocumentDB 4.0 cluster. To restore the indexes that you exported in the preceding step, use the Amazon DocumentDB Index Tool. The following command restores the indexes in your Amazon DocumentDB 4.0 cluster from the specified directory.

python migrationtools/documentdb_index_tool.py --restore-indexes
--dir ~/index.js/ 
--host docdb-40-xx.cluster-xxxxxxxx.us-west-2.docdb.amazonaws.com:27017
--tls --tls-ca-file ~/rds-ca-2019-root.pem 
--username user 
--password <password>
2020-02-11 21:51:23,245: Successfully authenticated to database: admin
2020-02-11 21:51:23,245: Successfully connected to instance docdb-40-xx.cluster-xxxxxxxx.us-west-2.docdb.amazonaws.com:27017
2020-02-11 21:51:23,264: testdb.coll: added index: _id

To confirm that you restored the indexes correctly, connect to your Amazon DocumentDB 4.0 cluster with the mongo shell and list the indexes for a given collection. See the following code:

mongo --ssl 
--host docdb-40-xx.cluster-xxxxxxxx.us-west-2.docdb.amazonaws.com:27017
--sslCAFile rds-ca-2019-root.pem --username documentdb --password documentdb
db.coll.getIndexes()

Step 4: Create a AWS DMS Replication Instance

An AWS DMS replication instance connects and reads data from your source (in this case an Amazon DocumentDB 3.6 cluster) and writes it your target (Amazon DocumentDB 4.0 cluster). The AWS DMS replication instance can perform both bulk load and CDC operations. Most of this processing happen in memory. However, large operations might require some buffering on disk. Cached transactions and log files are also written to disk. Once the data is migrated, the replication instance also streams any change events to make sure the source and target are in sync.

To create an AWS DMS replication instance:

1. Open the AWS DMS console.

2. In the navigation pane, choose Replication instances.

3. Choose Create replication instance and enter the following information:

   • For Name, enter a name of your choice. For example, docdb36todocdb40.

   • For Description, enter a description of your choice. For listitem, Amazon DocumentDB 3.6 to Amazon DocumentDB 4.0 replication instance.

   • For Instance class, choose the size based on your needs.

   • For Engine version, choose 3.4.1.

   • For Amazon VPC, choose the Amazon VPC that houses your Amazon DocumentDB 3.6 and 4.0 clusters.

   • For Allocated storage (GiB), use the default of 50 GiB. If you have a high write throughput workload, increase this value to match your workload.

   • For Multi-AZ, choose Yes if you need high availability and failover support.

   • For Publicly accessible, enable this option.

   

4. Choose Create replication instance.

Step 5: Create an AWS DMS Source Endpoint

The source endpoint is used for the Amazon DocumentDB 3.6 cluster you are looking to upgrade to 4.0.

To create a source endpoint

1. Open the AWS DMS console.

2. In the navigation pane, choose Endpoints.

3. Choose Create endpoint and enter the following information:

   • For Endpoint type, choose Source.

   • >For Endpoint identifier, enter a name that’s easy to remember, for example docdb-source.

   • For Source engine, choose docdb.

   • For Server name, enter the DNS name of your Amazon DocumentDB v3.6 cluster.

   • For Port, enter the port number of your Amazon DocumentDB v3.6 cluster.

   • For SSL mode, choose verify-full.

   • For CA certificate, choose Add new CA certificate. Download the new CA certificate to create TLS connections bundle. For Certificate identifier, enter rds-combined-ca-bundle . For Import certificate file, choose Choose file and navigate to the .pem file that you previously downloaded. Select and open the file. Choose Import certificate, then choose rds-combined-ca-bundle from the Choose a certificate drop down

   • For Username, enter the master username of your Amazon DocumentDB v3.6 cluster.

   • For Password, enter the master password of your Amazon DocumentDB v3.6 cluster.

   • For Database name, enter the database name you are looking to upgrade.

   

4. Test your connection to verify it was successfully setup.

   

5. Choose Create Endpoint.

Note

AWS DMS can only migrate one database at a time.

Step 6: Create an AWS DMS Target Endpoint

The target endpoint is for your Amazon DocumentDB 4.0 cluster.

To create a target endpoint:

1. Open the AWS DMS console.

2. In the navigation pane, choose Endpoints.

3. Choose Create endpoint and enter the following information:

   • For Endpoint type, choose Source.

   • For Endpoint identifier, enter a name that’s easy to remember, for example docdb-source.

   • For Source engine, choose docdb.

   • For Server name, enter the DNS name of your Amazon DocumentDB v4.0 cluster.

   • For Port, enter the port number of your Amazon DocumentDB v4.0 cluster.

   • For SSL mode, choose verify-full.

   • For CA certificate, choose the existing rds-combined-ca-bundle certificate from the Choose a certificate drop down.

   • For Username, enter the master username of your Amazon DocumentDB v4.0 cluster.

   • For Password, enter the master password of your Amazon DocumentDB v4.0 cluster.

   • For Database name, enter the same database name you used to setup your source endpoint.

   

4. Test your connection to verify it was successfully set up.

   

5. Choose Create Endpoint.

Step 7: Create and run a migration task

An AWS DMS task binds the replication instance with your source and target instance. When you create a migration task, you specify the source endpoint, target endpoint, replication instance and any desired migration settings. An AWS DMS task can be created with three different migration types - migrate existing data, migrate existing data and replicate ongoing changes, or replicate data changes only. Since the purpose of this walk through is to upgrade an Amazon DocumentDB 3.6 cluster to Amazon DocumentDB 4.0 with minimal downtime, the steps utilize the option to migrate existing data and replicate ongoing changes. With this option, AWS DMS captures changes while migrating your existing data. AWS DMS continues to capture and apply changes even after the bulk data has been loaded. Eventually the source and target databases will be in sync, allowing for a minimal downtime migration.

Below are the steps to create a migration task for a minimal downtime migration:

1. Open the AWS DMS console.

2. In the navigation pane, choose Tasks.

3. Choose Create task and enter the following information:

   • For Task name, enter a name that’s easy to remember, for example my-dms-upgrade-task.

   • For Replication instance, choose the replication instance that you created in Step3: Create an AWS Database Migration service Replication Instance

   • For Source endpoint, choose the source endpoint that you created in Step 4: Create an AWS Database Migration service Source Endpoint

   • For Target endpoint, choose the target endpoint that you created in Step 5: Create an AWS Database Migration service Target Endpoint

   • For Migration type, choose Migrate existing data and replicate ongoing changes.

   

4. In the Task Settings section, enable CloudWatch logs.

5. For Table mappings section, keep everything at its default setting. This will ensure all collections from your database are migrated.

6. For Migration task startup configuration, choose Automatically on create. This will start the migration task automatically once you create it.

7. Choose Create task.

AWS DMS now begins migrating data from your Amazon DocumentDB 3.6 cluster to your Amazon DocumentDB 4.0 cluster. The task status should change from Starting to Running. You can monitor the progress by choosing Tasks in the AWS DMS console. After several minutes/hours (depending on the size of your migration), the status should change from to Load complete, replication ongoing. This means that AWS DMS has completed a full load migration of your Amazon DocumentDB 3.6 cluster to an Amazon DocumentDB 4.0 cluster and is now replicating change events.

Eventually your source and target will be in sync. You can verify whether they are in sync by running a count() operation on your collections to verify all change events have migrated.

Step 8: Changing the application endpoint to the Amazon DocumentDB cluster 4.0

After the full load is complete and the CDC process is replicating continuously, you are ready to change your application’s database connection endpoint from your Amazon DocumentDB 3.6 cluster to your Amazon DocumentDB cluster 4.0 cluster.