DynamoDB to Cosmos DB NoSQL
Steps for executing a live production migration from DynamoDB to Cosmos DB NoSQL using Dsynct.
Prerequisites
Set up Temporal and SigNoz (or another OTEL collector). You can follow the instructions inRunning Dsynct
Obtain DynamoDB credentials - you can create a new dedicated user in IAM with AmazonDynamoDBFullAccess permissions, and create security keys for it.
Obtain Cosmos DB credentials - URI and Primary Key ("Settings" -> "Keys" in the left menu)
Enable Streams in DynamoDB - for the tables being migrated. Use "New image".
Worker VM setup
## Set variables
export SIGNOZ=http://<SIGNOZ_HOST>:4317
export TEMPORAL=<TEMPORAL_HOST>:7233
export AWS_ACCESS_KEY_ID=<...>
export AWS_SECRET_ACCESS_KEY=<...>
export AWS_REGION=<...>
export COSMOS_URI=<...> #e.g. https://cosmos-nosql-west.documents.azure.com:443/
export COSMOS_KEY=<...>
## Create internal network for Docker
docker network create mynet
## Cosmos DB NoSQL Connector
nohup docker run \
--network mynet --name cosmosnosqlconnector \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
markadiom/cosmosnosqlconnector 8089 $COSMOS_URI $COSMOS_KEY 2>&1 > /tmp/java.log &
## Worker
nohup docker run \
--network mynet --name dsyncworker \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e AWS_REGION=$AWS_REGION \
markadiom/dsynct worker --flow-name myflow \
--namespace-mapping "dynamo_table:cosmos_db.container,dynamo_table1:cosmos_db.container1" \
--concurrent-activities 4 --sync-writer-workers 8 \
dynamodb \
grpc://cosmosnosqlconnector:8089 --insecure \
temporal --host-port $TEMPORAL \
app --otel 2>&1 > /tmp/dsynct-worker.log &
Running the workflow
## Runner
docker run \
--name dsyncrunner \
-p 8080:8080 \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
markadiom/dsynct run --flow-name myflow \
--namespace "dynamo_table" --namespace "dynamo_table1" \
temporal --host-port $TEMPORAL \
app --otel --host-port 0.0.0.0:8080
Last updated