DynamoDB to Cosmos DB NoSQL

Steps for executing a live production migration from DynamoDB to Cosmos DB NoSQL using Dsynct.

Prerequisites

1. Set up Temporal and SigNoz (or another OTEL collector). You can follow the instructions in Running Dsynct

2. Obtain DynamoDB credentials - you can create a new dedicated user in IAM with AmazonDynamoDBFullAccess permissions, and create security keys for it.

3. Obtain Cosmos DB credentials - URI and Primary Key ("Settings" -> "Keys" in the left menu)

4. Enable Streams in DynamoDB - for the tables being migrated. Use "New image".

Worker VM setup

## Set variables
export SIGNOZ=http://<SIGNOZ_HOST>:4317
export TEMPORAL=<TEMPORAL_HOST>:7233
export AWS_ACCESS_KEY_ID=<...>
export AWS_SECRET_ACCESS_KEY=<...>
export AWS_REGION=<...>
export COSMOS_URI=<...> #e.g. https://cosmos-nosql-west.documents.azure.com:443/
export COSMOS_KEY=<...>

## Create internal network for Docker
docker network create mynet

## Cosmos DB NoSQL Connector
nohup docker run \
--network mynet --name cosmosnosqlconnector \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
markadiom/cosmosnosqlconnector 8089 $COSMOS_URI $COSMOS_KEY 2>&1 > /tmp/java.log &

## Worker
nohup docker run \
--network mynet --name dsyncworker \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
-e AWS_REGION=$AWS_REGION \
markadiom/dsynct worker --flow-name myflow \
--namespace-mapping "dynamo_table:cosmos_db.container,dynamo_table1:cosmos_db.container1" \
--concurrent-activities 4 --sync-writer-workers 8 \
dynamodb \
grpc://cosmosnosqlconnector:8089 --insecure \
temporal --host-port $TEMPORAL \
app --otel 2>&1 > /tmp/dsynct-worker.log &

If data transformations are needed, you can use the embedded YAML transformer.

Running the workflow

## Runner 
docker run \
--name dsyncrunner \
-p 8080:8080 \
-e OTEL_EXPORTER_OTLP_ENDPOINT=$SIGNOZ \
markadiom/dsynct run --flow-name myflow \
--namespace "dynamo_table" --namespace "dynamo_table1" \
temporal --host-port $TEMPORAL \
app --otel --host-port 0.0.0.0:8080