Running Dsynct

Learn how to run Enterpise Dsynct with Docker containers

Core Components

Temporal

Dsynct is able to leverage an existing Temporal instance, if you're already using it in your organization.

Otherwise, you start the Temporal development server on a VM. It can be reused across different migrations and dsynct executions.

For optimal performance, we recommend provisioning a dedicated disk for the Temporal database with the following configuration or better: 50 GB / 3000 IOPS / 125 MBps.

We recommend provisioning a 4 vCPU, 8 GiB memory VM for Temporal development server.

OpenTelemetry

Dsynct is able to leverage an existing OpenTelemetry gRPC collector, if you're already using it in your organization.

Otherwise, you start the SigNoz observability tool on a VM. It can be reused across different migrations and dsynct executions. For convenience, it can be co-located with Temporal on the same VM.

We recommend provisioning a 2 vCPU, 4 GiB memory VM for SigNoz.

Dsynct Worker

Dsynct is a Temporal-enabled version of Dsync. It only uses compute resources (CPU and RAM), and doesn't store the data on disk.

Dsynct worker is the unit of scale. You can run as many as you need, and add/stop them dynamically even while the migration is running. New workers will automatically pick up tasks from the queue, and any tasks assigned to stopped workers will automatically get reallocated to those still available.

Each Dsynct worker instance (dsynct worker command) can process up to a certain number of tasks in parallel, with the level of parallelism controlled by these parameters:

• --concurrent-activities N - max number of concurrent initial sync tasks

• --sync-transform-workers N - parallelism for the transformer during initial sync

• --sync-writer-workers N - parallelism for writing to the destination during initial sync

• --per-stream-workers N - parallelism for writing to the destination during CDC

Dsynct can run on a regular VM or directly as a container. We recommend provisioning at least 4 CPU and 8 GB RAM for each Dsynct worker. The level of parallelism can be adjusted based on the CPUs available, for example for a 4 CPU instance:

--concurrent-activities 4 --sync-transform-workers 4 --sync-writer-workers 8 --per-stream-workers 4

Dsynct Runner

The runner (dsynct run command) is responsible for starting and monitoring a migration workflow. For a given flow, it ensures that it has been started on Temporal and servers a Web-based progress dashboard.

The runner can run on a lightweight VM or directly as a container. We recommend provisioning 1 CPU and 2 GB RAM.

Networking

Instructions

1. Provision VMs or containers. If you're using Azure Marketplace images for migrations to Cosmos DB vCore or Cosmos DB NoSQL, they come with all the software components preinstalled already. You can also use Kubernetes (native, AKS, or others) - see instructions here. We recommend provisioning at least two VMs - one for Temporal and SigNoz, and another one (or more) to run the Runner and Dsynct workers. Ubuntu on x64 is a common choice for VM.

2. [Optional if using Marketplace images] Install Docker You can follow the official instructions here.

3. [Optional if using Marketplace images] Install Temporal and SigNoz

   brew install temporal
   git clone -b main https://github.com/SigNoz/signoz.git

4. [Optional] Get most recent Docker image versions

   sudo docker pull markadiom/dsynct

5. Start Temporal and SigNoz When placing the Temporal database on a dedicated disk (we recommend doing that), adjust the --db-filename parameter to point at the mount point (e.g. /mnt/data.db) and ensure that the current user has write access to that folder.

   screen -dmS Temporal temporal server start-dev --db-filename data.db --ip 0.0.0.0 --dynamic-config-value limit.numPendingActivities.error=10000 --dynamic-config-value frontend.activityAPIsEnabled=true
   cd signoz/deploy/docker
   sudo docker compose up -d --remove-orphans

6. Start Dsynct worker(s) For testing purposes, you can use /dev/fakesource as the source, and /dev/null as the destination.

   nohup docker run --name dsyncworker \
   -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
   markadiom/dsynct worker --flow-name <FLOW_NAME> \
   <OPTIONAL PARAMETERS> \
   <SOURCE> \
   <DESTINATION> 
   temporal --host-port <TEMPORAL_HOSTNAME>:7233 \
   app --otel 2>&1 > /tmp/dsynct-worker.log &

7. Start the workflow

   docker run --name dsyncrunner \
   -p 8080:8080 \
   -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
   markadiom/dsynct run --flow-name <FLOW_NAME> \
   temporal --host-port <TEMPORAL_HOSTNAME>:7233 \
   app --otel --host-port 0.0.0.0:8080

8. Monitor migration progress

   1. The Web Progress dashboard is served on the port 8080 on the host where you ran the dsynct run command

   2. Temporal workflow can be observed by connecting to <TEMPORAL_HOSTNAME>:8233

   3. You can view detailed logs and metrics in SigNoz by connecting to <SIGNOZ_HOSTNAME>:8080 You can import one of our pre-configured dashboards by following the instructions here.