Running Dsynct
Learn how to run Enterpise Dsynct with Docker containers
Core Components
Dsynct is able to leverage an existing Temporal instance, if you're already using it in your organization.
Otherwise, you start the Temporal development server on a VM. It can be reused across different migrations and dsynct executions.
For optimal performance, we recommend provisioning a dedicated disk for the Temporal database with the following configuration or better: 50 GB / 3000 IOPS / 125 MBps.
We recommend provisioning a 4 vCPU, 8 GiB memory VM for Temporal development server.
Dsynct is able to leverage an existing OpenTelemetry gRPC collector, if you're already using it in your organization.
Otherwise, you start the SigNoz observability tool on a VM. It can be reused across different migrations and dsynct executions. For convenience, it can be co-located with Temporal on the same VM.
We recommend provisioning a 2 vCPU, 4 GiB memory VM for SigNoz.
Dsynct is a Temporal-enabled version of Dsync. It only uses compute resources (CPU and RAM), and doesn't store the data on disk.
Each Dsynct worker instance (dsynct worker command) can process up to a certain number of tasks in parallel, with the level of parallelism controlled by these parameters:
--concurrent-activities N- max number of concurrent initial sync tasks--sync-transform-workers N- parallelism for the transformer during initial sync--sync-writer-workers N- parallelism for writing to the destination during initial sync--per-stream-workers N- parallelism for writing to the destination during CDC
Dsynct can run on a regular VM or directly as a container. We recommend provisioning at least 4 CPU and 8 GB RAM for each Dsynct worker. The level of parallelism can be adjusted based on the CPUs available, for example for a 4 CPU instance:
--concurrent-activities 4 --sync-transform-workers 4 --sync-writer-workers 8 --per-stream-workers 4
The runner (dsynct run command) is responsible for starting and monitoring a migration workflow. For a given flow, it ensures that it has been started on Temporal and servers a Web-based progress dashboard.
The runner can run on a lightweight VM or directly as a container. We recommend provisioning 1 CPU and 2 GB RAM.
Networking
Instructions
Provision VMs or containers. If you're using Azure Marketplace images for migrations to Cosmos DB vCore or Cosmos DB NoSQL, they come with all the software components preinstalled already. You can also use Kubernetes (native, AKS, or others) - see instructions here. We recommend provisioning at least two VMs - one for Temporal and SigNoz, and another one (or more) to run the Runner and Dsynct workers. Ubuntu on x64 is a common choice for VM.
[Optional if using Marketplace images] Install Docker You can follow the official instructions here.
[Optional if using Marketplace images] Install Temporal and SigNoz
brew install temporal git clone -b main https://github.com/SigNoz/signoz.git[Optional] Get most recent Docker image versions
sudo docker pull markadiom/dsynctStart Temporal and SigNoz When placing the Temporal database on a dedicated disk (we recommend doing that), adjust the --db-filename parameter to point at the mount point (e.g. /mnt/data.db) and ensure that the current user has write access to that folder.
screen -dmS Temporal temporal server start-dev --db-filename data.db --ip 0.0.0.0 --dynamic-config-value limit.numPendingActivities.error=10000 --dynamic-config-value frontend.activityAPIsEnabled=true cd signoz/deploy/docker sudo docker compose up -d --remove-orphansStart Dsynct worker(s) For testing purposes, you can use
/dev/fakesourceas the source, and/dev/nullas the destination.nohup docker run --name dsyncworker \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \ markadiom/dsynct worker --flow-name <FLOW_NAME> \ <OPTIONAL PARAMETERS> \ <SOURCE> \ <DESTINATION> temporal --host-port <TEMPORAL_HOSTNAME>:7233 \ app --otel 2>&1 > /tmp/dsynct-worker.log &Start the workflow
docker run --name dsyncrunner \ -p 8080:8080 \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \ markadiom/dsynct run --flow-name <FLOW_NAME> \ temporal --host-port <TEMPORAL_HOSTNAME>:7233 \ app --otel --host-port 0.0.0.0:8080Monitor migration progress
The Web Progress dashboard is served on the port
8080on the host where you ran thedsynct runcommandTemporal workflow can be observed by connecting to
<TEMPORAL_HOSTNAME>:8233You can view detailed logs and metrics in SigNoz by connecting to
<SIGNOZ_HOSTNAME>:8080You can import one of our pre-configured dashboards by following the instructions here.
Last updated