Instructions

Learn how to run Enterprise Dsynct with Docker containers

Core Components

Temporal

Dsynct is able to leverage an existing Temporal instance, if you're already using it in your organization.

Otherwise, you start the Temporal development server on a VM. It can be reused across different migrations and dsynct executions.

For optimal performance, we recommend provisioning a dedicated disk for the Temporal database with the following configuration or better: 50 GB / 3000 IOPS / 125 MBps.

We recommend provisioning a 4 vCPU, 8 GiB memory VM for Temporal development server.

OpenTelemetry

Dsynct is able to leverage an existing OpenTelemetry gRPC collector, if you're already using it in your organization.

Otherwise, you start the SigNoz observability tool on a VM. It can be reused across different migrations and dsynct executions. For convenience, it can be co-located with Temporal on the same VM.

We recommend provisioning a 2 vCPU, 4 GiB memory VM for SigNoz.

Dsynct Worker

Dsynct is a Temporal-enabled version of Dsync. It only uses compute resources (CPU and RAM), and doesn't store the data on disk.

Dsynct worker is the unit of scale. You can run as many as you need, and add/stop them dynamically even while the migration is running. New workers will automatically pick up tasks from the queue, and any tasks assigned to stopped workers will automatically get reallocated to those still available.

Each Dsynct worker instance (dsynct worker command) can process up to a certain number of tasks in parallel, with the level of parallelism controlled by these parameters:

• --concurrent-activities N - max number of concurrent initial sync tasks

• --sync-transform-workers N - parallelism for the transformer during initial sync

• --sync-writer-workers N - parallelism for writing to the destination during initial sync

• --per-stream-workers N - parallelism for writing to the destination during CDC

Dsynct can run on a regular VM or directly as a container. We recommend provisioning at least 4 CPU and 8 GB RAM for each Dsynct worker. The level of parallelism can be adjusted based on the CPUs available, for example for a 4 CPU instance:

--concurrent-activities 4 --sync-transform-workers 4 --sync-writer-workers 8 --per-stream-workers 4

If you are running multiple sets of workers with different configurations (e.g. different source/destination pairs or different transform configs), each set must use a distinct --queue-name. Workers sharing the same queue will receive tasks interchangeably, so mixing different configurations on the same queue will cause errors. The corresponding dsynct run command must also specify the same --queue-name to route work to the correct workers.

Dsynct Runner

The runner (dsynct run command) is responsible for starting and monitoring a migration workflow. For a given flow, it ensures that it has been started on Temporal and servers a Web-based progress dashboard.

The runner can run on a lightweight VM or directly as a container. We recommend provisioning 1 CPU and 2 GB RAM.

Networking

Instructions

1. Provision VMs or containers. If you're using Azure Marketplace images for migrations to Cosmos DB vCore or Cosmos DB NoSQL, they come with all the software components preinstalled already. You can also use Kubernetes (native, AKS, or others) - see instructions here. We recommend provisioning at least two VMs - one for Temporal and SigNoz, and another one (or more) to run the Runner and Dsynct workers. Ubuntu on x64 is a common choice for VM.

2. [Optional if using Marketplace images] Install Docker You can follow the official instructions here.

3. [Optional if using Marketplace images] Install Temporal and SigNoz

   brew install temporal
   git clone -b main https://github.com/SigNoz/signoz.git

4. [Optional] Get most recent Docker image versions

   sudo docker pull markadiom/dsynct

5. Start Temporal and SigNoz &#xNAN;When placing the Temporal database on a dedicated disk (we recommend doing that), adjust the --db-filename parameter to point at the mount point (e.g. /mnt/data.db) and ensure that the current user has write access to that folder. &#xNAN;Note that you need to connect to SigNoz web UI (port 8080) and create an account prior to starting Dsync in order for SigNoz telemetry collector to start.

   screen -dmS Temporal temporal server start-dev --db-filename data.db --ip 0.0.0.0 --dynamic-config-value limit.numPendingActivities.error=10000 --dynamic-config-value frontend.activityAPIsEnabled=true
   cd signoz/deploy/docker
   sudo docker compose up -d --remove-orphans

6. Start Dsynct worker(s) For testing purposes, you can use /dev/fakesource as the source, and /dev/null as the destination.

   docker run -d --name dsyncworker \
   -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
   markadiom/dsynct worker \
   <OPTIONAL PARAMETERS> \
   <SOURCE> \
   <DESTINATION> 
   temporal --host-port <TEMPORAL_HOSTNAME>:7233 \
   app --otel

7. Start the workflow

   docker run --name dsyncrunner \
   -p 8080:8080 \
   -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
   markadiom/dsynct run \
   temporal --host-port <TEMPORAL_HOSTNAME>:7233 \
   app --otel --host-port 0.0.0.0:8080

8. Monitor migration progress

   1. The Web Progress dashboard is served on the port 8080 on the host where you ran the dsynct run command

   2. Temporal workflow can be observed by connecting to <TEMPORAL_HOSTNAME>:8233

   3. Container logs can be viewed with docker logs -f dsyncworker or docker logs -f dsyncrunner

   4. You can view detailed logs and metrics in SigNoz by connecting to <SIGNOZ_HOSTNAME>:8080 You can import one of our pre-configured dashboards by following the instructions here.

Temporal Tools

Dsynct can run in a temporaltools mode that helps make pausing and unpausing more convenient by setting the environment variable DSYNCT_MODE=temporaltools.

docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct --help

docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct pause --workflow-id=<workflow-id> --dry-run

docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct unpause --workflow-id=<workflow-id> --dry-run

No Temporal

Dsynct can also be run without temporal as a single node. This mode currently only supports a single save file to coordinate (subject to change). To run in this mode, pass set the environment variable DSYNCT_MODE=simple and use the sync command.

Example command that uses resume.file to save state:

docker run -d --name dsync \
-e 'DSYNCT_MODE=simple' \
-p 8080:8080 \
-e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
markadiom/dsynct \
--host-port=0.0.0.0:8080 \
--otel \
sync \
--save-file resume.file \
<OPTIONAL PARAMETERS> \
<SOURCE> \
<DESTINATION>

The command has parameters that somewhat combine the options of the run and worker command of dsynct into one. You can run docker run -e 'DSYNCT_MODE=simple' markadiom/dsynct sync --help to see all available options.

testsync

The testsync command fetches specific documents by ID from the source, optionally transforms them, and writes them to the destination. It is useful for testing transformations or connectivity without running a full sync. It only works with connectors that support retrieving single entries via GetByIds (currently MongoDB and Cosmos DB NoSQL connectors).

docker run -e 'DSYNCT_MODE=simple' \
markadiom/dsynct testsync \
--namespace <SOURCE_NAMESPACE> \
--id <DOCUMENT_ID> \
<SOURCE> <DESTINATION>

To test with a transformation:

docker run -e 'DSYNCT_MODE=simple' \
-v "./transform.yaml:/transform.yaml" \
markadiom/dsynct testsync \
--namespace <SOURCE_NAMESPACE> \
--id <DOCUMENT_ID> \
--transform \
<SOURCE> <DESTINATION> dsync-transform://transform.yaml

Flag	Required	Description
--namespace	Yes	The source namespace to fetch from.
--id	No	Document ID (string). Can be specified multiple times for multiple documents. For composite keys, use --id-size to indicate how many --id values form a single key.
--jsonext-id	No	Document ID in extended JSON format, for non-string types (e.g. {"_id": {"$oid": "..."}}).
--id-file	No	Path to a file containing extended JSON IDs, one per line. Compatible with sample-ids output.
--id-size	No	Number of --id entries that form a single composite key. Default: 1.
--transform	No	Set if a transformer is provided as the third argument after source and destination.
--src-data-type	No	Source data type. Inferred if not set.
--dst-data-type	No	Destination data type. Inferred if not set.
--namespace-mapping	No	Namespace mapping from source to destination (e.g. srcns:dstns).
--mapping-delimiter	No	Delimiter for namespace mappings and ID key-value pairs. Default: :.

At least one of --id, --jsonext-id, or --id-file must be provided.