Instructions
Learn how to run Enterprise Dsynct with Docker containers
Core Components
Dsynct is able to leverage an existing Temporal instance, if you're already using it in your organization.
Otherwise, you start the Temporal development server on a VM. It can be reused across different migrations and dsynct executions.
For optimal performance, we recommend provisioning a dedicated disk for the Temporal database with the following configuration or better: 50 GB / 3000 IOPS / 125 MBps.
We recommend provisioning a 4 vCPU, 8 GiB memory VM for Temporal development server.
Dsynct is able to leverage an existing OpenTelemetry gRPC collector, if you're already using it in your organization.
Otherwise, you start the SigNoz observability tool on a VM. It can be reused across different migrations and dsynct executions. For convenience, it can be co-located with Temporal on the same VM.
We recommend provisioning a 2 vCPU, 4 GiB memory VM for SigNoz.
Dsynct is a Temporal-enabled version of Dsync. It only uses compute resources (CPU and RAM), and doesn't store the data on disk.
Dsynct worker is the unit of scale. You can run as many as you need, and add/stop them dynamically even while the migration is running. New workers will automatically pick up tasks from the queue, and any tasks assigned to stopped workers will automatically get reallocated to those still available.
Each Dsynct worker instance (dsynct worker command) can process up to a certain number of tasks in parallel, with the level of parallelism controlled by these parameters:
--concurrent-activities N- max number of concurrent initial sync tasks--sync-transform-workers N- parallelism for the transformer during initial sync--sync-writer-workers N- parallelism for writing to the destination during initial sync--per-stream-workers N- parallelism for writing to the destination during CDC
Dsynct can run on a regular VM or directly as a container. We recommend provisioning at least 4 CPU and 8 GB RAM for each Dsynct worker. The level of parallelism can be adjusted based on the CPUs available, for example for a 4 CPU instance:
--concurrent-activities 4 --sync-transform-workers 4 --sync-writer-workers 8 --per-stream-workers 4
If you are running multiple sets of workers with different configurations (e.g. different source/destination pairs or different transform configs), each set must use a distinct --queue-name. Workers sharing the same queue will receive tasks interchangeably, so mixing different configurations on the same queue will cause errors. The corresponding dsynct run command must also specify the same --queue-name to route work to the correct workers.
The runner (dsynct run command) is responsible for starting and monitoring a migration workflow. For a given flow, it ensures that it has been started on Temporal and servers a Web-based progress dashboard.
The runner can run on a lightweight VM or directly as a container. We recommend provisioning 1 CPU and 2 GB RAM.
Networking
Instructions
Provision VMs or containers. If you're using Azure Marketplace images for migrations to Cosmos DB vCore or Cosmos DB NoSQL, they come with all the software components preinstalled already. You can also use Kubernetes (native, AKS, or others) - see instructions here. We recommend provisioning at least two VMs - one for Temporal and SigNoz, and another one (or more) to run the Runner and Dsynct workers. Ubuntu on x64 is a common choice for VM.
[Optional if using Marketplace images] Install Docker You can follow the official instructions here.
[Optional if using Marketplace images] Install Temporal and SigNoz
[Optional] Get most recent Docker image versions
Start Temporal and SigNoz When placing the Temporal database on a dedicated disk (we recommend doing that), adjust the --db-filename parameter to point at the mount point (e.g. /mnt/data.db) and ensure that the current user has write access to that folder. Note that you need to connect to SigNoz web UI (port 8080) and create an account prior to starting Dsync in order for SigNoz telemetry collector to start.
Start Dsynct worker(s) For testing purposes, you can use
/dev/fakesourceas the source, and/dev/nullas the destination.Start the workflow
Monitor migration progress
The Web Progress dashboard is served on the port
8080on the host where you ran thedsynct runcommandTemporal workflow can be observed by connecting to
<TEMPORAL_HOSTNAME>:8233You can view detailed logs and metrics in SigNoz by connecting to
<SIGNOZ_HOSTNAME>:8080You can import one of our pre-configured dashboards by following the instructions here.
Temporal Tools
Dsynct can run in a temporaltools mode that helps make pausing and unpausing more convenient by setting the environment variable DSYNCT_MODE=temporaltools.
No Temporal
Dsynct can also be run without temporal as a single node. This mode currently only supports a single save file to coordinate (subject to change). To run in this mode, pass set the environment variable DSYNCT_MODE=simple and use the sync command.
Example command that uses resume.file to save state:
The command has parameters that somewhat combine the options of the run and worker command of dsynct into one. You can run docker run -e 'DSYNCT_MODE=simple' markadiom/dsynct sync --help to see all available options.
testsync
The testsync command fetches specific documents by ID from the source, optionally transforms them, and writes them to the destination. It is useful for testing transformations or connectivity without running a full sync. It only works with connectors that support retrieving single entries via GetByIds (currently MongoDB and Cosmos DB NoSQL connectors).
To test with a transformation:
--namespace
Yes
The source namespace to fetch from.
--id
No
Document ID (string). Can be specified multiple times for multiple documents. For composite keys, use --id-size to indicate how many --id values form a single key.
--jsonext-id
No
Document ID in extended JSON format, for non-string types (e.g. {"_id": {"$oid": "..."}}).
--id-file
No
Path to a file containing extended JSON IDs, one per line. Compatible with sample-ids output.
--id-size
No
Number of --id entries that form a single composite key. Default: 1.
--transform
No
Set if a transformer is provided as the third argument after source and destination.
--src-data-type
No
Source data type. Inferred if not set.
--dst-data-type
No
Destination data type. Inferred if not set.
--namespace-mapping
No
Namespace mapping from source to destination (e.g. srcns:dstns).
--mapping-delimiter
No
Delimiter for namespace mappings and ID key-value pairs. Default: :.
At least one of --id, --jsonext-id, or --id-file must be provided.
Last updated