> For the complete documentation index, see [llms.txt](https://docs.adiom.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.adiom.io/enterprise/running-dsynct/running-dsynct.md). # Instructions Learn how to run Enterprise Dsynct with Docker containers ## Core Components {% tabs %} {% tab title="Temporal" %} Dsynct is able to leverage an existing [Temporal](https://temporal.io/) instance, if you're already using it in your organization. Otherwise, you start the Temporal development server on a VM. It can be reused across different migrations and dsynct executions. For optimal performance, we recommend provisioning a dedicated disk for the Temporal database with the following configuration or better: 50 GB / 3000 IOPS / 125 MBps. We recommend provisioning a 4 vCPU, 8 GiB memory VM for Temporal development server. {% endtab %} {% tab title="OpenTelemetry" %} Dsynct is able to leverage an existing OpenTelemetry gRPC collector, if you're already using it in your organization. Otherwise, you start the [SigNoz](https://signoz.io/) observability tool on a VM. It can be reused across different migrations and dsynct executions. For convenience, it can be co-located with Temporal on the same VM. We recommend provisioning a 2 vCPU, 4 GiB memory VM for SigNoz. {% endtab %} {% tab title="Dsynct Worker" %} Dsync**t** is a Temporal-enabled version of Dsync. It only uses compute resources (CPU and RAM), and doesn't store the data on disk. {% hint style="info" %} Dsynct worker is the unit of scale. You can run as many as you need, and add/stop them dynamically even while the migration is running. New workers will automatically pick up tasks from the queue, and any tasks assigned to stopped workers will automatically get reallocated to those still available. {% endhint %} Each Dsynct worker instance (`dsynct worker` command) can process up to a certain number of tasks in parallel, with the level of parallelism controlled by these parameters: * `--concurrent-activities N` - max number of concurrent initial sync tasks * `--sync-transform-workers N` - parallelism for the transformer during initial sync * `--sync-writer-workers N` - parallelism for writing to the destination during initial sync * `--per-stream-workers N` - parallelism for writing to the destination during CDC Dsynct can run on a regular VM or directly as a container. We recommend provisioning at least 4 CPU and 8 GB RAM for each Dsynct worker. The level of parallelism can be adjusted based on the CPUs available, for example for a 4 CPU instance: `--concurrent-activities 4 --sync-transform-workers 4 --sync-writer-workers 8 --per-stream-workers 4` {% hint style="warning" %} If you are running multiple sets of workers with different configurations (e.g. different source/destination pairs or different transform configs), each set must use a distinct `--queue-name`. Workers sharing the same queue will receive tasks interchangeably, so mixing different configurations on the same queue will cause errors. The corresponding `dsynct run` command must also specify the same `--queue-name` to route work to the correct workers. {% endhint %} {% endtab %} {% tab title="Dsynct Runner" %} The runner (`dsynct run` command) is responsible for starting and monitoring a migration workflow. For a given flow, it ensures that it has been started on Temporal and servers a Web-based progress dashboard. The runner can run on a lightweight VM or directly as a container. We recommend provisioning 1 CPU and 2 GB RAM. {% endtab %} {% endtabs %} ## Networking
## Instructions 1. **Provision VMs or containers.** If you're using Azure Marketplace images for migrations to [Cosmos DB vCore](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/adiom.adiom_dsync_vm?tab=Overview) or [Cosmos DB NoSQL](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/adiom.adiom_dsync_vm_dynamo?tab=Overview), they come with all the software components preinstalled already.\ \ You can also use Kubernetes (native, AKS, or others) - see instructions [here](https://github.com/adiom-data/public/tree/main/kubernetes).\ \ We recommend provisioning at least two VMs - one for Temporal and SigNoz, and another one (or more) to run the Runner and Dsynct workers. Ubuntu on x64 is a common choice for VM. 2. \[Optional if using Marketplace images] **Install Docker**\ You can follow the official instructions [here](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository). 3. \[Optional if using Marketplace images] **Install Temporal and SigNoz** ```bash brew install temporal git clone -b main https://github.com/SigNoz/signoz.git ``` 4. \[Optional] Get most recent Docker image versions ```bash sudo docker pull markadiom/dsynct ``` 5. **Start Temporal and SigNoz**\ \&#xNAN;*When placing the Temporal database on a dedicated disk (we recommend doing that), adjust the --db-filename parameter to point at the mount point (e.g. /mnt/data.db) and ensure that the current user has write access to that folder.*\ \&#xNAN;*Note that you need to connect to SigNoz web UI (port 8080) and create an account prior to starting Dsync in order for SigNoz telemetry collector to start.* ```bash screen -dmS Temporal temporal server start-dev --db-filename data.db --ip 0.0.0.0 --dynamic-config-value limit.numPendingActivities.error=10000 --dynamic-config-value frontend.activityAPIsEnabled=true cd signoz/deploy/docker sudo docker compose up -d --remove-orphans ``` 6. **Start Dsynct worker(s)**\ For testing purposes, you can use `/dev/fakesource` as the source, and `/dev/null` as the destination. ```bash docker run -d --name dsyncworker \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://:4317' \ markadiom/dsynct worker \ \ \ temporal --host-port :7233 \ app --otel ``` 7. **Start the workflow** ```bash docker run --name dsyncrunner \ -p 8080:8080 \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://:4317' \ markadiom/dsynct run \ temporal --host-port :7233 \ app --otel --host-port 0.0.0.0:8080 ``` 8. **Monitor migration progress** 1. The Web Progress dashboard is served on the port `8080` on the host where you ran the `dsynct run` command 2. Temporal workflow can be observed by connecting to `:8233` 3. Container logs can be viewed with `docker logs -f dsyncworker` or `docker logs -f dsyncrunner` 4. You can view detailed logs and metrics in SigNoz by connecting to `:8080` You can import one of our pre-configured [dashboards](https://github.com/adiom-data/public/tree/main/kubernetes/system/signoz_dashboards) by following the instructions [here](https://signoz.io/docs/dashboards/import-dashboard/). ### Temporal Tools Dsynct can run in a `temporaltools` mode that helps make pausing and unpausing more convenient by setting the environment variable DSYNCT\_MODE=temporaltools. ``` docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct --help docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct pause --workflow-id= --dry-run docker run -e 'DSYNCT_MODE=temporaltools' markadiom/dsynct unpause --workflow-id= --dry-run ``` ## No Temporal Dsynct can also be run without temporal as a single node. This mode currently only supports a single save file to coordinate (subject to change). To run in this mode, pass set the environment variable DSYNCT\_MODE=simple and use the `sync` command. Example command that uses `resume.file` to save state: ```bash docker run -d --name dsync \ -e 'DSYNCT_MODE=simple' \ -p 8080:8080 \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://:4317' \ markadiom/dsynct \ --host-port=0.0.0.0:8080 \ --otel \ sync \ --save-file resume.file \ \ \ ``` The command has parameters that somewhat combine the options of the `run` and `worker` command of dsynct into one. You can run `docker run -e 'DSYNCT_MODE=simple' markadiom/dsynct sync --help` to see all available options. ### testsync The `testsync` command fetches specific documents by ID from the source, optionally transforms them, and writes them to the destination. It is useful for testing transformations or connectivity without running a full sync. It only works with connectors that support retrieving single entries via `GetByIds` (currently MongoDB and Cosmos DB NoSQL connectors). ```bash docker run -e 'DSYNCT_MODE=simple' \ markadiom/dsynct testsync \ --namespace \ --id \ ``` To test with a transformation: ```bash docker run -e 'DSYNCT_MODE=simple' \ -v "./transform.yaml:/transform.yaml" \ markadiom/dsynct testsync \ --namespace \ --id \ --transform \ dsync-transform://transform.yaml ``` | Flag | Required | Description | | --------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--namespace` | Yes | The source namespace to fetch from. | | `--id` | No | Document ID (string). Can be specified multiple times for multiple documents. For composite keys, use `--id-size` to indicate how many `--id` values form a single key. | | `--jsonext-id` | No | Document ID in extended JSON format, for non-string types (e.g. `{"_id": {"$oid": "..."}}`). | | `--id-file` | No | Path to a file containing extended JSON IDs, one per line. Compatible with `sample-ids` output. | | `--id-size` | No | Number of `--id` entries that form a single composite key. Default: `1`. | | `--transform` | No | Set if a transformer is provided as the third argument after source and destination. | | `--src-data-type` | No | Source data type. Inferred if not set. | | `--dst-data-type` | No | Destination data type. Inferred if not set. | | `--namespace-mapping` | No | Namespace mapping from source to destination (e.g. `srcns:dstns`). | | `--mapping-delimiter` | No | Delimiter for namespace mappings and ID key-value pairs. Default: `:`. | At least one of `--id`, `--jsonext-id`, or `--id-file` must be provided.