Scalable Deployment

Overview

While a single dsync already packs a powerhouse in single-binary format, it's not perfectly suitable for all use cases. For example:

• Large and Very large database migrations (100GB to 100TB and beyond)

• Moving or replicating data within a fleet of database instances (data mobility)

For these and other use cases, we have developed a horizontally scalable deployment using Temporal durable execution platform as an external Coordinator and individual dsync's as workers.

This Enterprise deployment offers scalability, flexibility, and additional observability on top of Open Source dsync.

Topology

Components

1. Temporal durable workflow execution engine

   • Coordinates workflows and assigns tasks to workers

   • Handles resumability and retries on the task level, maintains state

2. SigNoz observability tool

   • Logs and metrics collector (OpenTelemetry)

   • Advanced dashboards

3. Dsynct

   • Worker processes: dsynct worker

   • Runner: dsynct run

     • Initiates a new workflow (or connects to existing one)

     • Serves real-time web progress dashboard over HTTP

How it works

In this configuration, each worker is a standalone dsync process with its own connectors and transformer instances. A migration or replication job is structured as a Temporal workflow, including tasks for Read Planning, Initial Data Copy, and CDC. When a job is submitted to Temporal, individual tasks are allocated to dsync workers based on their suitability and availability, allowing for a high degree of parallelization.

Temporal stores the job metadata and makes task execution durable. Should a worker or a task fail, Temporal will automatically reallocate the task to the next available worker.

Distribution

The Enterprise version of dsync is distributed under Commercial License that allows for a free trial period.

To download and run this configuration, you can use docker containers or Kubernetes charts: https://github.com/adiom-data/public.

We also have modified versions of Kubernetes charts available for Azure, Google, and AWS services.

Contact us at [email protected] if you're interested and want to know more.