Scalable Deployment

Overview

While a single dsync already packs a powerhouse in single-binary format, it's not perfectly suitable for all use cases. For example:

• Large and Very large database migrations (10TB+)

• Moving or replicating data within a fleet of database instances (data mobility)

For these and other use cases, we have developed a horizontally scalable deployment using Temporal durable execution platform as an external Coordinator and individual dsync's as workers.

Topology

How it works

In this configuration, each worker is a standalone dsync process with its own connectors and transformer instances. A migration or replication job is structured as a Temporal workflow, including Read Planning, Initial Data Copy, and CDC. When a job is submitted to Temporal, individual tasks are allocated to dsync workers based on their suitability and availability, allowing for a high degree of parallelization.

Temporal stores the job metadata and makes task execution durable. Should a worker or a task fail, Temporal will automatically reallocate the task to the next available worker.

Distribution

This configuration is distributed under Commercial License. Contact us at [email protected] if you're interested and want to know more.