Overview
Learn about dsync internals on a high-level
From the user's perspective, we aimed to make an otherwise complicated process as seamless and as easy as possible:
But behind the scenes, dsync features specific machinery and orchestration to make data movement fast and reliable. Below are the key steps, see the Glossary for specific terms definitions:
The Coordinator initiates the data flow by checking Source and Destination connectors' capabilities and negotiating compatible options (such as data format or database-specific functionality)
The Coordinator requests the Source connector to generate the best Read Plan for the initial data copy and create a checkpoint in the CDC stream (T0). The Read Plan consists of Tasks, each of which represents a particular subset of source data. Tasks are effectively units of read parallelization, and they are specific to source database type, since connectors take maximum advantage of native parallel scan features that are offered by source API.
The Source connector initiates the reading into the in-memory data channel using the Read Plan. Tasks are read in parallel according to the level or parallelism set for dsync. When the Read Plan is completed, the Source connector switches to the CDC mode starting to read the change stream from the source at the initial checkpoint (T0) onwards to ensure that no writes have been lost.
The Destination connector is reading the data from the in-memory data channel and writes it to the destination database as fast as possible according to the level or parallelism set for dsync. Each destination connector leverages database-specific optimizations to maximize writes efficiency.
During the process, the Source connector keeps the Coordinator updated on read progress, and uses an in-band signaling mechanism to communicate task boundaries, CDC progress and metadata updates to the Destination connector. Destination connector uses those signals to update the Coordinator on write progress.
Last updated