Web-API
The web-api command starts an HTTP server that serves both a gRPC API and a single-page web UI for managing dsynct. Whereas the dsynct run command is scoped to a single migration workflow, the web-api is intended as a longer-lived control plane: it can list and start syncs, manage Temporal flows, register dynamic workers, and run interactive testing tools.
To start the web-api:
docker run --name dsyncwebapi \
-p 8080:8080 \
-e 'DSYNCT_MODE=simple' \
-e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://<SIGNOZ_HOSTNAME>:4317' \
markadiom/dsynct \
--host-port 0.0.0.0:8080 \
--otel \
web-api \
--temporal-host-port <TEMPORAL_HOSTNAME>:7233The relevant flags for the web-api subcommand are:
--temporal-host-port
Address of the Temporal frontend, e.g. temporal:7233. Required unless --simple-only is set.
--temporal-namespace
Temporal namespace to use. Default: default.
--simple-only
Run without Temporal. Only Local Syncs and Tools are available; the Flows and Workers sections are hidden.
--cert-file / --key-file
TLS certificate and key for serving HTTPS.
--no-gzip
Disable gzip compression on the API.
Open the served address in a browser. The left-hand drawer contains four sections: Progress, Syncs, Workers, and Tools.
Progress
The Progress section shows live progress for any sync that the web-api is currently tracking. This includes both Local Syncs started from the Syncs tab and Temporal flows that have been added in the Flows tab.
Use the dropdown at the top of the page to switch between tracked flows. For each selected flow the page shows:
A summary card with the workflow ID, current state, and (where applicable) a button to terminate the workflow or cancel the sync.
The latest activity errors, if any.
An Initial Sync card with per-namespace progress and controls to pause/unpause initial sync activities (overall or for a specific namespace).
A Change Stream card with controls to pause/unpause change stream and read-ahead activities.
An Operation Log that records the result of pause/unpause/terminate actions issued from the page.
For local (non-Temporal) syncs, pause/unpause/terminate controls are not shown; only the cancel-sync button is available.
Syncs
The Syncs section is where you configure which syncs are tracked in the Progress view. It has two tabs:
Local Syncs
Local Syncs are non-Temporal syncs that run directly inside the web-api server process. They are intended for testing and for smaller projects where standing up a Temporal cluster would be overkill. Each local sync uses the same underlying pipeline as a worker, but progress is tracked in-memory by the web-api.
To start a local sync:
Type a connector argument string into the Run New Sync input, for example
/dev/fakesource /dev/null. Quoting follows shell-style rules.Click Run to start the sync, or Form to open a guided form that exposes all
syncflags and connector options.
Started syncs are listed below with their current state (Starting, Running, Done, Error, or Cancelled) and the original arguments. Each row offers:
View -- jump to the Progress section for that sync.
Cancel -- cancel a running sync.
Remove -- remove a finished or errored sync from the list.
Flows
The Flows tab is the interface to Temporal. From here you can either start a brand-new flow or attach to one that already exists in Temporal:
Run -- type or build a
dsynct run-style argument string (e.g.--workflow-id myflow --queue-name dsync ...) and submit. The web-api will start the workflow in Temporal and begin tracking it. The Form button opens a guided form for the same set of flags.Add Workflow -- attach to an existing Temporal workflow. Click List Workflow IDs to fetch known workflow IDs from Temporal, then click Add next to any of them to start tracking it. The Check Interval and Publish Interval control how often the web-api polls Temporal for state and pushes updates to the UI.
Active Flows -- the list of currently tracked flows. Use Remove to stop tracking a flow (this does not affect the workflow on Temporal itself).
Both Local Syncs and Flows feed into the Progress dropdown, so any sync started here can be monitored from the Progress section.
Workers
The Workers section is only shown when the web-api is running with Temporal (i.e. without --simple-only). It is used to control dynamic workers.
Dynamic workers are dsynct worker processes started by long-running node processes that register themselves with the web-api over a coordinator API. Once registered, the web-api can ask a node to start or stop a worker using the same arguments you would pass to dsynct worker. This makes it possible to add or remove worker capacity from the UI without restarting any containers.
Starting a node
A node is started with the node command (run via the simple-mode entrypoint). Each node needs to know the Temporal address and the URL of the web-api that acts as its coordinator:
The relevant flags are:
--temporal-host-port
Address of the Temporal frontend.
--temporal-namespace
Temporal namespace to use. Default: default.
--coordinator-url
URL of the web-api that this node should poll for commands. If unset, the node still serves its node API directly but will not auto-register with a coordinator.
--poll-interval
How often the node polls the coordinator for new commands. Default: 10s.
--heartbeat-timeout
The coordinator marks the node dead if it does not poll within this duration. Default: 60s.
Run as many node processes as you need; each one becomes a separate row in the Workers section once it polls the coordinator.
Managing workers from the UI
Once at least one node is registered, the Workers section displays:
Add Worker to All Nodes -- type or build a
dsynct worker-style argument string (e.g.--queue-name myq /dev/fakesource /dev/null) and click Add to All to start an identical worker on every registered node. The Form button opens a guided form with allworkerflags. Remove All Workers stops every managed worker on every node.Nodes -- one block per registered node, showing the node ID and last-seen timestamp. Inside each block:
A per-node argument input to start a worker only on that node.
A list of managed workers with their state (
Starting,Running,Stopping,Stopped,Error), the original arguments, Remove to stop just that worker, and Remove All Like This to stop every worker (across all nodes) that was started with the same arguments.Pending commands (add/remove) show their state (
pending,acked,error) until they are processed by the node.
Workers stopped by the coordinator continue to be reflected in the UI until they actually terminate, so there can be a short delay between clicking Remove and the row disappearing.
As with any other dsynct worker, every worker on the same --queue-name must be configured identically (same source/destination/transformer). Mixing configurations on the same queue will cause errors.
Tools
The Tools section provides utilities for testing transformations and syncs without running a full job. There are two tabs.
Transform Studio
Transform Studio is an interactive way to iterate on a dsync-transform YAML config against sample documents. It is fully local to the web-api process -- nothing is written to any source or destination.
Open Tools > Transform Studio and:
Paste your transform config into the YML Config field.
Set the Namespace to the source namespace the document belongs to.
Choose the Src Type and Dst Type (
DATA_TYPE_MONGO_BSONorDATA_TYPE_JSON_ID).Paste a sample document into Payload (extended JSON for BSON) and the document's ID into Id (also extended JSON, e.g.
{"_id": "123"}).Optionally tick Show partial update and supply a comma-separated list of fields to unset to test the
PARTIAL_UPDATEmode.Click Run. The Results panel shows the transformed output for each applicable mode (
Data,Insert,Update,Partial Update,Delete).
For details on the YAML format and CEL expressions, see Data Transformations. For converting between data types (e.g. string to BSON ObjectID), see Transform Data Types.
Example extended JSON values:
Sync Tester
The Sync Tester runs a small, initial-sync style copy of a specific set of document IDs from source to destination through the regular worker pipeline. It is meant for end-to-end testing of a connector / transformer combination against a small, known set of documents before kicking off a real sync. All testing is local to the web-api -- it does not go through Temporal.
Open Tools > Sync Tester. The tab exposes Verify IDs (read-only comparison, equivalent to the verify-ids command), Test Sync, and some previewing options.
Connector requirements
Not every action in this tab works for every connector. The constraints are:
Sampling IDs (the
Samplemenu) requires the connector being sampled to also be usable as a source.Verify IDs requires both the source and destination connectors to be usable as sources and to support
GetByIds.Test Sync requires the source connector to support
GetByIds. The destination connector only needs to be a valid sink.Some preview/compare options also require the Source Type and Destination Type to be set explicitly (not left on
auto); the UI will surface an error when this applies.
Test Sync
To use Test Sync:
Pick a Source and Destination connector (and optionally a Transformer).
Fill in the Namespace (and Destination/Transform Namespace if it differs).
Optionally set Source Type and Destination Type; leave them on
autoto infer.Paste one extended JSON ID per line into the IDs textarea, or use the Sample menu to populate them from the source or destination.
Click Test Sync.
The pipeline reads each ID from the source, applies the transformer (if enabled), and writes the resulting documents to the destination using the same code path as a real sync. Per-id write results are not surfaced by the pipeline.
When a transformer is in use, namespace mapping is applied before the transform. This means the Destination/Transform Namespace field is the namespace handed to the transformer, and any namespace referenced inside the transformer's mapping config should be that post-mapping namespace, not the original source namespace.
Use Verify IDs in the same tab afterwards to confirm the documents made it to the destination and match the expected payload. Note that the connector restrictions apply, so you may need to just check the destination database directly.
For more details on the underlying commands, see Verification.
Last updated