# Web-API The `web-api` command starts an HTTP server that serves both a gRPC API and a single-page web UI for managing dsynct. Whereas the `dsynct run` command is scoped to a single migration workflow, the web-api is intended as a longer-lived control plane: it can list and start syncs, manage Temporal flows, register dynamic workers, and run interactive testing tools. To start the web-api: ```bash docker run --name dsyncwebapi \ -p 8080:8080 \ -e 'DSYNCT_MODE=simple' \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://:4317' \ markadiom/dsynct \ --host-port 0.0.0.0:8080 \ --otel \ web-api \ --temporal-host-port :7233 ``` The relevant flags for the `web-api` subcommand are: | Flag | Description | | ---------------------------- | ---------------------------------------------------------------------------------------------------------- | | `--temporal-host-port` | Address of the Temporal frontend, e.g. `temporal:7233`. Required unless `--simple-only` is set. | | `--temporal-namespace` | Temporal namespace to use. Default: `default`. | | `--simple-only` | Run without Temporal. Only Local Syncs and Tools are available; the Flows and Workers sections are hidden. | | `--cert-file` / `--key-file` | TLS certificate and key for serving HTTPS. | | `--no-gzip` | Disable gzip compression on the API. | Open the served address in a browser. The left-hand drawer contains four sections: **Progress**, **Syncs**, **Workers**, and **Tools**. ## Progress The Progress section shows live progress for any sync that the web-api is currently tracking. This includes both Local Syncs started from the Syncs tab and Temporal flows that have been added in the Flows tab. Use the dropdown at the top of the page to switch between tracked flows. For each selected flow the page shows: * A summary card with the workflow ID, current state, and (where applicable) a button to terminate the workflow or cancel the sync. * The latest activity errors, if any. * An **Initial Sync** card with per-namespace progress and controls to pause/unpause initial sync activities (overall or for a specific namespace). * A **Change Stream** card with controls to pause/unpause change stream and read-ahead activities. * An **Operation Log** that records the result of pause/unpause/terminate actions issued from the page. For local (non-Temporal) syncs, pause/unpause/terminate controls are not shown; only the cancel-sync button is available. ## Syncs The Syncs section is where you configure which syncs are tracked in the Progress view. It has two tabs: ### Local Syncs Local Syncs are non-Temporal syncs that run directly inside the web-api server process. They are intended for testing and for smaller projects where standing up a Temporal cluster would be overkill. Each local sync uses the same underlying pipeline as a worker, but progress is tracked in-memory by the web-api. To start a local sync: 1. Type a connector argument string into the **Run New Sync** input, for example `/dev/fakesource /dev/null`. Quoting follows shell-style rules. 2. Click **Run** to start the sync, or **Form** to open a guided form that exposes all `sync` flags and connector options. Started syncs are listed below with their current state (`Starting`, `Running`, `Done`, `Error`, or `Cancelled`) and the original arguments. Each row offers: * **View** -- jump to the Progress section for that sync. * **Cancel** -- cancel a running sync. * **Remove** -- remove a finished or errored sync from the list. ### Flows The Flows tab is the interface to Temporal. From here you can either start a brand-new flow or attach to one that already exists in Temporal: * **Run** -- type or build a `dsynct run`-style argument string (e.g. `--workflow-id myflow --queue-name dsync ...`) and submit. The web-api will start the workflow in Temporal and begin tracking it. The **Form** button opens a guided form for the same set of flags. * **Add Workflow** -- attach to an existing Temporal workflow. Click **List Workflow IDs** to fetch known workflow IDs from Temporal, then click **Add** next to any of them to start tracking it. The **Check Interval** and **Publish Interval** control how often the web-api polls Temporal for state and pushes updates to the UI. * **Active Flows** -- the list of currently tracked flows. Use **Remove** to stop tracking a flow (this does not affect the workflow on Temporal itself). Both Local Syncs and Flows feed into the Progress dropdown, so any sync started here can be monitored from the Progress section. ## Workers The Workers section is only shown when the web-api is running with Temporal (i.e. without `--simple-only`). It is used to control **dynamic workers**. Dynamic workers are dsynct worker processes started by long-running **node** processes that register themselves with the web-api over a coordinator API. Once registered, the web-api can ask a node to start or stop a worker using the same arguments you would pass to `dsynct worker`. This makes it possible to add or remove worker capacity from the UI without restarting any containers. ### Starting a node A node is started with the `node` command (run via the simple-mode entrypoint). Each node needs to know the Temporal address and the URL of the web-api that acts as its coordinator: ```bash docker run --name dsyncnode \ -e 'DSYNCT_MODE=simple' \ -e 'OTEL_EXPORTER_OTLP_ENDPOINT=http://:4317' \ markadiom/dsynct \ --host-port 0.0.0.0:8081 \ --otel \ node \ --temporal-host-port :7233 \ --coordinator-url http://:8080 \ --poll-interval 10s \ --heartbeat-timeout 60s ``` The relevant flags are: | Flag | Description | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--temporal-host-port` | Address of the Temporal frontend. | | `--temporal-namespace` | Temporal namespace to use. Default: `default`. | | `--coordinator-url` | URL of the web-api that this node should poll for commands. If unset, the node still serves its node API directly but will not auto-register with a coordinator. | | `--poll-interval` | How often the node polls the coordinator for new commands. Default: `10s`. | | `--heartbeat-timeout` | The coordinator marks the node dead if it does not poll within this duration. Default: `60s`. | Run as many `node` processes as you need; each one becomes a separate row in the Workers section once it polls the coordinator. ### Managing workers from the UI Once at least one node is registered, the Workers section displays: * **Add Worker to All Nodes** -- type or build a `dsynct worker`-style argument string (e.g. `--queue-name myq /dev/fakesource /dev/null`) and click **Add to All** to start an identical worker on every registered node. The **Form** button opens a guided form with all `worker` flags. **Remove All Workers** stops every managed worker on every node. * **Nodes** -- one block per registered node, showing the node ID and last-seen timestamp. Inside each block: * A per-node argument input to start a worker only on that node. * A list of managed workers with their state (`Starting`, `Running`, `Stopping`, `Stopped`, `Error`), the original arguments, **Remove** to stop just that worker, and **Remove All Like This** to stop every worker (across all nodes) that was started with the same arguments. * Pending commands (add/remove) show their state (`pending`, `acked`, `error`) until they are processed by the node. Workers stopped by the coordinator continue to be reflected in the UI until they actually terminate, so there can be a short delay between clicking **Remove** and the row disappearing. {% hint style="warning" %} As with any other dsynct worker, every worker on the same `--queue-name` must be configured identically (same source/destination/transformer). Mixing configurations on the same queue will cause errors. {% endhint %} ## Tools The Tools section provides utilities for testing transformations and syncs **without** running a full job. There are two tabs. ### Transform Studio Transform Studio is an interactive way to iterate on a `dsync-transform` YAML config against sample documents. It is fully local to the web-api process -- nothing is written to any source or destination. Open `Tools > Transform Studio` and: 1. Paste your transform config into the **YML Config** field. 2. Set the **Namespace** to the source namespace the document belongs to. 3. Choose the **Src Type** and **Dst Type** (`DATA_TYPE_MONGO_BSON` or `DATA_TYPE_JSON_ID`). 4. Paste a sample document into **Payload** (extended JSON for BSON) and the document's ID into **Id** (also extended JSON, e.g. `{"_id": "123"}`). 5. Optionally tick **Show partial update** and supply a comma-separated list of fields to unset to test the `PARTIAL_UPDATE` mode. 6. Click **Run**. The Results panel shows the transformed output for each applicable mode (`Data`, `Insert`, `Update`, `Partial Update`, `Delete`). For details on the YAML format and CEL expressions, see [Data Transformations](/enterprise/running-dsynct/data-transformations.md). For converting between data types (e.g. string to BSON ObjectID), see [Transform Data Types](/enterprise/running-dsynct/data-types.md). Example extended JSON values: ``` {"$oid": "<24-character string>"} # Mongo Object ID {"$date": ""} # Date ``` ### Sync Tester The Sync Tester runs a small, initial-sync style copy of a specific set of document IDs from source to destination through the regular worker pipeline. It is meant for end-to-end testing of a connector / transformer combination against a small, known set of documents before kicking off a real sync. **All testing is local to the web-api -- it does not go through Temporal.** Open `Tools > Sync Tester`. The tab exposes `Verify IDs` (read-only comparison, equivalent to the `verify-ids` command), `Test Sync`, and some previewing options. #### Connector requirements Not every action in this tab works for every connector. The constraints are: * **Sampling IDs** (the `Sample` menu) requires the connector being sampled to also be usable as a source. * **Verify IDs** requires both the source and destination connectors to be usable as sources and to support `GetByIds`. * **Test Sync** requires the source connector to support `GetByIds`. The destination connector only needs to be a valid sink. * Some preview/compare options also require the **Source Type** and **Destination Type** to be set explicitly (not left on `auto`); the UI will surface an error when this applies. #### Test Sync To use Test Sync: 1. Pick a **Source** and **Destination** connector (and optionally a **Transformer**). 2. Fill in the **Namespace** (and **Destination/Transform Namespace** if it differs). 3. Optionally set **Source Type** and **Destination Type**; leave them on `auto` to infer. 4. Paste one extended JSON ID per line into the **IDs** textarea, or use the **Sample** menu to populate them from the source or destination. 5. Click **Test Sync**. The pipeline reads each ID from the source, applies the transformer (if enabled), and writes the resulting documents to the destination using the same code path as a real sync. Per-id write results are not surfaced by the pipeline. When a transformer is in use, namespace mapping is applied **before** the transform. This means the **Destination/Transform Namespace** field is the namespace handed to the transformer, and any namespace referenced inside the transformer's mapping config should be that post-mapping namespace, not the original source namespace. Use **Verify IDs** in the same tab afterwards to confirm the documents made it to the destination and match the expected payload. Note that the connector restrictions apply, so you may need to just check the destination database directly. For more details on the underlying commands, see [Verification](/enterprise/running-dsynct/verification.md). --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.adiom.io/enterprise/running-dsynct/web-api.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.