> For the complete documentation index, see [llms.txt](https://docs.adiom.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.adiom.io/enterprise/running-dsynct/data-transformations.md). # Data Transformations ## Overview Enterprise Dsync features a prebuilt YAML-based transformer that allows to add, remove and modify data elements using mappings and Common Expression Language ([CEL](https://github.com/google/cel-spec)). Transformations are applied on-the-fly during both initial sync and CDC, one record at a time. | If you want to... | Read | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Decide whether you need a transform at all | [When You Need a Transform](#when-you-need-a-transform) | | Know what the transformer cannot do | [What Transforms Cannot Do](#what-transforms-cannot-do) | | Go from sample data to a tested config, step by step | [Designing and Testing Transforms](/enterprise/running-dsynct/data-transformations/designing-and-testing.md) | | Start from a working config | [Templates](/enterprise/running-dsynct/data-transformations/templates.md) | | Look up a config option or function | [Config Reference](/enterprise/running-dsynct/data-transformations/config-reference.md) and [CEL Reference](/enterprise/running-dsynct/data-transformations/cel-reference.md) | | Map IDs, convert between JSON and BSON, or handle updates and deletes | [IDs, Data Types and CDC Events](/enterprise/running-dsynct/data-transformations/data-types.md) | | Understand a transform error | [Troubleshooting Transforms](/enterprise/running-dsynct/data-transformations/troubleshooting.md) | | See exactly what a source connector emits | [Connectors](/reference/connectors.md) | {% hint style="info" %} **Using an AI agent to write transforms?** Give it the [Designing and Testing Transforms](/enterprise/running-dsynct/data-transformations/designing-and-testing.md) page. Every page on this site is also available as Markdown by adding `.md` to its URL, and the full index is at [docs.adiom.io/llms.txt](https://docs.adiom.io/llms.txt). The agent can validate configs against sample documents through the [Web-API](/enterprise/running-dsynct/web-api/api-reference.md) without touching your databases. {% endhint %} ## When You Need a Transform You need a transform if any of the following is true. | Situation | Why | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | The source and destination use different data types (JSON and BSON) | Dsync does not convert between them on its own. Without a transform the sync fails to start with `no compatible data path`. | | The ID is named or shaped differently on the two sides | For example Cosmos DB NoSQL identifies a document by shard key plus `id`, while MongoDB uses a single `_id`. | | Fields must be renamed, added, removed, restructured or converted to another type | The default is to copy every field as it is. | | Some documents must be left out | | | Data must be anonymized or masked on the way | | You do not need a transform when the destination accepts the source's data type, the ID already has the right shape, and the documents should arrive unchanged. Renaming a namespace alone does not need a transform either: use `--namespace-mapping`. Common pairs: | Source | Destination | Transform | | --------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MongoDB | MongoDB, Azure DocumentDB, AWS DocumentDB | Not required. | | DynamoDB | MongoDB | Not required. The connector emits BSON with an `_id`. Check the [number and ID options](/reference/connectors/dynamodb.md) first. | | DynamoDB | Cosmos DB NoSQL | Not required. The connector emits JSON with an `id`. | | Cosmos DB NoSQL | MongoDB | Required (JSON to BSON, and the ID must be mapped). | | MongoDB | Cosmos DB NoSQL | Required (BSON to JSON, and the ID must be mapped). | | SQL Batch | MongoDB | Required (JSON to BSON, and the ID must be mapped). | | PostgreSQL | MongoDB | Not required. The connector emits BSON with an `_id`. Check the [type mapping](/reference/connectors/postgresql.md#emitted-format) first. | | MongoDB | PostgreSQL | Usually required, to make document fields match the table columns. See [PostgreSQL as a destination](/reference/connectors/postgresql.md#writing-to-postgresql). | | PostgreSQL | Cosmos DB NoSQL | Required (BSON to JSON, and the ID must be mapped). | If you are unsure, contact Adiom with a sample source document and the shape you want on the destination. ## What Transforms Cannot Do The transformer sees one record at a time and has no access to any database. That rules out the following. | Not possible | Why | What to do instead | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | Look up or join data from another record, collection or table | A transform only sees the current record, its ID and the static `env` values from the config. | Join at the source: with SQL Batch, put the join in the `query`. Otherwise use a custom transformer or enrich the data after the migration. | | Aggregate across records (counts, sums, latest-of) | Nothing is remembered between records. | Aggregate at the source or on the destination after the sync. | | Build the destination ID from fields other than the source ID | Deletes carry only the ID, so the destination ID must be computable from the source ID alone. | Keep the ID derived from the source ID. Store other fields as regular fields. | | Route or filter on document content in a way that also applies to deletes | `filter` only sees the ID. A document can be dropped by content, but a later delete for the same ID is still sent. | Usually harmless, as the delete matches nothing. | | Split one document into several documents, for example one per array element | Each mapping produces at most one output record per input record. | Copy the document to several namespaces with different fields (one mapping per target), or reshape at the source. | | Merge several source records into one destination document | Same reason as joins. | Join at the source. | | Address a single array element by position, or delete some elements by path | Paths apply to every element of an array. | Rewrite the whole array with a CEL expression on the array field. | | Run custom code or call external services | Expressions are limited to CEL and the built-in functions. | Use a custom transformer. | When the YAML transformer is not enough, you have three options: 1. **Reshape at the source.** A view, or the `query` of the [SQL Batch](/reference/connectors/sql-batch.md) connector, can join and aggregate before Dsync reads the data. 2. **Write a custom transformer.** Dsync can call any gRPC service that implements the transformer interface, in any language. See [Transformations](/basics/features.md#transformations). The rule about deletes still applies: the destination ID must be derivable from the source ID. 3. **Ask Adiom.** If you are missing a function (a new type conversion, a hash, a format), it can often be added to the built-in transformer. ## Running the Transformer Dsynct workers can run the transformer as an embedded process by providing the `--transform` option along with the path to the config file as the third argument: ```bash dsynct worker --transform $SOURCE $DESTINATION dsync-transform://transform.yaml ``` When using Docker to run Dsynct, the transformer config file needs to be mounted to the container. For example: ``` docker run \ -v "./transform.yaml:/transform.yaml" \ markadiom/dsynct worker \ --transform \ $SOURCE $DESTINATION dsync-transform://transform.yaml ``` The same `--transform` option and third argument work with the `sync`, `testsync`, `sample-ids` and `verify-ids` commands. The transformer can also be run as a standalone process that Dsynct connects to over gRPC. The detailed instructions can be found in our [public repository.](https://github.com/adiom-data/public/tree/main/dsync-transform) {% hint style="warning" %} Every worker on the same `--queue-name` must use the same transform config. To change a transform, restart all workers on that queue with the new file. {% endhint %} ## Testing Transforms Use [Transform Studio](/enterprise/running-dsynct/web-api.md#transform-studio) to try a config against sample documents, and the [Sync Tester](/enterprise/running-dsynct/web-api.md#sync-tester) or [`testsync`](/enterprise/running-dsynct/running-dsynct.md#testsync) to copy a few real documents end to end. The full process is in [Designing and Testing Transforms](/enterprise/running-dsynct/data-transformations/designing-and-testing.md).