Data Transformations

Overview

Enterprise Dsync features a prebuilt YAML-based transformer that allows to add, remove and modify data elements using mappings and Common Expression Language (CEL).

Transformations are applied on-the-fly during both initial sync and CDC. The transformer itself can be run as a standalone process that Dsynct connects to over gRPC. The detailed instructions can be found in our public repository.

For convenience, Dsynct workers can run the transformer as an embedded process by providing the --transform option along with the path to the config file as the third argument:

dsynct worker <OTHER_OPTIONS> --transform $SOURCE $DESTINATION dsync-transform://transform.yaml

When using Docker to run Dsynct, the transformer config file needs to be mounted to the container. For example:

docker run \
-v "./transform.yaml:/transform.yaml" \
markadiom/dsynct worker <OTHER_OPTIONS> \
--transform \
$SOURCE $DESTINATION dsync-transform://transform.yaml

Sample Config File

mappings:
  - namespace: srcnamespace
    mapnamespace: dstnamespace
    cel:
      name: self + "!"
      newfield: '"abcd"'
    add: ["newfield"]
    delete: ["existingfield"]

For additional details, please consult the README in the our public repository.