Data Transformations
Overview
Enterprise Dsync features a prebuilt YAML-based transformer that allows to add, remove and modify data elements using mappings and Common Expression Language (CEL).
Transformations are applied on-the-fly during both initial sync and CDC. The transformer itself can be run as a standalone process that Dsynct connects to over gRPC. The detailed instructions can be found in our public repository.
For convenience, Dsynct workers can run the transformer as an embedded process by providing the --transform option along with the path to the config file as the third argument:
dsynct worker <OTHER_OPTIONS> --transform $SOURCE $DESTINATION dsync-transform://transform.yamlWhen using Docker to run Dsynct, the transformer config file needs to be mounted to the container. For example:
docker run \
-v "./transform.yaml:/transform.yaml" \
markadiom/dsynct worker <OTHER_OPTIONS> \
--transform \
$SOURCE $DESTINATION dsync-transform://transform.yamlSample Config File
mappings:
  - namespace: srcnamespace
    mapnamespace: dstnamespace
    cel:
      name: self + "!"
      newfield: '"abcd"'
    add: ["newfield"]
    delete: ["existingfield"]For additional details, please consult the README in the our public repository.
Last updated