1. Home
  2. Documentation
  3. Synchronization
  4. Developer Extension Points
  5. Creating a Custom Transform

Creating a Custom Transform

DTL and the other transform types provide support for the majority of data transformation use-cases. However, there are times when a special kind of transform needs to be performed. In these cases it is possible add a customized transform. Typically, this is a transform where some external service should be contacted in order to convert a value or to import data based on entity specific information. Optimistic locking is an example of a Sesam pattern based on a custom transform.

Much like the Custom Data Source a custom transform can be connected to either an existing endpoint supporting Sesam protocols, or to a microservice.

The custom transform is generally configured as an HTTP transform connected to either a URL system or a Microservice system. This is defined as part of the transformation pipeline of a pipe.

The service that data is sent to as part of this transform is where the custom code should reside.

The set of transforms is wrapped in an array, and in this case the first transform sends the pipe’s source entities to the custom transform which in turn sends them to an other transform which copies all the returned data and adds a _id property.

Custom Transform - The HTTP transform

In the following configurations we will see how the HTTP transform in combination with the URL system can be used to create a Custom Data Transform. In the example below we show an example of an HTTP transform connected to a Microservice system.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
  {
    "_id": "custom-transform-pipe",
    "type": "pipe",
    "source": {
      "type": "dataset",
      "dataset": "my-dataset"
    },
    "transform": [{
      "type": "dtl",
      "rules": {
        "default": [
          ["copy", "*"],
          ["add", "foo", "_S.some-attribute"]
        ]
      }
    }, {
      "type": "http",
      "system": "custom-transform-microservice",
      "url": "/some-endpoint"
    }, {
      "type": "dtl",
      "rules": {
        "default": [
          ["copy", "*"],
          ["add", "_id", "_S.some-property"]
        ]
      }
    }]
  }

  {
    "_id": "custom-transform-microservice",
    "type": "system:microservice",
    "docker": {
      "environment": {
        "some-other-variable": "some-other-value",
        "some-variable": "some-value"
      },
      "image": "my-image-url",
      "port": 5000
    }
  }

In addition to the information on Microservices in Sesam, following microservice template projects for common languages are provided.

The transform will stream an array of JSON objects to the registered endpoint and expect back a list of entities. The result of the HTTP transform is passed along the transformation pipeline and into the pipe’s sink.


Tutorials

Custom Data Source - The HTTP Transform

Look closer into how to create a custom data transform with a microservice.

Start tutorial


previous

Creating a Custom Data Source

next

Creating a Custom Data Sink