1. Home
  2. Documentation
  3. Synchronization
  4. Developer Extension Points
  5. Creating a Custom Data Sink

Creating a Custom Data Sink

The last extension point is the ability to create custom sinks. A custom data sink is a sink service that exposes an internal Sesam stream of JSON objects consumed by a pipe to an external service over HTTP.

Sinks can be used to sit in front of legacy systems for which no Sesam adaptor exists. The main job of these sinks is to make the legacy system appear to be idempotent.

Just as with the Custom Data Source and the Custom Transform you need to define a system (such as the HTTP system or the Microservice system) connected to an external service.

Custom Data Sink - The URL system

The configuration of the custom URL sink is very similar to the custom URL source. The configurations below defines a URL system for the remote service with a JSON sink connected to it in the pipe.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
  {
    "_id": "custom-url-system",
    "type": "system:url",
    "url_pattern": "http://localhost:5000/api/%s"
  }

  {
    "_id": "custom-sink-pipe",
    "type": "pipe",
    "source": {
      "type": "dataset",
      "dataset": "my-dataset"
    },
    "sink": {
      "type": "json",
      "system": "custom-url-system",
      "url": "entities"
    }
  }

Custom Data Sink - The Microservice system

The microservice custom sink works the same way as the microservice custom source, only now with data being passed from a Sesam pipe’s sink to the microservice thgou the JSON push sink.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
  {
    "_id": "custom-sink-pipe",
    "type": "pipe",
    "source": {
      "type": "dataset",
      "dataset": "my-dataset"
    },
    "sink":{
      "type": "json",
      "system": "custom-microservice-system",
      "url": "/my-sink-endpoint"

    }
  }

  {
    "_id": "custom-microservice-system",
    "type": "system:microservice",
    "docker": {
      "environment": {
        "some-other-variable": "some-other-value",
        "some-variable": "some-value"
      },
      "image": "my-image-url",
      "port": 5000
    }
  }

There are some tips on how to create a microservice in Microservices in Sesam. In addition to those, the following templates for custom microservice data sinks are available:

Tip

When using the JSON push sink to send entities from Sesam to the microservice Sesam includes each entity’s reserved fields. There might very well be use for them in the microservice, but if there is not these may have to be removed before sending the data from the microservice to the target system. This may be especially prudent for entities with “_deleted”: true. This means the entity is marked as deleted in Sesam and might require some extra functionality to be handeled in the microservice.

When Sesam sends data through the JSON push sink it sends the data in batches. In addition Sesam will always send one final batch without any data inside it. Therefore, if you have a JSON push sink batch size of 100 (the default value) and try to send 150 entities Sesam will send three different batches. The first batch will contain 100 entities, the second 50 entities and the last one 0 entities. This is good to have in mind when setting up the microservice.


Tutorials

Custom Data Sink - The Microservice System

Look closer into how to create a custom data sink using the Microservice System.

Start tutorial


previous

Creating a Custom Transform

next

Microservices in Sesam