HTTP transform

This transform performs HTTP POST requests to a HTTP capable endpoint. The service at the endpoint then transforms the entities contained in the request body and returns them in the HTTP response message .

The HTTP endpoint must accept application/json and the response must also be application/json.

The endpoint must support lists of entities only, i.e. it should expect to receive a JSON array and it should always return a JSON array. If the endpoint returns anything other than a “2xx Success” HTTP status code, the transform will raise an exception.

The endpoint is free to decide how the entities are transformed. It’ll just have to produce a list of zero or more entities from the entities it was posted. This means that entities can be transformed, filtered out or new ones created.

Note that for most use cases you can and should use the more general and flexible REST transform instead.

Properties

Property

Type

Description

Default

Req

system

String

The id of the URL system or microservice system component to use.

Yes

url

Object

The URL to HTTP POST entities to.

Yes

batch_size

Integer

The maximum number of entities to POST in each request. If there are more entities than this then they’ll be split across multiple HTTP requests.

100

headers

Dict<String,String>

An optional set of header values to set in the HTTP request. Both keys and values must evaluate to strings.

side_effects

Boolean

Set to false if the transform does not have side-effects. A side-effect means that it causes changes to the system that it talks to. The intention of this property is to prevent inadvertent changes to the system by features like pipe preview. You can set this to false if you’re sure your transform is free from side-effects or if you don’t mind changes happening when previewing a pipe.

true

trigger_on

Object

A dictionary with two properties: "key" (optional, defaults to "_trigger") and "value". The "key" should point to a property in the entity (it supports path notation) and "value" should contain a value that this property should have to be passed into the transform. The "value" supports wildcards (“*”) for substring matching. If the "key" doesn’t exist or the "value" does not match the corresponding value in the entity, the entity will be passed through without being transformed.

Example configuration

{
    "_id": "my-http-transform-service",
    "type": "system:url",
    "base_url": "http://localhost:8080/transforms/"
},
{
    "_id": "deduplicated-men",
    "type": "pipe",
    "source": {
        "type": "dataset",
        "dataset": "men"
    },
    "transform": {
        "type": "http",
        "system":"my-http-transform-service",
        "url": "http://localhost:8080/transforms/deduplicate",
        "batch_size": 5,
        "headers": {
              "some-header": "some-value"
        },
        "trigger_on": {
            "key":"_trigger",
            "value": "some-value*"
        }
    }