HTTP transform

This transform performs HTTP POST requests to a HTTP capable endpoint. The service at the endpoint then transforms the entities contained in the request body and returns them in the HTTP response message .

The HTTP endpoint must accept application/json and the response must also be application/json.

The endpoint must support lists of entities only, i.e. it should expect to receive a JSON array and it should always return a JSON array. If the endpoint returns anything other than a “2xx Success” HTTP status code, the transform will raise an exception.

The endpoint is free to decide how the entities are transformed. It’ll just have to produce a list of zero or more entities from the entities it was posted. This means that entities can be transformed, filtered out or new ones created.

Properties

Property

Type

Description

Default

Req

system

String

The id of the URL system or microservice system component to use.

Yes

url

Object

The URL to HTTP POST entities to.

Yes

batch_size

Integer

The maximum number of entities to POST in each request. If there are more entities than this then they’ll be split across multiple HTTP requests.

100

headers

Dict<String,String>

An optional set of header values to set in the HTTP request. Both keys and values must evaluate to strings.

Example configuration

{
    "_id": "my-http-transform-service",
    "type": "system:url",
    "base_url": "http://localhost:8080/transforms/"
},
{
    "_id": "deduplicated-men",
    "type": "pipe",
    "source": {
        "type": "dataset",
        "dataset": "men"
    },
    "transform": {
        "type": "http",
        "system":"my-http-transform-service",
        "url": "http://localhost:8080/transforms/deduplicate",
        "batch_size": 5,
        "headers": {
              "some-header": "some-value"
        }
    }