HTTP transform¶

This transform performs HTTP POST requests to a HTTP capable endpoint. The service at the endpoint then transforms the entities contained in the request body and returns them in the HTTP response message .

The HTTP endpoint must accept application/json and the response must also be application/json.

The endpoint must support lists of entities only, i.e. it should expect to receive a JSON array and it should always return a JSON array. If the endpoint returns anything other than a “2xx Success” HTTP status code, the transform will raise an exception.

The endpoint is free to decide how the entities are transformed. It’ll just have to produce a list of zero or more entities from the entities it was posted. This means that entities can be transformed, filtered out or new ones created.

Properties¶

Property	Type	Description	Default	Req
`system`	String	The id of the URL system or microservice system component to use.		Yes
`url`	Object	The URL to HTTP POST entities to.		Yes
`batch_size`	Integer	The maximum number of entities to POST in each request. If there are more entities than this then they’ll be split across multiple HTTP requests.	100
`headers`	Dict<String,String>	An optional set of header values to set in the HTTP request. Both keys and values must evaluate to strings.

Example configuration¶

{
    "_id": "my-http-transform-service",
    "type": "system:url",
    "base_url": "http://localhost:8080/transforms/"
},
{
    "_id": "deduplicated-men",
    "type": "pipe",
    "source": {
        "type": "dataset",
        "dataset": "men"
    },
    "transform": {
        "type": "http",
        "system":"my-http-transform-service",
        "url": "http://localhost:8080/transforms/deduplicate",
        "batch_size": 5,
        "headers": {
              "some-header": "some-value"
        }
    }

Conditional transform

Emit children transform