HTTP transform¶

This transform performs HTTP POST requests to a HTTP capable endpoint. The service at the endpoint then transforms the entities contained in the request body and returns them in the HTTP response message .

The HTTP endpoint must accept application/json and the response must also be application/json.

The endpoint must support lists of entities only, i.e. it should expect to receive a JSON array and it should always return a JSON array. If the endpoint returns anything other than a “2xx Success” HTTP status code, the transform will raise an exception.

The endpoint is free to decide how the entities are transformed. It’ll just have to produce a list of zero or more entities from the entities it was posted. This means that entities can be transformed, filtered out or new ones created.

Note that for most use cases you can and should use the more general and flexible REST transform instead.

Properties¶

Property	Type	Description	Default	Req
`system`	String	The id of the URL system or microservice system component to use.		Yes
`url`	Object	The URL to HTTP POST entities to.		Yes
`batch_size`	Integer	The maximum number of entities to POST in each request. If there are more entities than this then they’ll be split across multiple HTTP requests.	100
`headers`	Dict<String,String>	An optional set of header values to set in the HTTP request. Both keys and values must evaluate to strings.
`side_effects`	Boolean	Set to `false` if the transform does not have side-effects. A side-effect means that it causes changes to the system that it talks to. The intention of this property is to prevent inadvertent changes to the system by features like pipe preview. You can set this to `false` if you’re sure your transform is free from side-effects or if you don’t mind changes happening when previewing a pipe.	`true`
`trigger_on`	Object	A dictionary with two properties: `"key"` (optional, defaults to `"_trigger"`) and `"value"`. The `"key"` should point to a property in the entity (it supports path notation) and `"value"` should contain a value that this property should have to be passed into the transform. The `"value"` supports wildcards (“*”) for substring matching. If the `"key"` doesn’t exist or the `"value"` does not match the corresponding value in the entity, the entity will be passed through without being transformed.

Example configuration¶

{
    "_id": "my-http-transform-service",
    "type": "system:url",
    "base_url": "http://localhost:8080/transforms/"
},
{
    "_id": "deduplicated-men",
    "type": "pipe",
    "source": {
        "type": "dataset",
        "dataset": "men"
    },
    "transform": {
        "type": "http",
        "system":"my-http-transform-service",
        "url": "http://localhost:8080/transforms/deduplicate",
        "batch_size": 5,
        "headers": {
              "some-header": "some-value"
        },
        "trigger_on": {
            "key":"_trigger",
            "value": "some-value*"
        }
    }

Emit children transform

JSON Schema validation transform