Transform Functions¶
A transform function is a function that has side-effects, typically modifying the target entity, and has no return value.
add¶
Description |
Examples |
|---|---|
Arguments:
PROPERTY(string{1})
VALUES(value-expression{1})
Adds the PROPERTY field(s) to the target entity with the values returned
by evaluating the VALUES expression.
Note This transform function is namespaced identifiers aware. If namespaced identifiers are enabled,
then the property will be prefixed by the current
namespace. If the property is |
["add", "age", 26]Adds the
age property with the value 26 to the target entity.["add", "upper_name", ["upper", "_S.name"]]Adds the
upper_name property to the target entity. The value is
the uppercased version of the source entity’s name property.["add",["concat", "x-", "_S.key"], "_S.value"]Adds the property returned by the
concat function and assigns it the
value returned by _S.value.With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Source entity:
{}["add", "age", 26]Result:
{"bar:age": 26}. Adds the bar:age property with the value
26 to the target entity.["add", "person:age", 26]Result:
{"person:age": 26}. Adds the person:age property with the
value 26 to the target entity.["add", "::age", 26]Result:
{"age": 26}. Adds the age property with the value 26 to
the target entity. Note
that if the PROPERTY arguments starts with :: it will be interpreted
to mean add what ever is after the double colons. See
NI escape syntax for more details.["add", ":age", 26]Result:
{"bar:age": 26}. Adds the bar:age property with the value
26 to the target entity.
Note that this example uses the NI escape syntax
to reference the current namespace. |
add-if¶
Description |
Examples |
|---|---|
Arguments:
PROPERTY(string{1})
CONDITION(boolean-expression{0|1})
VALUES(value-expression{1})
Adds the PROPERTY field(s) to the target entity with the values returned
by evaluating the VALUES expression, if the CONDITION expression evaluates to True. The CONDITION expression
is evaluated with the
_ variable bound to the result of the VALUES expression.
The CONDITION parameter is optional and defaults to ["neq", "_.", null].Note This transform function is namespaced identifiers aware. If namespaced identifiers are enabled,
then the property will be prefixed by the current
namespace. If the property is |
["add-if", "foo", null]No changes will be made to the target entity.
["add-if", "foo", ["list"]]Adds
"foo": [] to the target entity.["add-if", "foo", ["list", null]]Adds
"foo": [null] to the target entity.["add-if", "foo", 123]Adds
"foo": 123 to the target entity.["add-if", "foo", ["list", 123]]Adds
"foo": [123] to the target entity.["add-if", "foo", ["list", 123, 456]]Adds
"foo": [123, 456] to the target entity.["add-if", "foo", ["gte", "_.", 18], null]No changes will be made to the target entity.
["add-if", "foo", ["gte", "_.", 18], ["list"]]No changes will be made to the target entity.
["add-if", "foo", ["gte", "_.", 18], ["list", null]]No changes will be made to the target entity.
["add-if", "foo", ["gte", "_.", 18], 12]No changes will be made to the target entity.
["add-if", "foo", ["gte", "_.", 18], 123]Adds
"foo": 123 to the target entity.["add-if", "foo", ["gte", "_.", 18], ["list", 123]]Adds
"foo": [123] to the target entity. |
case¶
Description |
Examples |
|---|---|
Arguments:
(VALUE(value-expression{1},
THEN(transforms{1}))+,
ELSE(transforms{0|1})
Evaluates the first THEN for which VALUE is true. If there is no
match, then ELSE is evaluated. If there is no ELSE, then it is a no-op.
Note If you need to specify multiple transforms then wrap them in a list. |
["case",["gte", "_S.age", 18], ["add", "group", "adult"],["gte", "_S.age", 13], ["add", "group", "teenager"],["gte", "_S.age", 2], ["add", "group", "toddler"],["lt", "_S.age", 2], ["add", "group", "baby"],["add", "group", "unknown"]]Adds
{"group": "adult"} if the value of _S.age is greater than or equal to 18,or
{"group": "teenager"} if the value of _S.age is greater than or equal to 13,or
{"group": "toddler"} if the value of _S.age is less than 2,otherwise
{"group": "unknown"}. |
case-eq¶
Description |
Examples |
|---|---|
Arguments:
VALUE(value-expression{1}),
(VALUE_N(value-expression{1},
THEN(transforms{1}))+,
ELSE(transforms{0|1})
Evaluates the first THEN for which VALUE is equal to VALUE_N. If there is no
match, then ELSE is evaluated. If there is no ELSE, then it is a no-op.
Note If you need to specify multiple transforms then wrap them in a list. |
["case-eq", "_S.country","NO", ["add", "country", "Norway"],"SE", ["add", "country", "Sweden"],["add", "country", "Other"]]Given then value of
_S.country, adds {"country": "Norway"} if the value is "NO"
and {"country": "Sweden"} if the value is "SE", otherwise {"country": "Other"} is added.["case-eq", "_S.dialing_code",45, ["add", "country_code", "DK"],46, ["add", "country_code", "SE"],47, ["add", "country_code", "NO"]]Given the value of
_S.dialing_code, adds {"country": "DK"} if the value is
45 and {"country": "SE"} if the value is 46 and {"country": "NO"} if the value is 47,
otherwise it is a no-op. |
comment¶
Description |
Examples |
|---|---|
Arguments:
COMMENTS(value-expression{>=0})
A transform that does nothing except hold comments. Useful for
documenting the transforms, or just disabling transforms inside. Any
expressions inside the comment will not be evaluated.
|
["comment", "This is a comment"]A single line comment.
["comment","First line","Second line","Third line"]A comment that spans multiple lines.
|
copy¶
Description |
Examples |
|---|---|
Arguments:
INCLUDE_PROPERTIES(wildcard-string-list{1})
EXCLUDE_PROPERTIES(wildcard-string-list{1})
Copies properties in INCLUDE_PROPERTIES from the source entity to the
target entity. Any properties matching any of the EXCLUDE_PROPERTIES
patterns are not included. INCLUDE_PROPERTIES and EXCLUDE_PROPERTIES
can be a single string or a list of strings, where the strings are
patterns.
* and ? are valid pattern characters.Note This transform function is namespaced identifiers aware. The pattern in PROPERTY is namespace aware, so the last colon in the pattern is considered to be the separator between the namespace and the identifier. |
["copy", "age"]Copies the
age property from the source entity to the target entity.["copy", "a*", "ab*"]Copies all properties starting with
a from the source entity to the
target entity, but not those starting with ab.["copy",["list", "a*", "b*"],["list", "ab*", "ba*"]]Copies all properties starting with
a or b from the source entity
to the target entity, but not those starting with ab or ba.With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Source entity:
{"age": 36,"x:age": 36}"xyz:age": 36}["copy", "age"]Result:
{ "age": 36."x:age": 36,."xyz:age": 36}.Copies the
age property in all namespaces to the target entity.["copy", "x*:*e", "*y*:*e"]Result:
{ "x:age": 36 }.Copies the properties ending with
e in all namespaces starting with x,
except those in a namespace that contains the character y,
to the target entity. |
create¶
Description |
Examples |
|---|---|
Arguments:
VALUES(value-expression{1})
For each entity in VALUES emit them as new entities to the DTLs output
pipeline.
Note that these new entities must have an
_id string property.
Values that are not entities will be ignored. |
["create", "_S.orders"]Emit the orders in the source entity’s
orders field as new entities.["create", ["apply", "order", "_S.orders"]]Emit the orders in the source entity’s
orders field as new entities,
but apply the order transform to them first. |
create-child¶
Description |
Examples |
|---|---|
Arguments:
VALUES(value-expression{1})
For each entity in VALUES add it to the
$children property on the
target entity. Note that the entities must have an _id string property.
Values that are not entities will be ignored.This function is almost equivalent to:
["add", "$children",["union", "_T.$children", ...]]Note that the
$children property is special. This function should
really only be used when writing into a dataset sink.If an entity with a
$children property is written to the dataset
sink then it will compare it against the value of the $children
property in the previous version of the entity. It will detect deleted
entities and add them to the property before storing the entity.Note also that there is an emit_children
transform that can be used to expand the
$children entities into
standalone entities. |
["create-child", "_S.orders"]Adds the orders in the source entity’s
orders field to the “$children” property
on the target entity. |
default¶
Description |
Examples |
|---|---|
Arguments:
PROPERTY(string{1})
VALUES(value-expression{1})
Adds the PROPERTY field(s) to the target entity with the values returned
by evaluating the VALUES expression, unless the property already exists.
default behaves exactly like add, except that it does not add
the property if the property already exists on the target entity. If
the property exists it does nothing.Note This transform function is namespaced identifiers aware. If namespaced identifiers are enabled,
then the property will be prefixed by the current
namespace. If the property is |
["default", "age", 26]Adds the
age property with the value 26 to the target entity, if
the property does not exists.["default", "upper_name", ["upper", "_S.name"]]Adds the
upper_name property to the target entity, if
the property does not exists.. The value is
the uppercased version of the source entity’s name property.["default",["concat", "x-", "_S.key"], "_S.value"]Adds the property returned by the
concat function and assigns it the
value returned by _S.value, if the property does not exists..With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Source entity:
{}["default", "age", 26]Result:
{"bar:age": 26}. If the target entity does not already have
the bar:age property, then
the bar:age property with the value 26 is added to the target entity.["default", "person:age", 26]Result:
{"person:age": 26}. If the target entity does not already have
the person:age property, then
the person:age property with the value 26 is added to the target entity.["default", "::age", 26]Result:
{"age": 26}. If the target entity does not already have the
age property, thenthe
age property with the value 26 is added to the target entity. Note
that if the PROPERTY arguments starts with :: it will be interpreted
to mean add what ever is after the double colons. See
NI escape syntax for more details.["default", ":age", 26]Result:
{"bar:age": 26}. If the target entity does not already have the
bar:age property, then
the bar:age property with the value 26 is added to the target entity.
Note that this example uses the NI escape syntax
to reference the current namespace. |
discard¶
Description |
Examples |
|---|---|
Arguments:
UNLESS_CONDITION(boolean-expression{0|1})
This transform is almost identical to
filter, but will drop the target
entity on the floor. Use it with care as the discarded entity will not be
deleted in the sink.If the evaluation of the UNLESS_CONDITION expression returns false, then stop
applying transformations. In this case no target entity is emitted
for the source entity. Note that any entities already emitted by
create will not be stopped. If you want then make sure that you don’t
create them before the discard.If the UNLESS_CONDITION argument is not given then the target entity will be discarded.
Warning Only use this transform when you know that you’ve never sent an entity
with the same |
["discard", ["gt", "_S.age", 42]]Discard the target entity unless the source entity’s age is greater than 42.
["discard", ["eq", "_S.type", "person"]]Discard the target entity unless the source entity’s type is
person.["discard"]Discard the target entity unconditionally.
|
filter¶
Description |
Examples |
|---|---|
Arguments:
UNLESS_CONDITION(boolean-expression{0|1})
If the evaluation of the UNLESS_CONDITION expression returns false, then stop
applying transformations. In this case no target entity is emitted
for the source entity. Note that any entities already emitted by
create will not be stopped. If you want then make sure that you don’t
create them before the filter.If the UNLESS_CONDITION argument is not given then the filter evaluates to
false.
Note If used with a The reason for this is so that the The rationale for this behaviour is so that entities that have previous versions get deleted in the resulting dataset when they no longer pass the filter. If you have more than one transform then you may want to be
careful about how you process If you would like to control how deletions happen, then you
should not use the |
["filter", ["gt", "_S.age", 42]]Continue processing only if the source entity’s age is greater than 42.
["filter", ["eq", "_S.type", "person"]]Continue processing only if the source entity’s type is
person.["filter"]Stop processing.
|
if¶
Description |
Examples |
|---|---|
Arguments:
CONDITION(boolean-expression{1}),
THEN(transforms{1}),
ELSE(transforms{0|1})
If CONDITION evaluates to true then apply the transforms in THEN,
otherwise apply the transforms in ELSE.
Note that the THEN and ELSE arguments can either be a single transform
function or a list of transform functions. The list can be empty.
Note If you need to specify multiple transforms then wrap them in a list. ["if", ["gt", "_S.age", 18],[["add", "type", "adult"],["add", "is_adult", true]],["add", "type", "child"]] |
["if", ["eq", "_S.type", "person"], [["add", "type", "person"],["copy", ["list", "name", "age"]]]]If the source entity’s
type field is equal person then apply
the add and copy transforms. There is no else clause given,
which is effectively the same as an empty list with no transforms.["if", ["gt", "_S.age", 18],["add", "type", "adult"],["add", "type", "child"]]If the source entity’s
age is greater than 18 then add type
field with value adult, if not add child. |
make-ni¶
Description |
Examples |
|---|---|
Arguments:
NAMESPACE(string{1})
FROM_PROPERTY(string{1})
TO_PROPERTY(string{0|1})
Adds the FROM_PROPERTY field to the target entity’s TO_PROPERTY with
string values made into namespaced identifiers in the NAMESPACE namespace.
If none of the values can be made into namespaced identifiers then nothing is added. If
TO_PROPERTY is omitted then it defaults to FROM_PROPERTY +
-ni.Note This transform function is namespaced identifiers aware. If namespaced identifiers are enabled,
then the property will be prefixed by the current
namespace. If the property is |
["make-ni", "soccer", "referee", "ref"]If the source entity has
{"referee": "john.doe"} then adds the ref
property with the value ~:soccer:john.doe to the target entity.["make-ni", "hockey", "players"]Adds the
players-ni property to the target entity, if
any namespaced identifiers were created.With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Source entity:
{"referee": "john.doe","x:referee": "john.c.doe"}["make-ni", "soccer", "referee"]Result:
{ "referee-ni": "~:soccer:john.doe""x:referee-ni": "~:soccer:john.c.doe" }.Copies all
referee properties in all namespaces from the source entity
to the target entity. Note that the target properties are suffixed by -ni.
The values are converted into
NIs by adding the soccer namespace. Any non-string values are ignored.["make-ni", "soccer", "referee", "ref"]Result:
{ "ref": "~:soccer:john.doe""x:ref": "~:soccer:john.c.doe" }.Copies all
referee properties in all namespaces from the source entity
to the target entity. Note that the target properties are not suffixed by
-ni. The properties keep their namespaces, but the identifier
part is replaced by ref. The values are converted into
NIs by adding the soccer namespace. Any non-string values are ignored.["make-ni", "soccer", "x:referee"]Result:
{ "x:referee-ni": "~:soccer:john.c.doe" }.Copies the
x:referee property as x:referee-ni from the source entity
to the target entity. The values are converted into
NIs by adding the soccer namespace. Any non-string values are ignored.["make-ni", "soccer", "x:referee", "ref"]Result:
{ "x:ref": "~:soccer:john.c.doe" }.Copies the
x:referee property as x:ref from the source entity
to the target entity. The values are converted into
NIs by adding the soccer namespace. Any non-string values are ignored.["make-ni", "soccer", "x:referee", "y:ref"]Result:
{ "y:ref": "~:soccer:john.c.doe" }.Copies the
x:referee property as y:ref from the source entity
to the target entity. The values are converted into
NIs by adding the soccer namespace. Any non-string values are ignored. |
merge¶
Description |
Examples |
|---|---|
Arguments:
VALUES(value-expression{1})
For each entity in VALUES copy all the properties of the value onto the
target entity. If the property already exists, it will be overwritten. This means that
properties from later value entities win over earlier ones.
Note Any properties at the root level starting with |
["merge", "_S.orders"]Copies the properties of the entities in
_S.orders to the target.["merge", ["list", {"a": 1}, {"a": 2, "b": 3}]]Add the properties
a=2 and b=3 to the target entity. Note that
a=1 is not added because it gets overwritten with a=2 later. |
merge-union¶
Description |
Examples |
|---|---|
Arguments:
VALUES(value-expression{1})
For each entity in VALUES copy all the properties of the value onto the
target entity. If the property already exists on the target entity, add
the new values as a list of values.
Note Any properties at the root level starting with |
["merge-union", "_S.orders"]Copies the properties of the entities in
_S.orders to the target.
Merge the property values if the property already exists.["merge-union",["list", {"a": 1}, {"a": 2, "b": 3}]]Add the properties
a=[1, 2] and b=3 to the target entity. |
remove¶
Description |
Examples |
|---|---|
Arguments:
PROPERTY(wildcard-string{1})
Removes the PROPERTY field from the target entity. The PROPERTY can
be pattern with
* and ? characters in it. The pattern must match
the full property names.Note This transform function is namespaced identifiers aware. The pattern in PROPERTY is namespace aware, so the last colon in the pattern is considered to be the separator between the namespace and the identifier. |
["remove", "age"]Removes the
age property from the target entity.["remove", "temp_*"]Removes all properties matching the
temp_* wildcard pattern from
the target entity.With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Target entity:
{"age": 36,"x:age": 36}"y:age": 36}["remove", "age"]Result:
{}.Removes the
age property in all namespaces from the target entity.["remove", "x*:age"]Result:
{"age": 36, "y:age": 36}.Removes the
age property in all namespaces starting with x
from the target entity.["remove", "x:age"]Result:
{"age": 36, "y:age": 36}.Removes the
x:age property from the target entity. |
rename¶
Description |
Examples |
|---|---|
Arguments:
PROPERTY1(string{1})
PROPERTY2(string{1})
Copies the PROPERTY1 field from the source entity to the PROPERTY2 field
on the target entity. This is effectively a way to copy and rename
properties from the source entity to the target entity. No wildcard
patterns are supported.
Note This transform function is namespaced identifiers aware. If namespaced identifiers are enabled, then the renaming makes sure that any namespaces on the target entity either keep their original namespaces or all values are collected into a single property on the target. |
["rename", "age", "current_age"]Copies the
age field from the source entity and adds it as
current_age on the target entity.["rename",["concat", "in-", "_S.key"],["concat", "out-", "_S.key"]]Copies the value of the property returned by the first
concat function
and assigns it to the property returned by the second concat function.With namespaced identifiers enabled:
{"namespaced_identifiers": true,"namespaces": {"identity": "foo","property": "bar"}}Source entity:
{"a": 1,"x:a": 2}"xyz:a": 3}["rename", "a", "b"]Result:
{ "b": 1."x:b": 2,."xyz:b": 3}.Renames the
a property in all namespaces to b in the same namespace.
In practice this renames the identifier part of the namespaced identifiers
from a to b. If PROPERTY2 only contains the identifier part of
namespaced identifier, then the identifier part of PROPERTY1 will be renamed to this.["rename", "a", "z:b"]Result:
{"z:b": [1, 2, 3]}.Collects all the values in the
a property in all namespaces into
the z:b property. If PROPERTY2 contains a fully qualified namespaced
identifier then all values in PROPERTY1 will be collected into this property.["rename", "x:a", "x:b"]Result:
{"x:b": 2}.Renames the
x:a property to x:b. |
fail!¶
Description |
Examples |
|---|---|
Arguments:
MESSAGE(string{1})
A function that makes the pump fail. It can be used to conditionally fail the pump. A string error message can be specified in the first argument.
|
["fail!", "Processing stopped because of invalid input. Please review."]Causes the pump to fail. The error message is reported in the pump-failed event in the pump execution dataset.
|
sleep!¶
Description |
Examples |
|---|---|
Arguments:
PERIOD(integer{1})
A function that makes the processing to pause for the given time. The time period is specified in seconds. Use this function with care as it may block other pipes if it pauses for too long.
|
["sleep!", 10]The processing pauses for 10 seconds.
|
trip!¶
Description |
Examples |
|---|---|
Arguments:
MESSAGE(string{1})
A function that makes the pump trip the sink dataset’s circuit breaker. A string error message can be specified in the first argument.
|
["trip!", "Circuit breaker tripped because of invalid input. Please review and rollback or commit dataset."]Causes the pump to fail and the sink dataset’s circuit breaker to trip. The error message is reported in the pump-failed event in the pump execution dataset.
|