The Pump Execution Dataset¶

The pump execution dataset is a log which contains entities with information about a particular pump’s execution history. The dataset id is computed from the pump’s _id property as system:pump_execution:<pump_id>.

Warning

The pump execution entities are subject to change. Their ids and structure may be changed in the future. The dataset is not meant to be a public API that applications can use.

It will always contain at least two entities for each time the pump runs. The first will have the id “pump-started” and the last either “pump-ended” or “pump-completed”. In between these entities, there may be entities recording read errors and/or write errors. Other informational entities such as “pump-enabled”/”pump-disabled” or “pipe-offset-set” may also be present.

Depending on the pump settings, the write errors can then be retried one or more times during the next runs.

The pump-started entity¶

The “pump-started” is written to the pump execution dataset when the pump has started, but before any other processing takes place.

Prototype¶

{
    "_id":  "pump-started",
    "event_type": "pump-started",
    "start_time": "iso-timestamp-in-UTC",
    "sync_type": "type-of-sync",
    "pump_definition": "pump-configuration-id",
    "user": {
        "email": "some@user.com",
        "remote_addr": "127.0.0.1",
        "user_id": "some@user.com"
    }
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for pump-started entities is fixed and will always be “pump-started”
`event_type`	String	The `event_type` value for pump-started entities is fixed and will always be “pump-started”
`sync_type`	Enum<String>	The `sync_type` value denotes the type of pipe run that produced this log entry. It can be one of the followin values: `full`, `partial` or `incremental`.
`start_time`	String	The ISO-formatted timestamp for the timestamp the pump started (“YYYY-MM-DDTHH:mm:SS.fZ”)
`pump_definition`	String	The `_id` value of the pump configuration used to instantiate the pump
`user`	Object	Information about the user that started the run, if available.

The pump-completed entity¶

If the pump completes successfully, it will write a “pump-completed” entity to the execution log.

Prototype¶

{
    "_id":  "pump-completed",
    "event_type": "pump-completed",
    "pump_definition": "pump-configuration-id",
    "start_time": "pump-started-timestamp-in-UTC",
    "end_time": "pump-ended-iso-timestamp-in-UTC",
    "sync_type": "type-of-sync",
    "pump_started_location": 1234,
    "retry_entities_exist": false,
    "entities_succeeded": 123,
    "entities_failed": 0
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for pump-completed entities is fixed and will always be “pump-completed”.
`event_type`	String	The `event_type` value for pump-completed entities is fixed and will always be “pump-completed”.
`pump_definition`	String	The `_id` value of the pump configuration used to instantiate the pump
`sync_type`	Enum<String>	The `sync_type` value denotes the type of pipe run that produced this log entry. It can be one of the followin values: `full`, `partial` or `incremental`.
`start_time`	String	The ISO-formatted timestamp for the timestamp the pump started (“YYYY-MM-DDTHH:mm:SS.fZ”).
`end_time`	String	The ISO-formatted timestamp for the timestamp the pump ended (“YYYY-MM-DDTHH:mm:SS.fZ”).
`pump_started_location`	Integer	The absolute index into the log where the corresponding “pump-started” entity is located. It is used by the pump’s retry mechanism to “rewind” the log to the last successfully completed run.
`retry_entities_exist`	Boolean	A flag indicating if there was logged any entities that can be retried during this run.
`entities_succeeded`	Integer	A counter with the number of entities that was successfully written to the pipe’s sink during this run.
`entities_failed`	Integer	A counter with the number of entities that failed to be written to the pipe’s sink during this run.

The pump-failed entity¶

If the pump fails for some reason, it will write a “pump-failed” entity when it terminates.

Prototype¶

{
    "_id":  "pump-failed",
    "event_type": "pump-failed",
    "pump_definition": "pump-configuration-id",
    "start_time": "pump-started-timestamp-in-UTC",
    "end_time": "pump-ended-iso-timestamp-in-UTC",
    "pump_started_location": 1234,
    "retry_entities_exist": true,
    "entities_succeeded": 123,
    "entities_failed": 10,
    "reason_why_stopped": "traceback-info",
    "sync_type": "type-of-sync",
    "exception_entity": {
      "_id": "id-of-the-entity",
      "entity-property": "entity-value"
    },
    "node_build_info": {
        "build": null,
        "date": "2021-12-29T13:51:21.566325+00:00",
        "dirty": true,
        "git-revision": "6c8eb3c17",
        "hash": "6c8eb3c17",
        "release": "1.0",
        "teamcity-buildnumber": null,
        "version": "1.0.dev"
    }
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for pump-failed entities is fixed and will always be “pump-failed”.
`event_type`	String	The `event_type` value for pump-failed entities is fixed and will always be “pump-failed”.
`sync_type`	Enum<String>	The `sync_type` value denotes the type of pipe run that produced this log entry. It can be one of the followin values: `full`, `partial` or `incremental`.
`pump_definition`	String	The `_id` value of the pump configuration used to instantiate the pump
`start_time`	String	The ISO-formatted timestamp for the timestamp the pump started (“YYYY-MM-DDTHH:mm:SS.fZ”).
`end_time`	String	The ISO-formatted timestamp for the timestamp the pump ended (“YYYY-MM-DDTHH:mm:SS.fZ”).
`pump_started_location`	Integer	The absolute index into the log where the corresponding “pump-started” entity is located. It is used by the pump’s retry mechanism to “rewind” the log to the last successfully completed run.
`retry_entities_exist`	Boolean	A flag indicating if there was logged any entities that can be retried during this run.
`exception_entity`	Object	A complete embedded copy of the entity that caused the failure (if available).
`entities_succeeded`	Integer	A counter with the number of entities that was successfully written to the pipe’s sink during this run.
`entities_failed`	Integer	A counter with the number of entities that failed to be written to the pipe’s sink during this run.
`reason_why_stopped`	String	Information about why the pump failed. It contains among other things a stack trace of the execution failure.
`node_build_info`	Object	Information about the sesam instance the pipe was run on.

The read-error entity¶

The execution dataset also can contain entities that record failed reads and/or entities wich represents retryable entities (from write errors).

Prototype¶

{
    "_id":  "read-error:<GUID>",
    "event_type": "read-error",
    "error_code": 0,
    "event_time": "failure-ISO-timestamp-in-UTC",
    "exception": "traceback-info-from-pump",
    "original_exception": "the-exception-cast-by-source"
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for read-error entities is computed from the string prefix “read-error:” concatenated with a GUID string.
`event_type`	String	The `event_type` value for read-error entities is fixed and will always be “read-error”.
`error_code`	Integer	A integer value that will be either `0`, meaning that the source was unable to establish communications with the source system, or `1` - meaning that there was an error while trying to read a particular entity from the source.
`event_time`	String	The ISO-formatted timestamp for the timestamp when the read error happened (“YYYY-MM-DDTHH:mm:SS.fZ”).
`exception`	String	Information about from the pump failure. It a stack trace of the execution failure.
`original_exception`	String	Information about from the source about the read failure. It contains among other things a stack trace of the execution failure in the source.

The write-error entity¶

For retryable (write) errors, the entity has a similar form to the “read-error” entity, except its _id property is computed from the entity that was unsuccessfully written. It also contains the complete entity as an embedded child entity.

Prototype¶

{
    "_id":  "write-error:<entity_id>",
    "event_type": "write-error",
    "error_code": 0,
    "event_time": "failure-ISO-timestamp-in-UTC",
    "retry_attempts": 0,
    "retryable": false,
    "resolved": true,
    "dead": false,
    "entity": {
      "_id": "id-of-the-entity",
      "entity-property": "entity-value"
    },
    "resolved_entity": {
      "_id": "id-of-the-entity-that-resolved-the-error-if-different",
      "entity-property": "entity-value"
    },
    "exception": "traceback-info-from-pump",
    "original_exception": "the-exception-cast-by-sink",
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for read-error entities is computed from the string prefix “write-error:” concatenated with the failed entity `_id` property.
`event_type`	String	The `event_type` value for write-error entities is fixed and will always be “write-error”.
`error_code`	Integer	A integer value that will be either `0`, meaning that the sink was unable to establish communications with the target system, or `1` - meaning that there was an error while writing the particular entity to the target system.
`event_time`	String	The ISO-formatted timestamp for the timestamp when the write error happened (“YYYY-MM-DDTHH:mm:SS.fZ”).
`retry_attempts`	Integer	A counter of how many times the failing entity has been retried. Note that the first time it is written to the execution dataset it will be 0.
`retryable`	Boolean	A flag indicating if the entity can be retried by the retry mechanism. It is used for the case where a new version of a entity comes from the source while there also exist a previously failed version in the execution dataset. In this case, if the new version is sucessfully written to the sink a new write-error entity is written to the log for this entity, but marked as not retryable (i.e `false` value) so it can be skipped during retries.
`resolved`	Boolean	A flag indicating if the entity was successfully resolved either by a newer version of the entity or because a retry succeeded.
`dead`	Boolean	A flag indicating if the entity has been given up on, for example having exceeded some number of retries. If a dead letter dataset is specified for the pump, the “dead” entity will be written there and a final “write-error” entity written to the execution with the `dead` flag set to `true`. This entity will then never be retried again (until a new version comes along from the source).
`entity`	Object	A complete embedded copy of the failed entity.
`resolved_entity`	Object	A complete embedded copy of the entity that resolved the write-error if it was retried (and if it differs from `entity`). This property will only be set if `resolved` is also `true`.
`exception`	String	Information about from the pump failure. It a stack trace of the execution failure.
`original_exception`	String	Information about from the sink about the write failure. It contains among other things a stack trace of the execution failure in the sink.

The pump-offset-set entity¶

The “pump-offset-set” is written to the pump execution dataset when the pump offset has been set or reset explicitly.

Prototype¶

{
    "_id":  "pump-offset-set",
    "event_time": "2022-01-04T13:20:28.630647Z",
    "event_type": "pump-offset-set",
    "is_reset": true,
    "original_pipe_offset": "1033311",
    "pipe_offset": "",
    "user": {
        "email": "some@user.com",
        "remote_addr": "127.0.0.1",
        "user_id": "some@user.com"
    }
}

Properties¶

Property	Type	Description
`_id`	String	The `_id` value for pump-offset-set entities is fixed and will always be “pump-offset-set”
`event_type`	String	The `event_type` value for pump-started entities is fixed and will always be “pump-offset-set”
`event_time`	String	The ISO-formatted timestamp for the timestamp the pump offset was set (“YYYY-MM-DDTHH:mm:SS.fZ”)
`pipe_offset`	String	The pipe offset that was set.
`original_pipe_offset`	String	The pipe offset that was set before it was overwritten.
`is_reset`	Boolean	A flag indicating if this was caused by someone resetting the pipe.
`user`	Object	Information about the user that started the run, if available.

Metrics and monitoring

Configuration Management