The Connector Contract¶

This document describes the output of connectors and the input to connectors. The output is the dataset that the connector writes to when synchronizing entities from the system. The input is the desired state of the system and the input to the connector to make it so.

Output of the connector¶

The dataset must use the following naming convention: SYSTEM-DATATYPE-collect, e.g. foo-person-collect.

Connector should expose the data as close to the original API as possible. If any properties are added by the connector that are not in the API, they should be prefixed with sesam_, e.g. sesam_saleId so that we know which data has been added by the connector.

Properties¶

Property	Datatype	Description	Required
`$origin`	Opaque	The value of the `$origin` property is opaque and is managed by the model mapping, but in general it encapsulates what entities from other systems caused the entity to be created in this system. The `$origin` property is retrieved from the `$origin` property in the share sink dataset by joining the entity’s system primary key with the `$generated_id` property.	No
`$last-modified`	Datetime	This is a timestamp that tells when the entity was last modified in the system. The property is optional, but it is best to collect this information from the system or as close to the system as possible.	No

Example: existing entity in system¶

{
  "_id": "0",
  "id": 0,
  "key": "a",
  "value": 10,
  "$last-modified": "~t2023-01-30T12:00:01Z"
}

Example: an entity that has previously been inserted by Sesam¶

{
  "_id": "0",
  "id": 0,
  "key": "a",
  "value": 10,
  "$origin": "~:bar-person:123",
  "$last-modified": "~t2023-01-30T12:00:01Z"
}

The input to the connector¶

The dataset must use the following naming convention: SYSTEM-DATATYPE-transform, e.g. foo-person-transform.

The dataset represents the desired state of the target system. The connector will make sure that the system reflects the desired state. Entities in the dataset should have the same structure as the output of the connector. Namespaces have been removed from the properties, but not from the _id property value. The properties have the same names, data types and structure as in the collect dataset.

If the source entity consists of one or more entities from the system, then it must be split into an entity for each of the entities. The _id property must use the system specific enriched _id, e.g. if foo-person:123 (it must be namespaced) and foo:person:124 has been merged then one entity must be output for each of them.

If the source entity consists of no entities from the target system, then then output one entity. The _id property must be copied as-is from the source entity. This _id that will be treated as the origin of the inserted entity.

Properties¶

Property	Datatype	Description	Required
`$ids`	List<NI>	The property must contain `_id` and all other ids that may have been emitted as an `_id` for this object.	Required during insert. Optional for updated and deletes.
`$based_on`	Dict<String,Any>	The entity must contain the `$based_on` property, which should be a dict with property value pairs for all system raw properties that should be written back to the system. The `$based_on` property should be a subset of the properties from the connector output dataset as it should include the property values that was collected for this particular version of the entity. The property will be used to check if there has been modifications made to the entity in the target system after it was collected. The property can be used for preventing lost writes and avoiding unneccesary updates.	Required for updates and deletes. Disallowed for insert.
`$based_on_properties`	List<String>	This property enumerates the properties that are considered for `$based_on` comparison. The string values can be property names or a path of properties separated by a `.` character.	Required for updates and deletes. Disallowed for insert.
`$replaced`	Boolean	If set to true then it is not a real delete, but rather the entity got a new `_id`. This property is produced by the merge source.	Optional for deletes. Disallowed for update and insert.
`$origin`	Opaque	The share pipe will just pass through the `$origin` property to its sink entity. It is an error if the operation is an insert and the property is missing. See the description of `$origin` in the section Output of the connector above for more information on how it is used.	Optional if the entity originated in this particular system.

Example: insert¶

This entity does not have a system primary key, i.e. the id property, and will result in an insert into the system.

{
  "_id": "bar-person:1",
  "_deleted": false,
  "$ids": [
    "~:bar-person:1"
  ],
  "key": "a",
  "value": 10
}

Example: $replaced=true¶

The entity with this _id has been merged into another entity. The $replaced property and the _delete property was created by an upstream merge source and this must be communicated downstream to the dataset.

{
  "_id": "bar-person:1",
  "_deleted": true,
  "$replaced": true
}

Example: update¶

The properties in $based_on is different from the properties on the entity, so the entity will be updated in the system accordingly.

{
  "_id": "foo-person:0",
  "_deleted": false,
  "$based_on": [
    "id": 0,
    "key": "a",
    "value": 10
  ],
  "id": 0,
  "key": "a",
  "value": 20
}

Example: delete¶

The entity has been marked as deleted and will therefore be deleted in the system.

{
  "_id": "foo-person:0",
  "_deleted": true
}

Injected variables¶

Any strings on the form {{@ foo @}} in the non-expanded connector configuration represent Jinja variables that are injected into the configuration by a tool such as sesam-py. With some exceptions, most of the variables that support this must be listed in a connector’s manifest to be made available. Some of these variables are always available and do not need to be specified anywhere, such as datatype.

The table below lists the supported variables.

Overview¶

Variable	Type
account_id	String
base_url	String
connected_ts	String
<connector>_webhook_dataset	String
datatype	String
is_fullsync	Boolean
parent	String
service_url	String
system	String
token_url	String

Authentication-specific variables¶

The values for these variables are retrieved from the output of the Consumer portal for a given tenant.

The account_id Jinja variable can be used to inject the ID of the account that a tenant has connected to a system with in the Consumer portal.

The connected_ts Jinja variable injects the timestamp for when an entity type/datatype has been enabled in the Consumer portal.

The is_fullsync Jinja variable (EXPERIMENTAL) injects a boolean depending on whether a datatype has set fullsync to true or false by the user.

System-specific variables¶

The system Jinja variable is always available and injects the name of the system (for example “hubspot”, “wave” …)

Datatype-specific variables¶

The datatype Jinja variable is available for any configuration that belongs to a datatype and injects the name of the datatype. Datatypes in the manifest can also be set to use specific properties:

The parent Jinja variable is replaced with the value of the parent property set for a datatype.

Properties from connector configuration¶

Properties from a provided connector configuration can also be injected.

The token_url Jinja variable injects the URL of an endpoint that grants an OAuth2 access token.

The base_url Jinja variable injects the base URL of the API for the system.

Service API access¶

Setting requires_service_api_access to true in the manifest signals that any occurrences of the service_url Jinja variable should be replaced with “$ENV(service_url)”, and a JWT granting access to the service API is added as a secret to the connector’s system. The secret can then be used in the config with $SECRET(service_jwt).

Webhooks¶

Setting use_webhook_secret to true in the manifest signals that a secret intended for validating incoming requests to a receiver endpoint should be added to the system. The write permissions on all receiver endpoints that end with -event in this connector will also be set to group:Anonymous. This is meant to be used with the validation_expression in the HTTP endpoint source.

Setting <connector>_webhook_dataset under additional_parameters in the manifest signals that any occurrences of the <connector>_webhook_dataset Jinja variable should be replaced with “$ENV(<connector>_webhook_dataset)”.

Other parameters¶

Any parameter and its value can be specified under the parameters section of a datatype in the manifest, replacing any occurrence of that parameter in the configuration with the given value. For example, we can have a datatype contact that has this configuration in the manifest:

{
  "datatypes": {
    "contact": {
    ...
      "parameters": {
        "foo": "bar"
      }
    }
  }
}

This indicates that all occurrences of {{@ foo @}} in the contact template should be replaced with bar. Boolean values are also supported.

Injected configuration¶

In addition to injecting Jinja-type variables directly into the configuration (see the above section), certain properties in the expanded pipe configurations can be set with the connector configuration and the pipe metadata.

Overview¶

Property	Type
supports_since	String
sync_frequency	String
webhook	Boolean

Pump properties¶

Setting metadata.supports_since on a pipe template will modify the pump’s schedule_interval (if it is a collect pipe). By default, collect pipes run at a schedule of every 300 seconds. If metadata.supports_since is set to true, the pump will be set to run every 10 seconds instead.

Setting datatypes.<datatype>.sync_frequency to "slow" on a given datatype in the manifest will set the pump of the collect pipe to run only once per day at midnight.

Setting datatypes.<datatype>.sync_frequency to "weekly" on a given datatype in the manifest will set the pump of the collect pipe to run only at 00:00 on Mondays.

Setting datatypes.<datatype>.webhook to true on a given datatype in the manifest signals that the datatype supports webhooks, and the collect pipe will be set to run on a slower schedule (default is once per hour).

A custom value for the schedule_interval on the input pipe for a given datatype can be set under datatypes.<datatype>.schedule_interval in the manifest. This takes precedence over all the above options.

Injected secrets¶

Some variables are injected as secrets. Those variables typically contain sensitive information, like api keys and oauth2 tokens.

Api key¶

For connectors that use api_key authentication, the key will be injected as a system secret called api_key.

Oauth2¶

For connectors that use oauth2 authentication, the following secrets will be injected as system secrets:

Property	Description	Type
`oauth_access_token`	The initial oauth2 access token obtained during onboarding.	String
`oauth_client_id`	The client id as provided in the connector configuration.	String
`oauth_client_secret`	The client secret as provided in the connector configuration.	String
`oauth_refresh_token`	The initial oauth2 refresh token obtained during onboarding.	String

Tripletex authentication¶

Tripletex uses a custom style of authentication, and the following secrets will be injected as system secrets:

Property	Description	Type
`consumer_token`	The consumer token as provided in the connector configuration.	String
`employee_token`	The employee token as obtained during the onboarding.	String

SuperOffice authentication¶

SuperOffice uses a variant of Oauth2 authentication, and their keep alive key will be injected as so_ticket in addition to the regular oauth secrets.

Connectors

Connector configuration