Free feature
Change Tracking¶
Inside Sesam, entities flow between datasets before being sent to their corresponding target system. This flow of entities is controlled by pipes, and one of the tasks of pipes in Sesam is to make sure that no unnecessary entities are processed.
Before a pipe writes an entity to a dataset, it will compare the new version of the entity to the last version of that entity the pipe produced. If the two versions are different, i.e. the entity has been updated, the dataset will be updated and the new entity will be placed at the front of the processing queue. This way the next pipe in line knows this entity has been updated and needs reprocessing. If the two versions are identical, the sink will drop the new entity and the dataset will not be updated. This way we ensure that only data that needs reprocessing will be reprocessed.
The comparison is done by comparing the new and the old entity’s _hash values.
Criteria for posting entities to sink dataset
New
_idvalue - The entity did not exist beforeChange in entity metadata - all non reserved fields
Change in entity delete status - the entity’s
_deletedstatus changes
Each entity version written to a dataset will receive an _updated value. An entity’s _updated value, unique to each dataset, is an incremental counter assigned to each entity version when they are stored in the target dataset. When a pipe’s pump has successfully finished, the pipe will store the last (newest) entity’s _updated value as the pipe’s since value, thereby keeping track of the entities that have already been processed.
Good to know:
Change tracking is a built-in feature in Sesam and will always act in the background. You could however easily make a pipe reprocess all or some entitites in the source dataset by changing the pipe’s since value in the following ways:
Resets can be done automatically by defining a rescan property for the pump.
Resets can be done manually through the reset function in the management studio pipe menu.
Since changes can be done manually through the Update last seen function in the management studio pipe menu.