Free feature
Automatic Reprocessing¶
When enabled, Automatic Reprocessing will detect when a pipe has fallen out of sync and needs to be reset. When enabled in a pipe, or in service metadata, the pipe will automatically reset itself and perform a full rescan – making sure that it is no longer out of sync. In some situations it may rewind a bit instead of doing a full rescan. Both actions will put the pipe back in sync.
Use case¶
Automatic Reprocessing is useful when pipes fall out of sync. There are many reasons why a pipe may fall out of sync, including:
The configuration may change
Datasets may be deleted and then recreated
Sources may be truncated
Data may be restored from a backup
Joins to new datasets can be added, or new datasets may be added as pipe inputs
Datasets that are hop-ed to by a pipe may be deleted.
When a pipe falls out of sync the pipe should be reset, and it should perform a full rescan to get a new view of the data.
When enabled, Automatic Reprocessing will automatically detect that the pipe is out of sync and reset it.
How to enable¶
By default, a pipe will not reset automatically if it goes out of sync, but it will maintain a list of effected datasets. You can set the property reprocessing_policy so that resets happen automatically.
Properties¶
Property |
Type |
Description |
Default |
Req |
|---|---|---|---|---|
|
Enum<String> |
Specifies the policy that the pipe uses to decide if a pipe needs to be reset or not. The default policy for all pipes can be set in service metadata.
|
|
No |