1. Home
2. Sesam Hub
3. Documentation

Documentation

Important

Features marked with “Experimental” can change or disappear without warning, do not use them in production.

• Quick Reference
  • Configuration
  • DTL Transform Functions
  • DTL Expression Functions
  • Entity model
• Project kickstart
  • Planning a project
    • Identify master data
    • Identify master data sources
    • Identify globals
    • Connect data types to the globals
  • Setting up an environment
    • Subscriptions
    • Network connectivity
    • Development environment
    • Security
  • Creating a dataflow
    • Collect
    • Enrich
    • Connect
    • Transform
    • Share
• Building blocks
• Data Transformation Language (DTL)
• Data management
  • Patterns
    • Enrich Patterns
    • Connect Patterns
    • Transform Patterns
    • Generic Patterns
  • Data management
    • Summary
    • Data handling philosophy
    • Global datasets
    • Namespace and namespaced identifiers
    • Naming conventions
    • Workflow for transforming data in Sesam
    • Tips for global datasets
    • How to do global datasets in Sesam
    • Recipe for generating global datasets
    • Additional Sesam tips
  • Time-based masterdata management
    • Summary
    • Claims
    • Last modified value
    • Masterdata management
    • DTL examples
  • Entity Data Model
    • Reserved fields
    • Special fields
    • Standard types
    • Extension types
    • Mixed type ordering
• Data synchronization
  • API Best Practices
  • Bi-directional synchronization
    • Introduction
    • Benefits
    • Main patterns to be managed
    • Duplicate management
    • Preservation of source system data integrity
    • Race condition management
  • Cron Expressions
    • Syntax
    • Examples
  • Developer Extension Points
    • Custom Data Sources
    • Custom Data Transforms
    • Custom Data Sinks
    • Microservices in Sesam
  • JSON Pull Protocol
    • Requests
    • Response headers
    • Example: published endpoint
    • Example: dataset subset
  • JSON Push Protocol
    • Requests
    • Conflicts
    • Examples
  • Non-idempotency
    • The Challenge
    • Piece of Advice
  • Patterns
    • Collect Patterns
    • Transform Patterns
    • Share Patterns
  • RDF Support
    • Introduction
    • RDF input
    • RDF output
  • Source Datatype Mapping
    • Introduction
    • SQL systems
• Connectors
  • The Connector Contract
    • Output of the connector
    • The input to the connector
    • Injected variables
    • Injected configuration
    • Injected secrets
  • Configuring connectors
    • Properties
    • Properties in the auth object for API key based connectors
    • Properties in the auth object for OAuth2 based connectors
  • Patterns
    • Authentication schemes
    • Combine all and webhook events
    • Continuation
    • Datatype naming
    • Datatypes should follow the API
    • Datatypes with lists of linked objects
    • Duplicate hops block
    • Connector asymmetry
    • Open to extension closed for modification
    • Pagination
    • Parameterized datatypes
    • Receiving webhook events
    • Registering webhooks
    • Share template extension points
    • Single share
    • Two step collect
    • Use datatype specific system operations
    • Use connector templates
    • Deletion tracking of webhook events
• Operations
  • Security
    • Introduction
    • Subscriptions, Users, Roles and Permissions
    • Role-based access overview
    • Sesam node runtime
    • The secrets manager
  • Metrics and monitoring
  • Pump Execution Dataset
    • The pump-started entity
    • The pump-completed entity
    • The pump-failed entity
    • The read-error entity
    • The write-error entity
    • The pump-offset-set entity
  • Configuration Management
    • Using Sesam with a version control system
    • Testing a Sesam configuration locally
    • Continuous integration
    • Automatic deployment
  • Self-hosted service
    • Introduction
    • Hardware requirements
    • Software requirements
    • Firewall requirements
    • Installation
  • Self-hosted on Kubernetes
    • Introduction
    • Hardware requirements
    • Software requirements
    • Run Sesam in Google Kubernetes Engine(gke)
    • Install Sesam and other needed services to the cluster
    • Install the helm-charts
  • Debugging
    • Changing the loglevel via the “/api/logs” api endpoint
    • Profiling all Sesam API calls
    • Profiling one specific Sesam API call
    • Profiling a pump run
    • Using the “/api/status/stacktrace” api endpoint
    • Sending the SIGUSR1 signal to the sesam node process
    • Checking if the health-checker thread in the node has created a stacktrace
    • Using the “Instance configuration” gui
    • Upload a full node-config with a node metadata entity
    • Writing a “/sesam/node/data/startup_options.json” file
    • /api/status/heap
    • /api/status
    • /api/status/debugmallocstats
  • Behind the scenes
    • Introduction
    • Single machine service
    • Clustered service
    • Monitoring
    • Isolation
    • Firewall
    • Termination
    • Secure Development policy
  • Troubleshooting
    • Common issues with data synchronization
    • Solutions
• Data platforms
  • Sesam in Microsoft Azure
    • Reference Architecture
    • Using Azure Data Factory with Sesam
    • Using Azure Synapse Analytics with Sesam
  • Sesam in Google Cloud Platform
    • Reading from BigQuery
    • Writing to BigQuery
• Tools
  • Management Studio
    • Introduction
    • Overview
    • Browse
    • Pipes
    • Systems
    • Flows
    • Settings
    • User accounts
  • Sesam Client
    • Introduction
    • Pre-requisites
  • Consumer Portal
    • Authentication
  • GDPR platform
    • Getting started
    • Platform configuration
    • Datatypes and purposes configuration
    • The GDPR data access portal
    • Developer documentation
    • Changelog
  • Connector Tools
    • Introduction
    • Pre-requisites
• Service Configuration
  • General
  • Environment variables
  • Conditional properties
  • Service metadata
  • Properties
  • Pipes
    • Prototype
    • Batching
    • Properties
    • Example configuration
    • Pipe Features
  • Systems
    • Prototype
    • Properties
    • Type of systems
• Service API
  • Introduction
    • Examples
  • Instance API Reference
  • Portal API Reference
  • Swagger Files
  • Config groups
  • Metrics endpoint
• Deprecated features
  • DTL Functions
  • Sources
    • The Kafka source
    • Diff datasets source
  • Transforms
    • The properties to CURIEs transform
    • The URIs to CURIEs transform
    • The lower keys transform
    • The upper keys transform
    • The undirected graph transform
  • Sinks
    • Deprecated Properties
    • The Kafka sink
  • Systems
    • Deprecated Properties
    • The Kafka system
  • The RDF registry
    • RDF registry items
  • RDF input
    • The URIs to CURIEs transform
  • RDF in transforms
    • Accessing CURIEs properties
    • CURIE expansion in DTL
  • Pipes
    • Short-hand configuration

previous

Software channels

next

Quick Reference