Working on a Sesam project¶

Table of Contents

Setting up a new Sesam project
Using Git in a Sesam project

Setting up a new Sesam project ¶

When setting up a new Sesam project there are several tasks that needs to be done in order to ensure a good working environment for the project and its team. One thing is the Sesam installation itself, but in every software development project, concepts such as version control, continuous integration (CI) testing and automatic deployment are key factors for a successful project.

As a product, Sesam in itself does not contain any features that supports versioning of its configuration, local or automated CI testing and automatic deployment, but it provides a set of features (mostly through its API) that can be used in regular development tools or scripts for this purpose. This is flexible in terms of requirements set by the various customer, whose development platforms might vary. It does, however, mean that some time must be spent at the beginning of the project. In this document we will go through how to setup a Sesam project with these features, with examples and links to more in-depth documents.

Set up version control ¶

Sesam will always store the previous version of a pipe or system that has been uploaded to the node, in the same way it stores the previous data for an updated entity in a dataset. These entities can be found in the ''system:config'' dataset, remember to check 'system' under 'Origin' to display the system datasets. However, this only lets you view the versions of the uploaded configurations, there is nothing you can do with these entities. As so, this cannot be regarded as a proper version control of the configuration (although you can always use this dataset to retrieve the current or a previous configuration of a pipe or system).

A Sesam configuration should hence always be stored in a separate version control system such as Git, Concurrent Versions System (CVS), Subversion, TeamCity, Mercurial or other. Git is the most commonly used and will be used in the examples in this document, but in terms of a project, the overall handling should be the same.

We will describe how to set up a version control repository for Sesam with Git in mind, but the overall way of how to work with branches and flows in another version control system should be the same.

Set up a new Git project / repository for Sesam ¶

Given that you already have a directory with a Sesam configuration that you want to put into a repository, there are some additional files and folders that needs to be added in order to add it to a Git repository.

The optimal directory structure of a Sesam Node project in Git should look like this:

my-project-directory
  ├ node
  | ├ expected
  | ├ pipes
  | ├ systems
  | └ variables
  ├ README.md
  ├ LICENSE
  ├ .gitignore
  └ ++

Set up local tests ¶

In a normal workflow you should test your local changes before committing them to a repository. For setting up local testing of a Sesam configuration we recommend using the Sesam client, which is a command line tool for interacting with a Sesam service instance. To install and configure the Sesam client, read this document.

Testing a Sesam configuration should be done by running all the pipes in your local node until no pipes has any entities left in their queues, except the output pipes (which should either be disabled or unable to send data to their respective systems). Doing this manually could hence be a tedious job, especially if your configuration consists of numerous pipes, having to repeatedly start all or several pipes. The output of the endpoint pipes should then be verified against the expected output of the pipes. The Sesam client includes a nifty feature for this, both for running the pipes to their conclusion and verifying the end result.

Configuring tests¶

However, for this to work, you have to configure your Sesam configuration with a test-file for each endpoint pipe. In the same directory as you have your 'pipes' and 'systems' folders, you need to add a new folder named 'expected', this will be the folder that contains the test files and the expected result for each pipe.

For each endpoint pipe in your Sesam configuration, you need to add two files (replace <name_of_pipe> with the name of the pipe):

<name_of_pipe>.json

<name_of_pipe>.test.json

The file with the .json extension is the file that shall contain the expected result. The file with the .test.json extension should contain the test configuration for that pipe, the available settings for this file are listed below:

Property	Description	Type	Required	Default
`_id`	Name of the test.	`string`	No	Name of the `.test.json file`
`type`	Config type so that this later can just be part of the rest of the config.	`string`	No	Test
`description`	A description of the test.	`string`	No
`ignore`	If the output should be ignored during tests.	`boolean`	No	`false`
`endpoint`	If the output should be fetched from a published endpoint instead.	`string`	No	By default the json is grabbed from `/pipes/<my-pipe>/entities`
`stage`	In which pipe stage to get the entities (source/before-transform/after-transform/sink).	`string`	No	By default the stage is `sink`
`file`	File that contains the expected results.	`string`	No	Name of the .test.json file without .test (e.g. foo.test.json looks for foo.json).
`pipe`	Pipe that contains the output to test.	`string`	No	Name of the .test.json file without .test (e.g. foo.test.json looks for foo.json).
`blacklist`	Properties to ignore in the output.	`Array of strings`	No	`[]`
`parameters`	Which parameters to pass as bound parameters. Note that parameters only works for published endpoints.	`Object`	No	`{}`

Example:

{
    $ cat foo.test.json
    {
          "_id": "foo",
          "type": "test",
          "file": "foo.json"
          "blacklist": ["my-last-updated-ts"],
          "ignore": false
    }
}

DTL parameters¶

If you need to pass various variations of bound parameters to the DTL, you just create multiple .test.json files for each combination of parameters.

Example:

{
    $ cat foo-A.test.json
        {
          "pipe": "foo",
          "file": "foo-A.xml",
          "endpoint": "xml",
          "parameters": {
            "my-param": "A"
          }
        }

    $ cat foo-B.test.json
        {
          "pipe": "foo",
          "file": "foo-B.xml",
          "endpoint": "xml",
          "parameters": {
            "my-param": "B"
          }
        }
    }

This will compare the output of /publishers/foo/xml?my-param=A with the contents of foo-A.xml and /publishers/foo/xml?my-param=B with the contents of foo-B.xml.

Internal properties¶

All internal properties except _id and _deleted are removed from the output. Entities that has _deleted set to false will also be removed.

Endpoints¶

By default the entities are fetched from /pipes/<my-pipe>/entities, but if endpoint is set it will be fetched from /publishers/<my-pipe>/<endpoint-type> based on the endpoint type specified. Note that the pipe needs to be configured to publish to this endpoint.

Example:

{
  "_id": "foo",
  "type": "test",
  "endpoint": "xml",
  "file": "foo.xml"
}

This will compare the output of /publishers/foo/xml with the contents of foo.xml.

Example:

{
  "_id": "foo",
  "type": "test",
  "endpoint": "json",
  "stage": "source"
}

This will compare the output of /pipes/foo/entities?stage=source with the contents of foo.json, useful when the pipe's sink strips away the "_id" property for example.

Running tests locally¶

To test your Sesam configuration locally, run the following commmand:

sesam -vv test

If you haven't configured up the tests correctly or there are endpoint pipes that doesn't have any corresponding test file, you will be notified. If so, fix the missing tests and then run the commmand again. If the tests runs ok, you will get a message that all the tests has passed. If any test failed, you will be notified which test / pipe that failed and get a comparision of the expected result and the received result.

Set up automatic CI testing ¶

Automatic tests are needed to verify that your pull request does not break any existing pipes/flows inside sesam. To perform these types of tests we need to set up automatic tests. Since there are a few different CI/CD tools, we are going to explain a few of the most common ones.

In order to fully be able to have an automated CI test of your Sesam configuration, you need to have a designated CI node. The only purpose of the CI node is to provide an environment to test changes to the configuration so that it doesn't break.

When setting up automatic CI testing for a Sesam project, the following check should be required for the test to pass:

Running, validating and passing a test of the configuration against the CI node.

Another check that should be considered is:

Only pull requests (PR) that are approved by another person in the team should be valid (this is however, )

CI node¶

In order to be able to test your Sesam configuration across the project team, you need to have a CI node. This node's sole purpose is to provide a Sesam environment to test changes to the configuration against. Tests will be run in the same way as you will test your configuration locally, except that it should be initiated from your automatic CI testing system instead.

It is important that when running testing against the CI node, only one test should run at the same time. When running a test against a CI node with the Sesam client, the configuration will be overwritten, which will cause a running test to fail or not to finish. Setting up your automatic CI testing, you need to have this in mind.

The CI node should be unable to write any data to another system, bear this in mind when configuring up the node. As the least safety regarding this, the node should not contain any variables or secrets necessary to connect to a system that the Sesam configuration usually will send data to.

Usually, the CI node could be a smaller instance than the production node, as the data used in the tests should not be of a considerable size.

Jenkins¶

This section describes how to set up Jenkins build with GCloud.

Jenkins is a CI/CD tool that does not support single build pipeline. The reason for the need of single build pipeline is that we upload the node config to a single node, if there are mulitple builds running at the same time there will be pushed multiple configs to the one node, which will result into tests not completing.

To set up builds in jenkins, you will need to add a few file to your repository my-project-directory

my-project-directory
  ├ deployment
  | ├ jenkins
  | | └ jobs
  | |   └ build
  | |     ├ dm-pod.yaml
  | |     └ Jenkinsfile
  | └ sesam
  |   ├ cloudbuild.yaml
  |   ├ Dockerfile
  |   └ Readme.md
  ├ node
  | └ ++
  └ ++

dm-pod.yaml:

Describes what type of container that should be used in the build process.

apiVersion: v1
kind: Pod
spec:

  containers:
  - name: sesam-ci-container
    image: eu.gcr.io/<your_gcr_repo>/sesam:<version_of_sesam_client>
    tty: true
    command:
    - cat
    resources:
      limits:
        memory: 6Gi
        cpu: 1.7

Jenkinsfile:

The Jenkinsfile contains the stages that are supposed to run when the tests are running. The three default stages are:

Set environment variables for container
Verify usage of correct Sesam client version.
Running the tests and printing scheduler logs to see error messages in output.

#!groovy

pipeline {
    options {
        disableConcurrentBuilds()
    }
    agent {
        kubernetes {
            label "dm-${BRANCH_NAME}-${BUILD_ID}"
            defaultContainer 'jnlp'
            yamlFile 'deployment/jenkins/jobs/build/dm-pod.yaml'
        }
    }
    environment {
        Sesam_CI_node_jwt = credentials('Sesam_CI_node_jwt')
    }
    stages {
        stage('Set Sesam env vars') {
            steps {
                script {
                    env.Sesam_CI_node = "datahub-****.sesam.cloud"
                }
            }
        }
        stage("Verify Sesam version") {
            steps {
                dir('') {
                    container('sesam-ci-container') {
                        sh "/./sesam -version"
                    }
                }
            }
        }
        stage("Run Sesam tests") {
            steps {
                dir('') {
                    container('sesam-ci-container') {
                        sh "export NODE='${env.Sesam_CI_node}'; export JWT='$Sesam_CI_node_jwt'; cd node && /./sesam -vv test  -print-scheduler-log"
                    }
                }
            }
        }
    }
}

The files under the sesam folder here describes the files that should exist in the repository where jenkins is configured. Usually you do not have access to this repository, but you will need to provide these files.

cloudbuild.yaml:

cloudbuild.yaml A build config file defines the fields that are needed for Cloud Build to perform your tasks. You'll need a build config file if you're starting builds using the gcloud command-line tool or build triggers. You can write the build config file using the YAML or the JSON syntax.

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', 'eu.gcr.io/<your_gcr_repo>/sesam:latest',
      '-t', 'eu.gcr.io/<your_gcr_repo>/sesam:1.16.1',
      '.'
    ]
images:
  - 'eu.gcr.io/<your_gcr_repo>/sesam'
tags:
  - '1.16.1'
  - 'latest'

Dockerfile:

The dockerfile describes the contianer that should run when the build process is executed. This container should be deployed to the repository that is used

FROM debian:9.9-slim
MAINTAINER [Your name] "your.email.address@domain.no"

ARG SESAM_CI_VERSION=1.16.1

SHELL ["/bin/bash", "-c"]

RUN apt-get update
RUN apt-get install -y wget

RUN set -x
RUN wget -O sesam.tar.gz https://github.com/sesam-community/sesam-py/releases/download/$SESAM_CI_VERSION/sesam-linux-$SESAM_CI_VERSION.tar.gz
RUN tar -xf sesam.tar.gz
RUN rm sesam.tar.gz

This dockerfile builds a container with the sesam client that is needed to execute the build. Replace [Your name] with the name of the person responsible for the build process, alongside his or hers email-address.

Azure DevOps¶

Azure DevOps is a bit easier to set up with single build pipeline. You will need to add the following config to your Azure DevOps setup under Pipelines

# Sesam AzureDevops Pipeline

trigger: none

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: |
    wget -O sesam.tar.gz https://github.com/sesam-community/sesam-py/releases/download/$(sesam_cli_version)/sesam-linux-$(sesam_cli_version).tar.gz
    tar -xf sesam.tar.gz
    rm sesam.tar.gz
  displayName: 'Download Sesam CLI'

- script: ./sesam -version
  displayName: 'Verify Sesam CLI version'

- script: |
    export NODE='$(node)'
    export JWT='$(node_jwt)'
    cd node
    .././sesam -vv test  -print-scheduler-log
  displayName: 'Run Tests'

You will also have to add variables

sesam_cli_version = 1.16.1 (version of the CLI used in your project)
node              = datahub-***.sesam.cloud (the node url to the CI server used in your project)
node_jwt          = bearer ****** (jwt for the CI server used in your project)

Branch permissions are also needed to not be able to merge a Pull Request unless the tests have completed successfully. These permissions needs to be set under

Repos->Branches->More->Branch Policies->Add Build Policy

Use the default settings.

You will also need to turn on Require a minimum number of reviewers, and set it to 1 and Check for linked work items. This makes it Easier to trace and close the tasks/issues for the Pull Request.

These settings are required for your main branches develop and master.

Since the trigger parameter is set to none, the build process will only trigger on PR's. There is no need to build master and develop after merge.

Note if there is support for parallel builds on the agent pool you will need to disable this so that only one build process runs and the second build is queued up. This can be done by adding capability on the build agent. You will also need to add a this in the yaml file to enable this. Add user capabilities in the agent pool (key value pair), key = Limit and value = DisAbleParallel

Your yaml file:

pool:
  name: {agent pool name}
  demands: Limit -equals DisAbleParallel

Your configuration will end up being in your repository under the main directory:

my-project-directory
  ├ node
  | ├ pipes
  | ├ systems
  | ├ expected
  | └ ++
  └ azure-pipelines.yml

Set up automatic deployment ¶

Whether setting up automatic deployment of a Sesam configuration is a disputed theme. In normal usecases, you would like to have more control of when a release is deployed to a production environment, especially in larger or business critical installations. But if you decide upon setting up automatic deployment of your Sesam configuration, it can be done in several ways.

GitHub Autodeployer microservice¶

One way to easy set up automatic deployment of your Sesam configuration is to use the GitHub Autodeployer microservice. This is a microservice that you can configure in your Sesam node that at given intervals will check the configured Git repository for changes. If any changes to the repo is found, it will read the configuration from the repo and deploy it to the node.

In the configuration you can either specify a branch or a tag. Use tags when deploying a release branch with a version number (which should be a tag in the repo). If no tag is specified, the autodeployer will use the branch variable, which defaults to "master" if not set. Depending on the specified branch or tag, the autodeployer will compare the current Sesam configuration against the configuration in the repository, if any changes are found, the deployer will read the updated configuration from the repository and deploy it to the node.

WARNING! Any existing pipes and systems will be overwritten when the autodeployer deploys a new version to the node. Any pipe or system configuration in the node not existing in the branch will be removed.

Also note that the autodeployer only deploys a configuration, it does not do any other actions on the node, such as starting or resetting pipes. If any pipes need to be reset as part of the deployment for instance, the autodeployer will not perform any such task and this must be done manually.

Information on how to configure the GitHub Autodeployer microservice can be found at its corresponding GitHub page: https://github.com/sesam-community/github-autodeployer.

Using Jenkins, Azure DevOps or any other CD tools¶

Automatic deployment could also be done using the same tools you use for your automatic CI testing, like Jenkins or Azure DevOps. For this, you need to change the step for testing with a step for deploying the given branch. See the document about the Sesam client for the correct parameters to use.

Remember to add parameters to your configuration for which release version to deploy.

Using Git in a Sesam project ¶

Initial repository setup ¶

The initial repository should contain two main branches with an infinite lifetime. Parallel to the master branch, another branch should exist, called develop.

master

develop

We consider origin/master to be the main branch where the source code always reflects a production-ready state. We consider origin/develop to be the main branch where the source code always reflects a state with the latest delivered development changes for the next release. Some would call this the “integration branch”.

When the source code in the develop branch reaches a stable point and is ready to be released, all of the changes should be merged back into master branch and then tagged with a release number.

Therefore, each time when changes are merged back into master, this is a new production release by definition. We follow this practice very strictly.

Supporting branches

Next to the main branches master and develop, our development model uses a variety of supporting branches to aid parallel development between team members, ease tracking of features, prepare for production releases and to assist in quickly fixing live production problems.

Unlike the main branches, these branches always have a limited life time, since they will be removed eventually.

The different types of branches we may use are:

Feature branches

Release branches

Hotfix branches

Each of these branches have a specific purpose and are bound to strict rules as to which branches may be their originating branch and which branches must be their merge targets.

Branch Type	May branch off from	Must merge back into	Branch naming convention
Feature	develop	develop	Aything except master, develop, release-, or hotfix-
Release	master	develop and master	release-*
Hotfix	master	develop and master	hotfix-*

One exception to the rule here is that, when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of develop.

For information on how to set up the master branch in Git, read here. For information on how to set up the development branch in Git, read here.

Versioning of releases¶

You should use semantic versioning for any of your releases to production.

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible changes to the configuration,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes

Set up master branch¶

Based on the structure described above you should navigate to the project root (my-project-directory) and run the following command:

git init

Then your directory will become a Git repository (repo). After you've done this, go to your source control website (i.e. github.com). Here you will need to create a new repo under your organisation. Make sure that you don't initialize the repo from the website. When the repo has been created you should be presented with a url to use. (i.e. git@github.com:my_org/my_repo.git) Connect the Git repo to your local repo:

git remote add origin git@github.com:my_org/my_repo.git

Push your local repo to Git:

git push -u origin master

(Tip: Sometimes you need to first add and commit README.md file, to make your first push to remote repo.)

Set up development branch¶

Since we want to use the master branch as the production branch, we need to setup a new branch called develop to use for development. To do this we need to type the following in terminal:

git checkout -b develop

This creates a new branch called develop that mirrors master. To push it to Git:

git push --set-upstream origin develop

Now you should have two branches in Git. Before we go forward you should go to your repository settings (in Github or equal) and configure the default branch to be develop. After that you should set both master and develop branches as protected. This means that you won't be able to directly push commits to these branches. We want to force users to do that by creating pull requests.

More information about pull requests can be read in this document https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request.

Set up feature branches¶

Creation:

$ git checkout -b myfeature develop
Switched to a new branch "myfeature"

Incorporating a finished feature on develop :

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff myfeature
Updating ea1b82a..05e9557
(Summary of changes)
$ git branch -d myfeature
Deleted branch myfeature (was 05e9557).
$ git push origin develop

Tip: The --no-ff flag causes the merge to always create a new commit object, even if the merge could be performed with a fast-forward. This avoids losing information about the historical existence of a feature branch and groups together all commits that together added the feature.

Set up release branches¶

Creation :

$ git checkout -b release-1.0.0 master
Switched to a new branch "release-1.1.0"
$ ./bump-version.sh 1.1.0
Files modified successfully, version bumped to 1.1.0

(Here, bump-version.sh is a fictional shell script that changes some files in the working copy to reflect the new version. (This can of course be a manual change—the point being that some files change.) Then, the bumped version number is committed.))

$ git commit -a -m "Bumped version number to 1.1.0"
[release-1.2 74d9424] Bumped version number to 1.1.0
1 files changed, 1 insertions(+), 1 deletions(-)

Finishing a release branch:

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff release-1.1.0
Merge made by recursive.
(Summary of changes)
$ git tag -a 1.2

The release is now done, and tagged for future reference. To keep the changes made in the release branch, we need to merge those back into develop, though:

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff release-1.1.0
Merge made by recursive.
(Summary of changes)

This step may well lead to a merge conflict (probably even, since we have changed the version number). If so, fix it and commit. Now we are really done and the release branch may be removed, since we don’t need it anymore:

$ git branch -d release-1.1.0
Deleted branch release-1.1.0 (was ff452fe).

Set up hotfix branches¶

Creation:

$ git checkout -b hotfix-1.1.1 master
Switched to a new branch "hotfix-1.1.1"
$ ./bump-version.sh 1.1.1
Files modified successfully, version bumped to 1.1.1.
$ git commit -a -m "Bumped version number to 1.1.1"
[hotfix-1.1.1 41e61bb] Bumped version number to 1.1.1
1 files changed, 1 insertions(+), 1 deletions(-)

Finishing a hotfix branch:

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff hotfix-1.1.1
Merge made by recursive.
(Summary of changes)
$ git tag -a 1.1.1

Next, include the bugfix in develop too:

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff hotfix-1.1.1
Merge made by recursive.
(Summary of changes)

Important: The one exception to the rule here is that, when a release branch currently exists, the changes in the hotfix needs to be merged into that release branch, instead of develop.

$ git branch -d hotfix-1.1.1
Deleted branch hotfix-1.1.1 (was abbe5d6).

Working on a new feature/change ¶

Branching¶

When you want to start working on a new feature, you should start by creating a new feature branch. When checking out the new branch, make sure that you have the latest version of the source branch. Generally new feature branches should be checkout out from the develop branch. Generally we want feature branches to be named after the relevant task/issue id. You can read more about how to name the branches correctly in Branch naming.

git checkout master
git pull
git checkout -b <issue_id>

Now you have a feature branch to start working on. Next you should proceed to read about how to write commit messages.

Commit messages¶

Start the commit message with a task/issue id
Use the imperative mood in the subject line, as described here.

There are some simple rules to follow. A properly formed Git commit subject line should always be able to complete the following sentence:

If applied, this commit will <your subject line here>

For example, a commit message like "update the rdf:type in proarc-document pipe" will result in:

If applied, this commit will update the rdf:type in proarc-document pipe

Try to avoid having commit messages like: "Fixed bug with Y". This is a non-imperative form and when we apply the imperative mood to the text "Fixed bug with Y" the sentence will result into:

If applied, this commit will Fixed bug with Y.

An example of a commit message with a task/issue id:

AB-123: Update requirements to fix deprecation error

In this example AB-123 is the issue id. When this pattern is utilized, it makes it much easier to determine why a commit where applied regardless of branch.

Pull request¶

At this point you should have a feature branch with some changes that you would like to merge into your develop branch. If you've been working on your feature branch for a while, it might be a good idea to merge the develop branch back into your feature branch before creating the pull request.

git fetch develop
git merge develop

When doing this, you might encounter conflicts. To resolve these, go to the mentioned files and look to see what version of the code is the one that should be kept. Edit out the code that shouldn't be kept and add the files:

git add <my_file_with_conflict>
git merge --continue

When this is done, you should push your latest changes to github or similar and create a pull request with their GUI.

Deploy a new feature ¶

Creating a release¶

Release branches contain production ready new features and bug fixes that come from a stable develop branch. In most cases, the master branch is always behind the develop branch as new features will first be pushed to that branch. After finishing release branches, they get merged back into develop and master branches so that both of these branches eventually will match each other.

We can split a release into two different categories, minor releases and major releases. These two different release types are defined by how big the change to master is.

Usually you would have feature releases as minor releases, while major releases would include big changes like restructuring pipe-combinations and merge rules.

Hotfixes¶

Hotfixes are used to deploy critical changes to production. It also includes small fixes to pipes (as long as it is something that already is deployed to production*). When creating a hotfix you should branch off from master branch, merge into master and back to develop so that both of the main branches gets the update.

*Small fixes will often be forgotten and end up in the develop branch without being added to a release. This validates having small fixes/changes to pipes/systems as a hotfix and not only beeing added as a part of a release.

Tagging¶

Tags are a simple aspect of Git, they allow you to identify specific release versions of your code. You can think of a tag as a branch that doesn't change. Once it is created, it loses the ability to change the history of commits.

In a Sesam perspective we add tags if we need to revert to a previous version, if we figure out that a release or hotfix is not working as expected.

Tags are also a good way to have different versions of config in different environments. A good example of this is if multiple releases are done, but one version has not been tested to the full extent. You can run one tag in the staging environment, and another in the production environment.

For tags we use semantic versioning. You can read more about semantic versioning here semantic versioning.

Variables¶

Variable files are often added to Git so that we are able to track and keep control of existing environment variables. Environment variables should exist in the repository under the folder nodevariables. You should have 3 files:

-variables-dev.json

-variables-staging.json

-variables-prod.json

These three files should reflect what the variables are in your/the projects node environment. Changes/addition of environment variables should be added to Git with the feature you are editing or in the hotfix you are creating.

When creating a release you must remember to add the updated files to your release branch.

Secrets¶

Secrets should ideally be saved in a keymanager, and not in Git.

When you want to deploy all changes in develop into master¶

First off we will need to create a ticket for your release so we get a task number. This is done in your projects issuetracker. In this case the ticket created is named AB-2324.

When you are ready to deploy your changes to production, you will have to create a release to master.

This is done with:

git checkout master
git checkoub -b release-*.*.*
(creating release branch that is semantically versioned)
git checkout develop -- .
(checkout all files from the develop branch and add it to your current release-*.*.*. )

this will add all the expected files that you have in your expected folder as well.

you should now run tests to see if everything works as expected.

sesam -vv test

If the result of the test comes back as OK, you are ready to commit.

git add .
(adds all files)
git commit -m "AB-2324: add all files from develop to release-*.*.*"
(When using task number AB-2324 you will create a reference to the ticket and in some issuetrackers you will be able to see a link to the Pull request)
git push

You are now ready to create the pull request in your version control system. This will trigger your build process to trigger a new build. When your build has completed successfully, you are ready to merge your release branch into master.

When the merge is completed you can now tag your release in your version control system to release-..*

When you can't deploy everything in develop into master¶

When you can't deploy everything from develop into production, and you would like to release some feature that is completed. you will need to find the config files manually. You will need to figure out what pipes/systems that are ready for deploy, but you would still need to go through the same process as noted in the "When you want to deploy all changes in develop into master" stage.

git checkout master
git checkout -b release-*.*.*
(creating a branch based on master branch)

You will now have to have a list of the pipes/systems you would like to deploy.

Considering you are in the node folder:

git checkout develop pipes/<my_pipe_name> systems/<my_system_name>

this will only checkout the pipes/systems that you would like to be included in this release. Note that your tests will fail now, since you have not checked out the corresponding tests to the pipe you just checked out.:

git checkout develop expected/<my_pipe_name>.*
*(this will check out the two expected files that are in relation to the pipe you have checked out)*
sesam -vv test
(run the test to see if testresults are ok)

Remember to checkout the environment config files as well.

If everything is ok, you can now add and commit the files to your new release-branch.

git add .
git commit -m "AB-2324: adding specific files from dev to my new release-*.*.*"
git push

You are now ready to create the pull request in your version control system. This will trigger your build process to trigger a new build. When your build has completed successfully, you are ready to merge your release branch into master.

When the merge is completed you can now tag your release in your version control system to release-..*. You are now ready to merge back to develop.

Often you might end up having merge conflicts when you merge back to develop. You can read more about this in Resolve common problems.

Branch naming/release tagging ¶

Branch naming¶

When we're creating a new feature branch, we want the branch to be named after the relevant issue/task id. Lets say we have a ticket called AB-123. Then you would create your branch like this:

git checkout develop -b AB-123

Release naming¶

When you want to create a new release to deploy, we want releases to use semantic version numbers. This makes it easier to determine what type of change a release involves.

To determine the next version number, you can follow this diagram:

Resolve common problems ¶

Merging back to develop creates merge conflicts¶

When you have worked on a release, there will be cases when your develop and master branch diverges. Lets say you have not created a release in a long time. You will end up having a lot of new features in your develop branch that does not exist in master.

Even though new pipes and systems will not have a merge conflict, you will have cases where your global pipes have many new features in dev that does not exist in master. You will need to fix the release so that you only add the features you want to release. An example of this follows:

your-global-pipe-in-dev:

"datasets": ["dataset_foo", "dataset_bar", "dataset_baz", "dataset_foobar", "dataset_foobaz"]

While your global-pipe in master looks like:

"datasets": ["dataset_foo", "dataset_bar", "dataset_foobar"]

Your feature with "dataset_baz" is now finished and you will only want to release this, and not all the others that are not finished. You will have to do changes as a commit in the release branch to get the correct structure in your master branch.

And your global pipe should look like this:

"datasets": ["dataset_foo", "dataset_bar", "dataset_foobar", "dataset_baz"]

You can see that the order in your dev global pipe vs your master global pipe is diverging now. Since our Master branch is the Main branch, and develop is continually under development we will need to restructure develop to match the newest release.

dev (currently):
"datasets": ["dataset_foo", "dataset_bar", "dataset_baz", "dataset_foobar", "dataset_foobaz"]
master (after changes to release-branch)
"datasets": ["dataset_foo", "dataset_bar", "dataset_foobar", "dataset_baz"]

When this type of change is merged back to develop you will get merge conflicts that needs to be resolved. The order that is primary choice is the changes from master. Which results into dev looking like:

dev (after merge back from release branch):
"datasets": ["dataset_foo", "dataset_bar", "dataset_foobar", "dataset_baz", "dataset_foobaz"]
master (after changes to release-branch)
"datasets": ["dataset_foo", "dataset_bar", "dataset_foobar", "dataset_baz"]

You can see that the order is changed in develop to match what is in master.

We found a bug in recently merged PR¶

The following strategy will revert a merge commit. This can be used in any branch where you want to undo a merge.

git checkout develop -b revert/my_feature_branch

Now you will need to find the commit hash of the merge commit. This can be found with "git log". Then use the hash in the next command:

git revert -m 1 <hash of merge commit>

Now you have a branch that reverts the merge. Use that for a new pull request against develop. If you want to fix the feature you can start with following steps after you have merged the previous revert.

git pull develop
..
git checkout develop -b my_feature_branch
..
git revert -m 1 <hash of revert commit from earlier>

Now you have a branch where the reverted changes have been re-applied. Now you can continue working in the feature branch and fix the issues that required the revert in the first place. When your changes are done, you can treat this branch as a regular feature branch and create a new pull request to merge your changes.

We found a critical bug in production¶

When this happens, you most likely have two choices. Either revert the change (see We found a bug in recently merged PR or fix it directly in production with a hofix branch.

To fix it directly in production, use the following steps:

Create an new hotfix branch from master: git checkout master -b hotfix_for_my_feature.
Do your changes and commit it to the hotfix branch.
Create a PR for both master (production) and develop (to get the correct version for future development).

Table Of Contents

Working on a Sesam project¶

Configuring tests¶

DTL parameters¶

Internal properties¶

Endpoints¶

Running tests locally¶

CI node¶

Jenkins¶

Azure DevOps¶

GitHub Autodeployer microservice¶

Using Jenkins, Azure DevOps or any other CD tools¶

Versioning of releases¶

Set up master branch¶

Set up development branch¶

Set up feature branches¶

Set up release branches¶

Set up hotfix branches¶

Branching¶

Commit messages¶

Pull request¶

Creating a release¶

Hotfixes¶

Tagging¶

Variables¶

Secrets¶

When you want to deploy all changes in develop into master¶

When you can't deploy everything in develop into master¶

Branch naming¶

Release naming¶

Merging back to develop creates merge conflicts¶

We found a bug in recently merged PR¶

We found a critical bug in production¶