Collect data¶

In this tutorial we will connect Sesam to two different data sources: your newly created Hubspot account and the Norwegian Central Coordinating Register for Legal Entities, “Enhetsregisteret”. Later in Getting Started we will use the data imported from Enhetsregisteret to improve on the data quality from HubSpot before sending the data to back to HubSpot.

After having succesfully connected to these providers you will create inbound pipes for each relevant entity type we want to work with.

Objectives:

After you complete this tutorial you will have learned the following:

How to create systems
How to create inbound pipes
Import data from a data source into sesam

Remember:

To create a HubSpot app developer account here
To set up a test account
To aquire an API key

Import data to HubSpot¶

In order to import data into Sesam we first need to make sure that HubSpot contains the data we want to import. Therefore, the first step is to populate HubSpot with some data by following the steps below:

Download the company data and contact data and save the csv files locally
Log into HubSpot and navigate to your Companies section, found under Contacts in the top menu.
Click Import on the right hand side of the page
Click Start an import and select File from computer and click Next
Select Multiple files with associations and click Next
Select Companies and Contacts and click Next
Upload learn-hubspot-company.csv under Company and learn-hubspot-contacts.csv under Contacts (don’t click on the “This file includes a … column” boxes) and click Next
Select Company ID as common column header and select Company as the object it is unique for and click Next
Select Don’t import data in unmapped column and click Next
Select Don’t import data in unmapped column again and click Next
Agree to the terms and click Finish import

The imported contacts and companies can be found under the Contacts tab.

Create systems¶

In Sesam, a system is a representation of the connection between Sesam and the outside world.

The Hubspot system¶

First, we will start by adding a new system for the HubSpot connection.

Follow the below steps in order to add HubSpot as a system in Sesam:

In the Sesam portal:

Navigate to Datahub > Variables and add your HubSpot API key as a Secret named “hubspot-api-key”
Navigate to Systems
Click New system
Paste and save the configuration below

{
  "_id": "hubspot",
  "type": "system:rest",
  "headers": {
    "Authorization": "Bearer $SECRET(hubspot-api-key)",
    "Content-Type": "application/json"
  },
  "operations": {
    "get": {
      "method": "GET",
      "url": "{{ properties.url }}&"
    },
    "get_company": {
      "method": "GET",
      "url": "companies/{{ id }}?properties=about_us,address,city,country,description,domain,founded_year,is_public,linkedin_company_page,name,numberofemployees,state,timezone,website,zip&associations=contacts,companies,deals,tickets,products,quotes&"
    },
    "update": {
      "method": "PATCH",
      "payload-type": "json",
      "url": "companies/{{ properties.id }}"
    }
  },
  "rate_limiting_delay": 60,
  "rate_limiting_retries": 3,
  "url_pattern": "https://api.hubapi.com/crm/v3/objects/%s",
  "verify_ssl": true
}

The Enhetsregistret system¶

Now we can add our second system, the “Enhetsregisteret” system.

In the Sesam portal:

Navigate to Systems
Click New system
Paste and save the configuration below

{
  "_id": "enhetsregisteret",
  "type": "system:microservice",
  "docker": {
    "image": "sesamcommunity/learn-sesam-crm:v1.1.2",
    "port": 5000
  }
}

Note

The connection to Enhetsregisteret is a mock connection in this tutorial. The data you will import to Sesam is actually test data generated for the purpose of this tutorial. The connections, as well as the data itself, are very much like how it might look in a real world scenario and therefore well serves the purposes of this tutorial.

After having successfully created both systems, you are now ready to move onto the next step of this tutorial, the creation of inbound pipes.

Create inbound pipes¶

“Inbound pipes” is the naming convention used for pipes that receive their data from a source system.

The HubSpot inbound pipe¶

The first inbound pipe we want to work on is the pipe that connects to our hubspot system. We want to pull in the company data that exists inside the CRM provider. Follow the steps below to create your inbound pipe hubspot-company-collect:

Navigate to Pipes
Click New pipe
Paste and save the configuration below
Click Start to ensure your pipe runs
Click refresh to see number of entities processed (should be 10). You can also see them in the pipe’s Output tab.

{
  "_id": "hubspot-company-collect",
  "type": "pipe",
  "source": {
    "type": "rest",
    "system": "hubspot",
    "id_expression": "{{ id }}",
    "operation": "get",
    "payload_property": "results",
    "properties": {
      "url": "companies?properties=about_us,address,city,country,description,domain,founded_year,is_public,linkedin_company_page,name,numberofemployees,state,timezone,website,zip&associations=contacts,companies,deals,tickets,products,quotes"
    }
  },
  "add_namespaces": false,
  "namespaced_identifiers": false
}

The Enhetsregisteret inbound pipe¶

The final thing to do is to pull in the company data from Enhetsregisteret by creating a pipe that connects to our enhetsregisteret system. Again, follow the below steps to create your inbound pipe enhetsregisteret-company-collect:

Navigate to Pipes
Click New pipe
Paste and save the configuration below
Click Start to ensure your pipe runs
Click refresh to see number of entities processed (should be 10). You can also see them in the pipe’s Output tab.

{
  "_id": "enhetsregisteret-company-collect",
  "type": "pipe",
  "source": {
    "type": "json",
    "system": "enhetsregisteret",
    "url": "/enhetsregisteret"
  },
  "transform": {
    "type": "dtl",
    "rules": {
      "default": [
        ["copy", "*"],
        ["add", "_id", "_S.orgnr"]
      ]
    }
  },
  "add_namespaces": false,
  "namespaced_identifiers": false
}

When done you should have 10 entities in the output of each of the two inbound pipes.

Data Synchronization

Share data