Datadog Integration

How to integrate Datadog with Connectware, including API key authentication, submitting machine metrics to the Metrics API, and shipping machine event logs to the Logs API.

This guide describes how to send shop floor data from Connectware to Datadog. You configure a service commissioning file that submits machine metrics to the Datadog Metrics API and ships machine event logs to the Datadog Logs API. A complete example file is available at the end of this guide.

Objectives

• Connecting Connectware to the Datadog intake APIs with API key authentication.

• Submitting a gauge metric per machine from the MQTT topic hierarchy.

• Shipping machine event logs with tags derived from the equipment hierarchy.

Prerequisites

To follow this guide, you will need the following:

• A running instance of Cybus Connectware.

• A Datadog account and an API key. The intake APIs authenticate with the API key alone, you do not need an application key.

• The Datadog site of your account, for example datadoghq.com or datadoghq.eu. You find it in the URL you use to log in to Datadog.

• Access to the Admin UI with sufficient user permissions.

• Basic knowledge of MQTT and the Connectware services concept (for example, service commissioning files, connections, and endpoints).

Connectware and Datadog Integration

Datadog ingests metrics and logs through HTTP intake APIs, authenticated with an API key in the DD-API-KEY request header. Connectware communicates with these APIs through the HTTP/REST connector, which adds the header to every request.

This guide uses the two intake APIs:

• Metrics API: POST /api/v2/series on the host api.<site> submits time series points that you can graph on Datadog dashboards and alert on with monitors.

• Logs API: POST /api/v2/logs on the host http-intake.logs.<site> ships log events to Datadog Log Management.

The two APIs live on different hosts, so the service uses two Cybus::Connection resources, one per intake host. Both use the same API key.

The MQTT topics in this guide follow an ISA-95-style equipment hierarchy (<enterprise>/<site>/<area>/<line>/<cell>). The mappings subscribe with wildcards across all levels, so any machine in the hierarchy is picked up without changing the integration. The topic levels become Datadog tags, so you can filter and group by site, area, and line in Datadog.

Datadog Connection Properties

We add the account-specific values as parameters to the service commissioning file, so you can set them when you install the service.

Do not worry about copying the service commissioning file snippets together into one, the complete example file is available at the end of this guide.

• datadogSite: The Datadog site of your account. For example, datadoghq.com for US1 or datadoghq.eu for EU1.

• datadogApiKey: The API key used for both intake APIs.

• metricName: The name of the submitted gauge metric. Defaults to machine.temperature.

• serviceName: The value of the service tag on metrics and logs. Defaults to shop-floor.

• topicRoot: The root of the MQTT topic hierarchy. Defaults to enterprise.

parameters:
  datadogSite:
    description: Datadog site of your account, for example datadoghq.com or datadoghq.eu
    type: string
    default: datadoghq.com

  datadogApiKey:
    description: Datadog API key used for the metrics and logs intake
    type: string

  metricName:
    description: Name of the submitted gauge metric
    type: string
    default: machine.temperature

  serviceName:
    description: Value of the service tag on metrics and logs
    type: string
    default: shop-floor

  topicRoot:
    description: Root of the MQTT topic hierarchy
    type: string
    default: enterprise

Datadog Connections

We set up one Cybus::Connection resource per intake host. The headers property adds the DD-API-KEY header to every request, so the endpoints that use these connections do not need any authentication configuration of their own.

resources:
  datadogMetricsConnection:
    type: Cybus::Connection
    properties:
      protocol: Http
      connection:
        scheme: https
        host: !sub 'api.${datadogSite}'
        port: 443
        headers:
          DD-API-KEY: !ref datadogApiKey

  datadogLogsConnection:
    type: Cybus::Connection
    properties:
      protocol: Http
      connection:
        scheme: https
        host: !sub 'http-intake.logs.${datadogSite}'
        port: 443
        headers:
          DD-API-KEY: !ref datadogApiKey

Submitting Machine Metrics

The Metrics API accepts a series array. Each series carries the metric name, the metric type as an integer (0 unspecified, 1 count, 2 rate, 3 gauge), the data points, and optional tags and resources. Each point is an object with a timestamp in epoch seconds and a numeric value. Datadog rejects timestamps that are more than ten minutes in the future or more than one hour in the past. A resource of type host sets the host that the metric is associated with in Datadog.

We define a write endpoint for the API path and a mapping that feeds it from the MQTT topic hierarchy. The HTTP/REST connector expects the request body in the body property of the message (see Publishing Data to REST Servers). The transform rule builds the series body: $floor($millis() / 1000) supplies the current time in epoch seconds, and the named wildcards of the topic make the equipment hierarchy levels available in $context.vars, so the machine identity ends up in the tags without the machine having to send it.

metricsEndpoint:
  type: Cybus::Endpoint
  properties:
    protocol: Http
    connection: !ref datadogMetricsConnection
    write:
      path: /api/v2/series

metricsMapping:
  type: Cybus::Mapping
  properties:
    mappings:
      - subscribe:
          topic: !sub '${topicRoot}/+site/+area/+line/+cell/temperature'
        publish:
          endpoint: !ref metricsEndpoint
        rules:
          - transform:
              expression: !sub |
                {
                  "body": {
                    "series": [
                      {
                        "metric": "${metricName}",
                        "type": 3,
                        "points": [
                          {
                            "timestamp": $floor($millis() / 1000),
                            "value": value
                          }
                        ],
                        "resources": [
                          { "name": $context.vars.cell, "type": "host" }
                        ],
                        "tags": [
                          "site:" & $context.vars.site,
                          "area:" & $context.vars.area,
                          "line:" & $context.vars.line,
                          "service:${serviceName}"
                        ]
                      }
                    ]
                  }
                }

A machine publishes its reading as a JSON object with a value property, for example to enterprise/hamburg/assembly/line-1/press-01/temperature:

{ "value": 42.1 }

This submits one gauge point for machine.temperature with the host press-01 and the tags site:hamburg, area:assembly, line:line-1, and service:shop-floor. If your machine payload already carries a timestamp in epoch milliseconds, for example because the message comes from another Connectware endpoint, use $floor(timestamp / 1000) in the transform instead of the current time.

Metrics submitted through this API count as custom metrics in Datadog. Every unique combination of metric name and tag values is a separate custom metric, which drives Datadog billing. Keep unbounded values, such as serial numbers, out of the tags.

Shipping Machine Event Logs

The Logs API accepts an array of up to 1000 log events per request. Each event carries the message and the reserved attributes ddsource, ddtags, hostname, service, and status. Unlike the tags array of the Metrics API, ddtags is a single comma-separated string. Any additional JSON properties become log attributes in Datadog.

The endpoint uses the logs connection and a mapping that turns every machine event into a one-element log array:

logsEndpoint:
  type: Cybus::Endpoint
  properties:
    protocol: Http
    connection: !ref datadogLogsConnection
    write:
      path: /api/v2/logs

logsMapping:
  type: Cybus::Mapping
  properties:
    mappings:
      - subscribe:
          topic: !sub '${topicRoot}/+site/+area/+line/+cell/events'
        publish:
          endpoint: !ref logsEndpoint
        rules:
          - transform:
              expression: !sub |
                {
                  "body": [
                    {
                      "message": message,
                      "status": status,
                      "ddsource": "connectware",
                      "service": "${serviceName}",
                      "hostname": $context.vars.cell,
                      "ddtags": "site:" & $context.vars.site
                        & ",area:" & $context.vars.area
                        & ",line:" & $context.vars.line
                    }
                  ]
                }

A machine publishes its events as a JSON object with a message and a status property, for example to enterprise/hamburg/assembly/line-1/press-01/events:

{ "message": "Material jam at infeed", "status": "error" }

Datadog timestamps each log on arrival. To keep the original event time instead, add a timestamp property in ISO 8601 format or epoch milliseconds to the log object. Datadog accepts log timestamps up to 18 hours in the past.

Verifying the Integration

1. Install the service and set the datadogSite and datadogApiKey parameters to the values of your Datadog account.

2. Check that both connections are in the Connected state on the service details page in the Admin UI.

3. Publish a test message with the payload { "value": 42.1 } to enterprise/hamburg/assembly/line-1/press-01/temperature, for example with an MQTT client or the Admin UI.

4. Open the Metrics Explorer in Datadog and query the machine.temperature metric. A newly submitted metric name can take a few minutes to appear the first time.

5. Publish a test message with the payload { "message": "Material jam at infeed", "status": "error" } to enterprise/hamburg/assembly/line-1/press-01/events and check the Log Explorer in Datadog, for example with the query source:connectware.

6. The result of every HTTP request is published to the /res topic of the endpoint. Use the Data Explorer to inspect it. A successful intake request returns HTTP status 202. If Datadog rejects a request, the message on the /res topic contains an error property with the HTTP status, for example 403 Forbidden for an invalid API key.

The Datadog intake hosts answer the connection probe without authentication. A connection in the Connected state therefore does not prove that the API key is valid, check the /res topic for authentication errors.

Service Commissioning File Example

datadog-example.yml

5KB

Open

datadog-example.yml

---
# ----------------------------------------------------------------------------
# Service Commissioning File
# ----------------------------------------------------------------------------
# Copyright: Cybus GmbH
# Contact: [e-mail]
# ----------------------------------------------------------------------------
# Datadog Integration (Example)
# ----------------------------------------------------------------------------

description: >
  Service commissioning file for sending shop floor metrics and logs from
  Connectware to Datadog (Example)

metadata:
  name: Datadog Integration
  provider: cybus
  homepage: https://www.cybus.io
  version: 1.0.0

parameters:
  datadogSite:
    description: Datadog site of your account, for example datadoghq.com or datadoghq.eu
    type: string
    default: datadoghq.com

  datadogApiKey:
    description: Datadog API key used for the metrics and logs intake
    type: string

  metricName:
    description: Name of the submitted gauge metric
    type: string
    default: machine.temperature

  serviceName:
    description: Value of the service tag on metrics and logs
    type: string
    default: shop-floor

  topicRoot:
    description: Root of the MQTT topic hierarchy
    type: string
    default: enterprise

resources:
  # Connection to the Datadog metrics intake
  datadogMetricsConnection:
    type: Cybus::Connection
    properties:
      protocol: Http
      connection:
        scheme: https
        host: !sub 'api.${datadogSite}'
        port: 443
        headers:
          DD-API-KEY: !ref datadogApiKey

  # Connection to the Datadog logs intake (separate host)
  datadogLogsConnection:
    type: Cybus::Connection
    properties:
      protocol: Http
      connection:
        scheme: https
        host: !sub 'http-intake.logs.${datadogSite}'
        port: 443
        headers:
          DD-API-KEY: !ref datadogApiKey

  # Submits one gauge metric point per machine reading
  metricsEndpoint:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref datadogMetricsConnection
      write:
        path: /api/v2/series

  metricsMapping:
    type: Cybus::Mapping
    properties:
      mappings:
        - subscribe:
            topic: !sub '${topicRoot}/+site/+area/+line/+cell/temperature'
          publish:
            endpoint: !ref metricsEndpoint
          rules:
            - transform:
                # Type 3 is a gauge, timestamps are in epoch seconds
                expression: !sub |
                  {
                    "body": {
                      "series": [
                        {
                          "metric": "${metricName}",
                          "type": 3,
                          "points": [
                            {
                              "timestamp": $floor($millis() / 1000),
                              "value": value
                            }
                          ],
                          "resources": [
                            { "name": $context.vars.cell, "type": "host" }
                          ],
                          "tags": [
                            "site:" & $context.vars.site,
                            "area:" & $context.vars.area,
                            "line:" & $context.vars.line,
                            "service:${serviceName}"
                          ]
                        }
                      ]
                    }
                  }

  # Ships one log event per machine event message
  logsEndpoint:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref datadogLogsConnection
      write:
        path: /api/v2/logs

  logsMapping:
    type: Cybus::Mapping
    properties:
      mappings:
        - subscribe:
            topic: !sub '${topicRoot}/+site/+area/+line/+cell/events'
          publish:
            endpoint: !ref logsEndpoint
          rules:
            - transform:
                # ddtags is a single comma-separated string
                expression: !sub |
                  {
                    "body": [
                      {
                        "message": message,
                        "status": status,
                        "ddsource": "connectware",
                        "service": "${serviceName}",
                        "hostname": $context.vars.cell,
                        "ddtags": "site:" & $context.vars.site
                          & ",area:" & $context.vars.area
                          & ",line:" & $context.vars.line
                      }
                    ]
                  }