HTTP/REST

Connect to REST APIs with support for polling, on-demand requests, and data publishing. Includes OAuth 2.0, Basic Auth, and header-based authentication.

The HTTP/REST Module supports the following interaction patterns with REST APIs:

• Manual polling (read) - Fetch data on-demand from endpoints.

• Continuous polling (subscribe) - Automatically poll endpoints at regular intervals.

• Publishing data (write) - Send data to endpoints.

Connection and Endpoint Configuration

HTTP servers are configured using the Cybus::Connection resource, which defines the hostname, scheme, port, authentication, and additional headers. Individual REST endpoints are configured with Cybus::Endpoint resources that specify the REST path, query parameters, and headers for each endpoint.

Authentication Methods

The following authentication methods are supported.

• Basic Authentication - Username/password with basicAuth scheme.

• Header-based Authentication - Arbitrary headers (e.g., Bearer tokens).

• OAuth 2.0 - Client Credentials Grant flow.

Service Commissioning File Configuration

You can configure endpoints with one of the following operations:

• subscribe - For continuous polling

• read - For event-driven polling

• write - For publishing data

Reference Documentation:

• Connection Properties

• Endpoint Properties

Subscribing to Data (Continuous Polling)

When using subscribe, the REST server is polled at regular intervals based on the interval property. Each endpoint requires a path and may include additional headers and query parameters.

Polling Behavior

• Scheduling: Polling occurs at the specified interval when possible. The system attempts to poll at the specified interval, but timing is not guaranteed to be exact. Actual intervals may be slightly longer due to network delays, server processing time, or system load.

• Serialized requests: If a server response takes longer than the polling interval, the next request waits for completion. This prevents the accumulation of an infinite number of queues in the event of an unfortunate combination of response times and polling intervals.

• Per-endpoint serialization: The serialization is implemented per endpoint (i.e., per REST path) so multiple requests to different paths can occur in parallel.

• Error handling: Failed requests log warnings but do not interrupt polling.

Response Message Format

All server responses are wrapped in a standardized JSON structure with timestamp and value:

{
  "timestamp": 12391238123, // milliseconds since epoch
  "value": ...              // server response
}

JSON Response Example

When the server returns Content-Type: application/json, the JSON is embedded as the value:

// Original server response
{
  "machineState": "okay",
  "temperature": 23,
  "logs": ["message1", "message2"]
}

// Resulting protocol module output at MQTT broker
{
  "timestamp": 12391238123, // milliseconds since epoch
  "value": {
    "machineState": "okay",
    "temperature": 23,
    "logs": ["message1", "message2"]
  }
}

Non-JSON Response Example

Non-JSON responses are embedded as strings:

// Original server response
<?xml><root><some interesting="other">protocol</some></root>

// Resulting protocol module output to Cybus broker
{
  "timestamp": 12391238123, // milliseconds since epoch
  "value": "<?xml><root><some interesting="other">protocol</some></root>"
}

Reading Data (Event-Driven Polling)

Data can be read on-demand by sending a message to the endpoint's /req topic. The server response is published to the /res topic of the endpoint (see Operation results).

The request message does not require specific content to trigger data reading.

Dynamic Path Configuration

Path Override

If the REST endpoint to be called requires a dynamic (changing) path, the configured path property can be overwritten by the message to the endpoint. This is done by providing a path property string in the message payload.

Example of a message with a dynamic path overwriting that will be sent on the /req topic:

// Message to /req topic
{ "path": "/bar" }


// Result: Configured path "/foo" becomes "/bar"

Path Append

# Configured path: /foo
# Actual path    : /bar

It is also possible to append a path suffix to the configured path property of the message sent to the endpoint. To do so, provide a pathAppend property string in the message payload.

Example of a message with a dynamic path appended using the pathAppend property that will be sent on the /req topic:

// Message to /req topic
{ "pathAppend": "/bar" }


// Result: Configured path "/foo" becomes "/foo/bar"

Dynamic Query Parameters

If the REST endpoint requires dynamic query parameters, they can be provided in the message payload as key-value pairs.

Example of a message with dynamic query parameters to be sent on the /req topic:

// Message to /req topic
{
  "query": {
    "foo": "bar",
    "user": "cybus"
  }
}

If the endpoint's configuration includes a query property, the two provided query parameters are merged. If any parameters are duplicated, the dynamic parameters from the message payload take precedence over the static configuration parameters.

Publishing Data to REST Servers

The HTTP/REST Protocol Mapper supports publishing (pushing) data from MQTT to a REST server in the other direction of data flow. This is done using the write property in the Cybus::Endpoint resource. Publishing does not occur at regular time intervals, but rather, it directly forwards any new message received from the broker on the /set topic of the endpoint.

Message Format Requirements

The MQTT message must be a JSON object containing a body property, which is forwarded as the body of the HTTP request:

{
  "body": {
    "foo": "bar"
  }
}

HTTP Request Details

The MIME type of the outgoing HTTP request will be set according to the value of the internal MQTT message. If the message contains parseable JSON, the MIME type of the HTTP request will be set to application/json. Otherwise, it will be set to application/octet-stream. No further changes will be made to the request. Any additional headers defined in the endpoint resource will be passed on as given.

Response Handling

The result of the HTTP request are published to the /res topic of the endpoint (see Operation results).

The data message on the result topic will have the following format.

Success Response

{
  "id": 29194,
  "timestamp": 1629351968526,
  "result": {
    "value": 0
  }
}

Error Response

This result message contains the following:

• id: The request identifier sent with the original request.

• timestamp: The Unix timestamp of when the result was received.

• result: The JSON object containing the actual result. Its content depends on the specific protocol implementation.

If an error occurred, the JSON object will not contain a property called result but instead a property called error. This property is a string variable that contains an error message.

Therefore, in the case of an error, the data message on the result topic will have the following format:

{
  "id": 29194,
  "timestamp": 1629351968526,
  "error": "POST request on /foo/bar failed with status: 403 Forbidden"
}

Dynamic Path for Publishing

Path Override for Write Operations

If the REST endpoint requires a dynamic path, the configured path property can be overwritten by the message. This is done by providing a string for the path property in the message payload.

Example of a message with a dynamic path overwriting to be sent on the /set topic:

// Message to /set topic
{
  "path": "/bar",
  "body": {
    "baz": "qux"
  }
}


// Result: Configured path "/foo" becomes "/bar"

Path Append for Write Operations

It is also possible to append a path suffix to the configured path property of the message sent to the endpoint. To do so, provide a pathAppend property string in the message payload.

Example of a message with a dynamic path appended using the pathAppend property to be sent on the /set topic:

// Message to /set topic
{
  "pathAppend": "/bar",
  "body": {
    "baz": "qux"
  }
}


// Result: Configured path "/foo" becomes "/foo/bar"

Encoded Request Body

To send binary data as the request body, the encoding of the message payload body parameter can be specified using a flag called bufferFromBody. The contents of the body parameter will then be converted into a buffer that respects the encoding. For a list of the supported encodings, see Node.js encodings.

{
  "bufferFromBody": "base64",
  "body": "YmFy"
}

Connection Monitoring and Probing

The HTTP protocol implementation continuously monitors connection health through successful operations and active probing.

Probing Mechanism

• Automatic probing: Runs periodically on the connection when there are no active requests on any endpoint using that connection. This ensures connection health is monitored even during idle periods.

• Probing interval: Configure the probing interval via the probeInterval property (default: 10 seconds).

• Default method: The probing is performed by default by sending an OPTIONS request to the endpoint server.

• Customization: Use probePath and probeMethod properties for configuring a custom HTTP path and HTTP method that is used by the probing function. See HTTP/REST Connection Properties.

Connection State Detection

• Connected: The connection is assumed to be connected if the probing succeeds or if it receives an HTTP response from the endpoint.

• Disconnected: If the probing fails due to a network error (e.g., ECONNRESET, ETIMEOUT, or EUNREACHABLE), the connection is assumed to be disconnected.

OAuth 2.0 Client Credentials Grant

The module supports OAuth 2.0 Client Credentials Grant for authenticating requests. Tokens are automatically refreshed when they expire.

Example configuration of the oauthClientCredentials property in your connection:

httpConnection:
  type: Cybus::Connection
  properties:
    protocol: Http
    targetState: connected
    connection:
      scheme: https
      host: one.host.io
      oauthClientCredentials:
        client_id: your_client_id
        client_secret: your_client_secret
        auth_url: oauth2.token.server
        audience: optional_audience # Optional
        grant_type: client_credentials # Default value

Required Properties:

• client_id - OAuth client identifier

• client_secret - OAuth client secret

• auth_url - OAuth token endpoint URL

Optional Properties:

• audience - Token audience (if required by OAuth provider)

• grant_type - Defaults to client_credentials

Fine-Tuning HTTPS Connections

Configuring Sockets and Connection Behavior

For HTTPS connections, you can fine-tune socket behavior, connection pooling, and resource utilization to optimize performance for your specific use case. These settings are particularly useful for high-throughput scenarios or when dealing with connection stability issues.

Configure these parameters under the agentOptions object in the connection section of a Cybus::Connection resource.

Key Configuration Parameters

• Connection Pooling: Control how many simultaneous connections are maintained.

• Socket Reuse: Manage idle socket behavior for improved performance.

• Timeouts: Set appropriate timeouts to handle network issues gracefully.

• Keep-Alive: Configure persistent connections to reduce overhead.

When to Use Custom Agent Options

• High-frequency polling: Increase maxSockets and maxFreeSockets for better throughput.

• Unstable networks: Adjust the timeout value to handle intermittent connectivity.

• Resource constraints: Lower socket limits to reduce memory usage.

• Long-running connections: Configure the keepAliveMsecs keep-alive parameter for stable, persistent connections.

For detailed information on these properties, see HTTP/REST Connection Properties.

Example: Configuring an HTTPS Agent

The following example demonstrates how to configure an HTTPS connection with agent-specific options for connection pooling, timeouts, and authentication.

This setup enables efficient and stable long-running connections to REST APIs while maintaining secure communication via HTTPS.

HttpConnection:
  type: Cybus::Connection
  properties:
    protocol: Http
    agentName: !ref agentName # Optional: Custom identifier for the HTTP agent instance
    connection:
      scheme: https # Use HTTPS for secure communication
      host: !ref httpHost # Hostname or IP address of the REST API server
      port: !ref httpPort # Port number for the HTTPS endpoint

      oauthClientCredentials: # OAuth 2.0 Client Credentials Grant configuration
        auth_url: !ref oauthUrl # Token endpoint of the OAuth server
        client_id: !ref oauthId # OAuth client ID
        client_secret: !ref oauthSecret # OAuth client secret
        grant_type: client_credentials # Grant type used for token retrieval (default)

      agentOptions: # Configuration for HTTPS agent behavior
        maxSockets: !ref maxSockets # Maximum concurrent sockets per host (default: 200)
        maxFreeSockets: !ref maxFreeSockets # Maximum idle sockets to keep open for reuse (default: 30)
        timeout: !ref timeout # Idle socket timeout in ms (default: 60000)
        keepAliveMsecs: !ref keepAliveMsecs # Keep-alive duration for idle sockets in ms (default: 30000)

      requestTimeout: !ref requestTimeout # Maximum duration for individual HTTP requests in ms (default: 10000)

Controlling How Long Requests Wait for a Response

All Axios-based HTTP operations use a default requestTimeout value of 10000 ms (10 s). This ensures that requests terminate if the target server does not respond within the specified time frame, improving reliability and preventing indefinitely hanging requests.

Service Commissioning File Example

3KB

http-example.yml

Open

http-example.yml

---
# ----------------------------------------------------------------------------
# Service Commissioning File
# ----------------------------------------------------------------------------
# Copyright: Cybus GmbH
# Contact: [email protected]
# ----------------------------------------------------------------------------
# Source Interface Definition - HTTP/REST Endpoint
# ----------------------------------------------------------------------------

description: >
  Service commissioning file for communicating with HTTP/REST endpoints (Example)

metadata:
  name: Http Connectivity
  icon: https://www.cybus.io/wp-content/uploads/2017/10/for-whom1.svg
  provider: cybus
  homepage: https://www.cybus.io
  version: 0.0.1

parameters:
  httpHost:
    description: IP address of the HTTP Server
    type: string
    default: 192.168.0.1

  httpPort:
    description: HTTP port of the server
    type: number
    default: 80

  initialReconnectDelay:
    type: integer
    default: 1000

  maxReconnectDelay:
    type: integer
    default: 30000

  factorReconnectDelay:
    type: integer
    default: 2

  maxSockets:
    type: integer
    default: 100

  maxFreeSockets:
    type: integer
    default: 50

  socketTimeout:
    type: integer
    default: 1000

  keepAliveMsecs:
    type: integer
    default: 3000

  requestTimeout:
    type: integer
    default: 10000

resources:
  httpConnection:
    type: Cybus::Connection
    properties:
      protocol: Http
      connection:
        scheme: https
        host: !ref httpHost
        port: !ref httpPort
        connectionStrategy:
          initialDelay: !ref initialReconnectDelay
          maxDelay: !ref maxReconnectDelay
          incrementFactor: !ref factorReconnectDelay
        agentOptions:
          maxSockets: !ref maxSockets
          maxFreeSockets: !ref maxFreeSockets
          timeout: !ref socketTimeout
          keepAliveMsecs: !ref keepAliveMsecs
        requestTimeout: !ref requestTimeout

  base:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref httpConnection
      subscribe:
        path: /

  counter:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref httpConnection
      subscribe:
        path: /counter

  getPost:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref httpConnection
      subscribe:
        path: /post

  writePost:
    type: Cybus::Endpoint
    properties:
      protocol: Http
      connection: !ref httpConnection
      write:
        path: /post

  httpMapping:
    type: Cybus::Mapping
    properties:
      mappings:
        - subscribe:
            endpoint: !ref base
          publish:
            topic: !sub '${Cybus::MqttRoot}/server/base'
        - subscribe:
            endpoint: !ref counter
          publish:
            topic: !sub '${Cybus::MqttRoot}/server/counter'
        - subscribe:
            endpoint: !ref getPost
          publish:
            topic: !sub '${Cybus::MqttRoot}/server/post'
        - subscribe:
            endpoint: !ref counter
          publish:
            endpoint: !ref writePost