.. _user/protocols/http:

*********
HTTP/REST
*********

Overview
========

The HTTP/REST Module is suitable for either of the following methods:

  - manual polling of endpoints (`read`)
  - continuous polling of endpoints (`subscribe`)
  - publishing data to endpoints (`write`)

The HTTP server to be contacted is described by the Cybus::Connection resource.
This resource defines hostname, scheme, port, but also additional headers or
authentication info. Each REST endpoint is described by a Cybus::Endpoint
resource (no pun intended) that defines access to a particular REST path,
optionally specifying additional query parameters or headers for this REST path.

The currently supported authentication methods are

* Username/password for a *basicAuth* scheme
* Arbitrary header for e.g. literal *Bearer Tokens*
* *Oauth 2.0* Client Credentials Grant

Commissioning File Specifics
============================

The Endpoints for the HTTP connections may either use ``subscribe`` for
subscribing, ``read`` for event-driven polling, or ``write`` for publishing, as
described above.

.. _user/protocols/http_connection:

.. include:: ../protocolSchemas/HttpConnection.rst

.. _user/protocols/http_endpoint:

.. include:: ../protocolSchemas/HttpEndpoint.rst

Subscribing Data from a REST Server
===================================

When subscribing, the REST Server is polled at regular intervals according to
the ``interval`` property of the endpoint resource. Each endpoint must contain a
path and may contain additional header and query parameters.

If the request fails, a warning is logged but no action is taken.

Polling is implemented with a best-effort strategy, which means polling will
take place *at best* with the desired interval. If the REST server response
takes longer than the polling interval, the next request will not be triggered
before the previous has been completed. This prevents building up of infinite queues
in the event of unfortunate response time / polling interval combinations.
However, this serialization is implemented per endpoint (per REST path), so that
multiple requests to different paths can actually be in-flight in parallel.

Response Message
----------------

Similar to most protocol implementations in the Cybus Connectware, the server
response will be wrapped in a JSON structure that follow the convention
timestamp/value, like so:

.. code-block:: javascript

  {
     "timestamp": 12391238123, // ms since epoch
     "value": ...              // server response
  }

If the server responds with a `Content-type=application/json`, the response JSON
will be embedded to this object as value of the ``"value"`` property, like so:

.. code-block:: javascript

  // Original server response
  {
    "machineState": "okay",
    "temperature": 23,
    "logs": [
      "message1",
      "message2"
    ]
  }

  // Resulting protocol module output at MQTT broker
  {
     "timestamp": 12391238123, // ms since epoch
     "value": {
        "machineState": "okay",
        "temperature": 23,
        "logs": [
          "message1",
          "message2"
        ]
      }
  }

If the server response is not parseable as JSON, it will be embedded into the
message object as a string, hence

.. code-block::

  // Original server response
  <?xml><root><some interesting="other">protocol</some></root>

  // Resulting protocol module output to Cybus broker
  {
     "timestamp": 12391238123, // ms since epoch
     "value": "<?xml><root><some interesting="other">protocol</some></root>"
  }

.. _user/protocols/http/reading:

Reading Data from a REST Server
===================================

Data can also be *read* from the server on certain events. Reading data from the
server is triggered by sending a message to the ``/req`` topic of the endpoint
(see also :ref:`user/services/structure/resources/endpoint/results`). The
response from the server is written to the ``/res`` topic of the endpoint.

The message sent on the endpoint's ``/req`` topic does not need to have any
content to trigger reading of data.

.. _user/protocols/http/reading/path:

Dynamic path
------------

If the REST endpoint to be called requires a dynamic (changing) *path*, the
configured *path* property can be overwritten by the message to the endpoint.
This is done by providing a ``path`` property string in the message payload.

Example message with dynamic path overwriting (to be sent on the ``/req`` topic):

.. code-block:: json

  {
    "path": "/bar"
  }

.. code-block:: yaml

    # Configured path: /foo
    # Actual path    : /bar

It is also possible to append some path suffix to the configured *path* property
by the message to the endpoint. Appending requires providing a ``pathAppend``
property string in the message payload.

Example message with dynamic path appended using ``pathAppend`` (to be sent on
the ``/req`` topic):

.. code-block:: json

  {
    "pathAppend": "/bar"
  }

.. code-block:: yaml

    # Configured path: /foo
    # Actual path    : /foo/bar

Dynamic query parameters
------------------------

If the REST endpoint to be called requires dynamic *query* parameters, these can
be provided in the message payload as key-value pairs.

Example message with dynamic query parameters (to be sent on the ``/req`` topic):

.. code-block:: json

  {
    "query": {
      "foo": "bar",
      "user": "cybus"
    }
  }

If the endpoint's configuration already has a ``query`` property, both provided
query parameters are merged. If any parameters are duplicated, the dynamic
parameters from the message payload override the ones from the static
configuration.


.. _user/protocols/http/publishing:

Publishing Data to a REST Server
================================

The HTTP/REST Protocol Mapper also supports the other direction of data flow,
namely publishing (pushing) data from MQTT to a REST server using the ``write``
property in the Cybus::Endpoint resource. Publishing does not work on a regular
time interval but will directly forward any new message that is received from
the broker on the ``/set`` topic of the endpoint.

The data of the internal MQTT message must be of type object and has to contain
a property called ``body``, which will be forwarded as the body of the HTTP request.

.. code-block:: json

  {
    "body": {
      "foo": "bar"
    }
  }

The request mime type header of the outgoing HTTP request will be set according
to the internal MQTT message value: If the message contains parseable JSON, the
mime type of the HTTP request will be set to `application/json`, otherwise it
will be set to `application/octet-stream`. Besides that, no further changes are
performed on the request. Any additional headers as defined in the endpoint
resource will be passed on as given.

The result of the HTTP request is returned on the ``/res`` topic of the
endpoint, as explained here:
:ref:`user/services/structure/resources/endpoint/results`. The data message on
the result topic will have the following format:

.. code-block:: json

  {
    "id": 29194,
    "timestamp": 1629351968526,
    "result": {
      "value":0
    }
  }

In this result message, ``id`` is the request identifier that was sent on the
original request, ``timestamp`` is the Unix timestamp of when the result was
received, and ``result`` is the JSON object with the actual result, but its
content depends on the concrete protocol implementation.

If there was an error, the resulting JSON object does not contain a property
`result` but instead a property ``error``. This property is simply a string
variable and contains an explanatory message of the error. Hence, in the error
case the data message on the result topic will have the following format:

.. code-block:: json

  {
    "id": 29194,
    "timestamp":1629351968526,
    "error": "POST request on /foo/bar failed with status: 403 Forbidden"
  }

Dynamic path
------------

If the REST endpoint to be called requires a dynamic (changing) *path*, the
configured *path* property can be overwritten by the message to the endpoint.
This is done by providing a ``path`` property string in the message payload.

Example message with dynamic path overwriting (to be sent on the ``/set`` topic):

.. code-block:: json

  {
    "path": "/bar",
    "body": {
      "baz": "qux"
    }
  }

.. code-block:: yaml

    # Configured path: /foo
    # Actual path    : /bar

It is also possible to append some path suffix to the configured *path* property
by the message to the endpoint. Appending requires providing a ``pathAppend``
property string in the message payload.

Example message with dynamic path appended using ``pathAppend`` (to be sent on
the ``/set`` topic):

.. code-block:: json

  {
    "pathAppend": "/bar",
    "body": {
      "baz": "qux"
    }
  }

.. code-block:: yaml

    # Configured path: /foo
    # Actual path    : /foo/bar

Encoded request body
---------------------

If it is required to send binary data as the request body, it is possible
to specify the encoding of the message payload body parameter using a flag
called ``bufferFromBody``. The contend of the body parameter will then be 
converted into a buffer, respecting the encoding. Allowed encodings can be
found here https://nodejs.org/api/buffer.html#buffers-and-character-encodings.

.. code-block:: json

  {
    "bufferFromBody": "base64",
    "body": "YmFy"
  }

.. _user/protocols/http/probing:

Connection Probing
==================

While HTTP by itself doesn't require a continuous connection, the HTTP protocol
implementation in the Connectware is monitoring the connection state for
up-to-date connectivity information. The connection state is estimated by
monitoring each successful read and write, and additionally by a probing
function scheduled to run if no read or write operation is in progress. The
additional probing function is run after the time interval from the property
``probeInterval`` (default: 10 seconds).

By default the probing is performed by sending an `OPTIONS` request against the
endpoint server. If this request fails with network error (e.g. ECONNRESET,
ETIMEOUT, EUNREACHABLE), the connection state is assumed to be disconnected. If
it succeeds, or if it at least gets a HTTP response from the endpoint, the
connection is assumed to be connected.

Additionally the properties ``probePath`` and ``probeMethod`` allow configuring
a custom HTTP path and HTTP method to be used by the probing function. See the
section `Connection Properties` for more details on these properties.

.. _user/protocols/http/oauth:

Oauth 2.0 Client Credentials Grant
==================================

This module supports the `Oauth 2.0 Client Credentials Grant` flow for authenticating
requests (see `rfc6749 Section-4.4  <https://datatracker.ietf.org/doc/html/rfc6749#section-4.4>`_).

Tokens are automatically refreshed when they expire.

To use this authentication method the property `oauthClientCredentials` needs
to be configured in the connection object providing `client_id`, `client_secret`,
`auth_url` and, optionally, as some Oauth 2.0 implementations do not require it, `audience`
and `grant_type` (which defaults to `client_credentials`).

Example configuration:

.. code-block:: yaml

    httpConnection:
      type: Cybus::Connection
      properties:
        protocol: Http
        targetState: connected
        connection:
          scheme: https
          host: one.host.io
          oauthClientCredentials:
            client_id: your_client_id
            client_secret: your_client_secret
            auth_url: oauth2.token.server

Sample Commissioning File
=========================

Download: :download:`http-example.yml`

.. literalinclude:: http-example.yml
  :language: yaml
  :linenos: