> For the complete documentation index, see [llms.txt](https://docs.cybus.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cybus.io/data-flows/rule-engine.md).

# Rule Engine

Transform and process data streams in Connectware with JSONata-based Rule Engine rules and test them in the Rule Sandbox.

The Rule Engine is a core component of Connectware's data processing capabilities, offering a sophisticated and flexible approach to handling and transforming data streams. With the Rule Engine, you can apply transformation rules to data for complex data manipulations. You can define these rules within the resources section of your service commissioning files. For testing data processing rules before deploying them to your live system, you can use the Rule Sandbox.

## How Does the Rule Engine Work?

The Rule Engine uses [JSONata](https://docs.jsonata.org/) 2.2.2 for data transformation. Additionally, Connectware provides custom rules for specialized data processing and manipulation tasks, extending the core JSONata functionality.

For example, you could collect data from different machines on a specific topic, such as energy consumption and preprocess this data. The Rule Engine allows you to aggregate, filter, or transform this data before passing it on to other services or applications.

* Rules are applied directly within Connectware's data processing pipeline, enabling powerful data transformations.
* They are defined as a list (array) and executed sequentially, from top to bottom.
* Individual rules may have specific input format requirements and can conditionally pass or block data messages to the next rule, such as with a change-of-value (COV) filter.
* Each rule is configured as a named object with properties tailored to its specific type.

<figure><img src="https://302108577-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKtZHLQPbzDkQV8fA69YG%2Fuploads%2Fgit-blob-719316f596012b0970a271c17c638e5daecc0486%2Fdiagram-rule-engine.png?alt=media" alt="Data processing with the Rule Engine"><figcaption><p>Data processing with the Rule Engine</p></figcaption></figure>

## Types of Data Processing Rules

The following types of data processing rules are available.

For a detailed description of all rule engine parameters, see [Data Processing Rules](https://docs.cybus.io/data-flows/rule-engine/data-processing-rules.md)

1. **Aggregate Messages with the Burst Rule**

Consolidate multiple messages into a payload array using the `burst` rule. You can set this up based on time intervals or a maximum size, which simplifies your data handling and improves transmission efficiency.

2. **Aggregate Data with the Collect Rule**

Use the `collect` rule to combine data from multiple endpoints. This rule acts as a Last Value Cache, creating a single object containing the most recent data from each subscribed endpoint or topic. Each data point in the output is identified by a key, which can be either the endpoint itself or a custom label you've defined.

3. **Apply Change-of-Value (COV) Filtering**

Utilize the change-of-value (`cov`) filter to transmit data only when it has changed compared to the previous message. You can specify a key to monitor for changes and set optional deadband parameters. This helps optimize your data transfer and reduce network bandwidth usage.

4. **Filter Data**

Employ `filter` rules to evaluate incoming data against a JSONata expression. You can choose to forward or block data based on the evaluation result. This allows you to focus on relevant information and reduce data noise.

5. **Convert Data Formats with the Parse Rule**

Employ the `parse` rule to transform non-JSON input data into a JSON object. You can specify the format of your input data, and the rule will convert it accordingly. This allows you to standardize your data format, making it compatible with other rule operations in your processing pipeline. After parsing, you can easily pass the resulting JSON object to subsequent rules for further processing or analysis.

6. **Use Context Variables**

Store values as context variables for later use within the Rule Engine pipeline. You can use the `setContextVars` rule to pass data between rule steps or store intermediate results for further processing. This allows you to create complex data processing workflows.

7. **Store Intermediate States with the Stash Rule**

Save intermediate states of messages for later reference using the `stash` rule. You can then use these stashed values in filter or transform rules, giving you additional data management capabilities and enabling more advanced processing techniques.

8. **Transform Data**

Use JSONata expressions in the Rule Engine to `transform` your data in real-time. This query and transformation language allows you to convert, restructure, and perform calculations on your data, making it easier to integrate and process information from various sources.

9. **Validate Payloads with the Schema Rule**

Use the `schema` rule to validate the current message payload against an inline JSON Schema (draft-07). You can configure whether invalid payloads stop the rule chain or continue through it, and inspect the validation result in `$context.schema` from downstream rules.

## Use Cases for the Rule Engine

Whether you're working with Industrial IoT sensor data, business analytics, or any other data-intensive application, the Rule Engine provides flexible solutions for common data processing tasks.

Here are some example use cases for the Rule Engine:

* Data cleansing: Remove invalid or inconsistent data.
* Data enrichment: Add missing information or context.
* Data normalization: Standardize data formats.
* Data filtering: Select specific data based on criteria.
* Data transformation: Convert data from one format to another.

The order of the rules decides what each rule sees. The following examples show complete rule chains. Add the resources of an example to the `resources` section of your service commissioning file.

### Example: Summarizing Sensor Values

A temperature sensor publishes a value every second on `factory/line1/temperature`, for example `{ "timestamp": 1763546400000, "value": 20.1 }`. A defective sensor reports `-999`. The following mapping drops these readings and publishes a summary every 10 seconds:

{% code lineNumbers="true" %}

```yaml
temperatureSummary:
  type: Cybus::Mapping
  properties:
    mappings:
      - subscribe:
          topic: factory/line1/temperature
        publish:
          topic: factory/line1/temperature/summary
        rules:
          - filter:
              expression: value != -999
          - burst:
              interval: 10000
          - transform:
              expression: |
                {
                  "timestamp": $[-1].timestamp,
                  "count": $count($),
                  "min": $min($.value),
                  "max": $max($.value),
                  "avg": $round($average($.value), 2)
                }
```

{% endcode %}

1. `filter` drops the `-999` readings. Because it runs before `burst`, they never reach the array and cannot distort `min` and `avg`.
2. `burst` collects the messages of 10 seconds and passes them on as one array, for example `[{ "timestamp": 1763546400000, "value": 20.1 }, { "timestamp": 1763546401000, "value": 21.4 }, ...]`.
3. `transform` turns the array into one summary. `$[-1]` is the last message of the array, and `$.value` is the list of all values.

If one of the 10 readings in a window was `-999`, the published summary looks like this:

```json
{ "timestamp": 1763546409000, "count": 9, "min": 20.1, "max": 21.4, "avg": 20.7 }
```

### Example: Reading Values from a Raw Modbus Block

A CNC machine provides its data in 12 Modbus holding registers (24 bytes). The spindle speed is a big-endian 32-bit float in bytes 10 to 13, counting from 0. One `raw` endpoint reads the whole block with a single request per interval, instead of one request per value:

{% code lineNumbers="true" %}

```yaml
modbusConnection:
  type: Cybus::Connection
  properties:
    protocol: Modbus
    connection:
      host: 192.168.1.100
      port: 502

registerBlock:
  type: Cybus::Endpoint
  properties:
    protocol: Modbus
    connection: !ref modbusConnection
    subscribe:
      fc: 3
      address: 0
      length: 12
      interval: 1000
      dataType: raw

spindleSpeed:
  type: Cybus::Mapping
  properties:
    mappings:
      - subscribe:
          endpoint: !ref registerBlock
        publish:
          topic: machine/spindle/speed
        rules:
          - stash:
              label: block
          - transform:
              expression: $.value.data[[10..13]]
          - parse:
              format: floatBE
          - transform:
              expression: '{ "timestamp": $last("block").timestamp, "value": $ }'
```

{% endcode %}

1. `stash` keeps the original message under the label `block`. Because `stash` runs earlier in the same chain, `$last("block")` in the last rule returns this message, not the previous one.
2. `transform` takes bytes 10 to 13 out of the block. The endpoint publishes the block as `{ "timestamp": ..., "value": { "type": "Buffer", "data": [ ... ] } }`, so the bytes are in `value.data`. In JSONata, `[[10..13]]` selects the array elements 10 to 13.
3. `parse` reads the array of 4 bytes as a big-endian float, for example `1234.5`.
4. `transform` puts the value and the timestamp of the original message together.

The published message looks like this:

```json
{ "timestamp": 1763546400000, "value": 1234.5 }
```

To publish further values from the same block, add one entry per value to `mappings`, with the matching byte range and [parse format](https://docs.cybus.io/data-flows/rule-engine/data-processing-rules.md#format). Each entry has its own rules, so every entry can use the stash label `block`.

{% hint style="warning" %}
Consider the potential impact of rule changes on your data processing pipeline. Test your rules thoroughly to ensure they produce the desired results.
{% endhint %}

## Testing Data Processing Rules

The Rule Sandbox offers a safe and interactive environment for testing and refining your data processing rules. It allows you to experiment with real-time data transformations, visualize the effects of your rules instantly, and debug any issues before deploying to your live system.

For more information, see [Rule Sandbox](https://docs.cybus.io/data-flows/rule-engine/rule-sandbox.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.cybus.io/data-flows/rule-engine.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
