> For the complete documentation index, see [llms.txt](https://docs.cybus.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file.md).

# File

Read CSV files from local directories, mounted network shares, SFTP servers, and FTP servers, and publish every row as an MQTT message.

The File connector reads CSV files and publishes every row as an MQTT message. Use it for machines and legacy systems whose only interface is a file export, such as a process log, one CSV file per production job, or one log file per shift.

* [File Connection Properties](https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file/fileconnection.md)
* [File Endpoint Properties](https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file/fileendpoint.md)

## How the File Connector Works

The File connector uses two resources:

* A **connection** with `protocol: File` defines where the files are.
* An **endpoint** defines which file or directory to read, and a mode that matches how your machine writes its files.

The connector checks the files for changes every `pollIntervalMs` (default 1000 ms), publishes new rows on the topic of the endpoint, and saves how far it has read. After a restart or a service update, it continues where it stopped. File endpoints support only the `subscribe` operation and only CSV files.

{% hint style="info" %}
On Kubernetes, the File connector requires the `connectware` Helm chart 3.4.0 or later.
{% endhint %}

## Connecting to the Files

The `transport.type` connection property selects where the connector reads files from:

| Transport | Reads from                                                                                                             | Supported modes                 |
| --------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| `local`   | A directory inside the Protocol Mapper or agent container, including SMB/CIFS and NFS shares that are mounted into it. | `follow`, `batch`, `followMany` |
| `sftp`    | An SSH File Transfer Protocol (SFTP) server.                                                                           | `follow`, `batch`, `followMany` |
| `ftp`     | An FTP or FTPS server, in passive mode.                                                                                | `batch`                         |

### Local Directories and Mounted Shares

The `local` transport reads from the file system of the container that runs the connection: the Protocol Mapper, or the [agent](https://docs.cybus.io/2-7-0/data-flows/agents.md) that the connection is assigned to. To read an SMB/CIFS or NFS share, [mount it into the container](#making-files-available-to-the-container).

Set `transport.basePath` to the directory that contains all files of the connection, usually the mount point of the share. Relative endpoint paths resolve against `basePath`, and paths outside it are rejected, including paths with `..` and symbolic links that point outside `basePath`. In `batch` and `followMany` mode, the connector skips a rejected file and reads the other files. In `follow` mode, a rejected path makes the whole connection reconnect, so the other endpoints of the connection stop too. Put `follow` endpoints whose file can be a symbolic link on a separate connection.

`basePath` must be inside one of the directories in the `CYBUS_FILE_LOCAL_ROOTS` environment variable of the Protocol Mapper or agent, a colon-separated list of absolute paths. The default is `/mnt`, so mount shares below `/mnt`, for example at `/mnt/line3`.

{% hint style="warning" %}
Setting `CYBUS_FILE_LOCAL_ROOTS` to `/` lets every service commissioning file with a File connection read, move, and delete any file that the container can access. Only do this for a dedicated agent that has nothing else mounted.
{% endhint %}

The `identity` endpoint property defines how the connector recognizes a file, so that it can follow the file across renames. The default `auto` uses the inode number on `local`, the first 4096 bytes of the file (content identity) on `sftp`, and the path, size, and modification time on `ftp`. On SMB/CIFS mounts, inode numbers can change while a file stays the same, and the connector would read the file again. Set `identity: content` on endpoints that read from such a mount, so that the connector recognizes files by their first bytes instead. Changing `identity` on an endpoint that has already read files makes the connector read them again, so set it when you create the endpoint.

The connector reads rows that a program appends to a file through the share. If the share is served by Samba and the program writes to the file locally on the Samba server instead, the connector does not see the new rows, because the SMB/CIFS mount keeps using its cached copy of the file while it holds a lease on it.

#### Making Files Available to the Container

Mount a network share on the host first. On Linux, use the following options:

{% code lineNumbers="true" %}

```bash
sudo mount -t cifs //${FILE_SERVER}/${SHARE} /mnt/line3 \
  -o credentials=/root/.smbcred-line3,uid=1000,gid=1000,vers=3.0,iocharset=utf8,soft,echo_interval=15
```

{% endcode %}

* `${FILE_SERVER}`: The hostname of the file server.
* `${SHARE}`: The name of the share.

With a `hard` mount, file operations on an unreachable server block the whole Protocol Mapper or agent instead of failing, so use `soft` and `echo_interval`. Set `uid` and `gid` to the user that the container runs as (check with `id` inside the container), otherwise moving and deleting files fails.

Then make the mount available inside the container, below one of the directories in `CYBUS_FILE_LOCAL_ROOTS`:

{% tabs %}
{% tab title="Docker" %}
For the Protocol Mapper of Connectware, add the mount to the `docker-compose.override.yml` file in your Connectware installation directory:

{% code lineNumbers="true" %}

```yaml
services:
  protocol-mapper:
    volumes:
      - /mnt/line3:/mnt/line3
```

{% endcode %}

To apply the change, run `docker compose up -d` in the installation directory.

For an agent, add the same `volumes` entry to the agent service in its `docker-compose.yml` file. See [Installing Agents via Docker Compose](https://docs.cybus.io/2-7-0/data-flows/agents/installing-agents/installing-agents-via-docker-compose.md).

To allow other directories than `/mnt`, add the `CYBUS_FILE_LOCAL_ROOTS` environment variable to the same service, for example `CYBUS_FILE_LOCAL_ROOTS: /mnt:/srv/exports`.
{% endtab %}

{% tab title="Kubernetes" %}
Make the share available as a volume, for example as a PersistentVolumeClaim from an SMB CSI driver. For the Protocol Mapper of Connectware, add the volume to the `values.yaml` file of the Connectware Helm chart:

{% code lineNumbers="true" %}

```yaml
protocolMapper:
  extraVolumes:
    - name: line3-share
      persistentVolumeClaim:
        claimName: line3-share
  extraVolumeMounts:
    - name: line3-share
      mountPath: /mnt/line3
```

{% endcode %}

For an agent, set `extraVolumes` and `extraVolumeMounts` on the agent in the `protocolMapperAgents` section of the Connectware Agent Helm chart. See [Installing Connectware Agents Using the Connectware Agent Helm Chart](https://docs.cybus.io/2-7-0/data-flows/agents/agents-in-kubernetes/installing-connectware-agents-using-the-connectware-agent-helm-chart.md).

To allow other directories than `/mnt`, set the `CYBUS_FILE_LOCAL_ROOTS` environment variable with `protocolMapper.env`, or with `env` on the agent.
{% endtab %}
{% endtabs %}

### SFTP Servers

The `sftp` transport needs `host`, `username`, and `password`, `privateKey`, or both. `privateKey` is an RSA or Ed25519 key in OpenSSH or PEM format. Set `passphrase` if the key is encrypted. The default `port` is 22. Relative endpoint paths resolve against the login directory of the user.

The `hostKeyVerification` property defines how the connector checks the identity of the server:

* `strict` (default): The server must present a key that is listed in `knownHosts`. `knownHosts` is required.
* `trustOnFirstUse`: The connector saves the key of the first connection. Every later connection must present the same key. If the server key changes legitimately, increase `hostKeyGeneration` (default 0) by one so that the connector accepts the next key. The Protocol Mapper and each agent save their own key, so after you move the service to another agent, that agent saves the key of its first connection. To verify the key on that connection too, list it in `knownHosts`. Deleting the service does not delete a saved key.
* `insecure`: The connector accepts any key. Use this mode only for testing.

The Admin UI shows the values of `hostKeyVerification` and `hostKeyGeneration` as `*****`. To check them, look in the service commissioning file.

To get the fingerprint for `knownHosts`, run the following command on a machine that can reach the server:

{% code lineNumbers="true" %}

```bash
ssh-keyscan ${SFTP_HOST} | ssh-keygen -lf -
```

{% endcode %}

* `${SFTP_HOST}`: The hostname or IP address of the SFTP server.

The output has one line per host key of the server. Copy the `SHA256:...` part of each line into `knownHosts`. Instead of a fingerprint, an entry can also be a public key line `<key-type> <base64>`, as in a `~/.ssh/known_hosts` file. The following connection reads the password from a [secret parameter](https://docs.cybus.io/2-7-0/data-flows/service-commissioning-files/parameters.md#secret):

{% code lineNumbers="true" %}

```yaml
parameters:
  sftpPassword:
    type: string
    secret: true

resources:
  sftpConnection:
    type: Cybus::Connection
    properties:
      protocol: File
      connection:
        transport:
          type: sftp
          host: example.com
          username: connectware
          password: !ref sftpPassword
          knownHosts:
            - 'SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s'
```

{% endcode %}

### FTP and FTPS Servers

The `ftp` transport supports the `batch` mode only. It connects in passive mode, so the passive port range of the server must be reachable from the Protocol Mapper or agent. The transport needs `host`, and the default `port` is 21. Without `username`, the connector logs in anonymously. Relative endpoint paths resolve against the login directory.

The `tls` property selects the encryption:

* `none` (default): Plain FTP. Credentials and file contents travel unencrypted.
* `explicit`: FTPS that switches to TLS after connecting, usually on port 21.
* `implicit`: FTPS with TLS from the first byte, usually on port 990.

To trust a self-signed or internal server certificate, add it in PEM format with `tlsCa`. The connector then trusts only the certificates in `tlsCa`. The following connection uses explicit FTPS:

{% code lineNumbers="true" %}

```yaml
parameters:
  ftpPassword:
    type: string
    secret: true

resources:
  ftpConnection:
    type: Cybus::Connection
    properties:
      protocol: File
      connection:
        transport:
          type: ftp
          host: example.com
          username: connectware
          password: !ref ftpPassword
          tls: explicit
```

{% endcode %}

## Choosing a Mode

The `mode` endpoint property defines how rows arrive, and `path` points to a file or a directory depending on the mode:

| Mode         | `path` points to | Publishes                                                      | Typical use                        |
| ------------ | ---------------- | -------------------------------------------------------------- | ---------------------------------- |
| `follow`     | A file           | Every row that is appended to the file.                        | A process log written all day.     |
| `batch`      | A directory      | Every row of each complete file, once per file.                | One CSV file per production job.   |
| `followMany` | A directory      | Every row that is appended to any matching file, file by file. | One log file per shift or per day. |

In `batch` and `followMany` mode, `pattern` (default `*.csv`) selects files by name, and `ignorePattern` skips files, for example `*.tmp` for files that a writer renames when it finishes. Both use case-sensitive glob syntax. The connector reads only files directly in the directory, not in subdirectories.

### Follow

On every poll, the connector publishes the rows that were appended since the last poll. If the file does not exist yet, the connector waits for it.

When the file is replaced or truncated, for example by log rotation, the connector reads the new file from the beginning. Rows that the writer adds to the old file after the last poll are not read. If your writer rotates files by renaming them in the same directory, use `followMany` with a `pattern` that matches both names instead, for example `process.log*` for `process.log` and `process.log.1`. The connector then keeps reading a renamed file where it stopped.

With inode identity, the default on `local`, the connector does not notice when a program rewrites the file in place and the new content has exactly the same size. The rows of the new content are then not read, and nothing is logged. With content identity, it notices the rewrite when the first 4096 bytes change.

### Batch

The connector reads each file once it is complete, which means that its size and modification time have not changed for `settleMs` (default 3000 ms). If the writer pauses for longer than that while it writes a file, increase `settleMs`. If the writer creates a marker file when it finishes, set `markerSuffix`, for example `.done` to read `job.csv` as soon as `job.csv.done` exists.

The `afterProcessing` property defines what happens to a file after all of its rows are published:

* `action: none` (default): The file stays where it is, and the connector does not read it again.
* `action: move`: The file moves to the `moveTo` directory, which the connector creates if it does not exist. A relative `moveTo` resolves like an endpoint `path`.
* `action: delete`: The connector deletes the file.

Move and delete need write access to the directory. If they fail, the file stays where it is and is not read again. The connector moves or deletes a file only after its rows were handed to the MQTT broker connection. If that takes longer, for example under heavy load, the file stays in place without being read again, and the connector moves or deletes it on a later poll, also after a restart.

If a file that was already read changes its size or modification time, the connector reads the whole file again and publishes all of its rows a second time. To read a file that keeps growing, use `follow` or `followMany`.

With `action: none`, the connector remembers every file in the directory, which works for about 19,000 files with short names on the `local` transport and a few thousand on SFTP and FTP. Longer file names lower this limit. At the limit, the connector stops reading new files and logs `Checkpoint full of entries of files still in the directory`. For directories that receive many files, use `move` or `delete`.

### Follow Many

The connector follows every file in the directory that matches `pattern`, each with its own read position, and reads new files from their beginning. A file that does not grow for `inactiveMs` (default 3600000 ms, one hour) counts as finished. The connector stops checking it and applies `afterProcessing`, as described for [Batch](#batch). Set `inactiveMs` longer than the longest pause between two writes to the same file. If a finished file grows again, the connector continues where it stopped.

The connector follows at most `maxTrackedFiles` (default 200) files at the same time. If more files are active, it follows the newest ones, and the others wait until a followed file finishes. A waiting file is then read from its beginning.

With `action: none`, the connector remembers every finished file that is still in the directory. With short file names, this works for about 10,000 files with inode identity and about 7,000 files with content identity. Longer file names lower this limit. Above it, the connector reads finished files again and publishes their rows again, once every `inactiveMs`. For directories that collect more files, use `move` or `delete`.

## Where Reading Starts

When an endpoint starts for the first time, it has no saved read position, and the following properties define where it starts. Once a position is saved, the connector continues from it, and changing `startAt` or `processExisting` has no effect. Changing `path` or `mode` starts the endpoint over. By default, new `follow` and `followMany` endpoints start at the end of each file, and a new `batch` endpoint reads only the files of the last day.

For `follow` and `followMany`, the `startAt.type` property defines the start:

| `startAt.type`  | Publishes                                                                                                                          |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `end` (default) | Only rows that are appended after the start.                                                                                       |
| `beginning`     | The whole file.                                                                                                                    |
| `lastRows`      | The last `startAt.rows` rows, then every appended row.                                                                             |
| `since`         | Rows whose `timestampColumn` value is at or after `startAt.since` (ISO 8601), then every appended row. Requires `timestampColumn`. |

In `followMany` mode, `startAt` applies to the files that exist at the first start. Files that appear later are read from their beginning.

For `batch`, the connector reads the files that are already in the directory at the first start. Set `processExisting` (default `true`) to `false` to read only files that arrive later.

In `batch` and `followMany` mode, the connector ignores files whose modification time is older than `maxFileAgeMs` (default 86400000 ms, one day). Set it to `0` to read files of any age.

While it catches up on existing rows, the connector publishes at most `backfillRateLimitRowsPerSecond` rows per second (default 500, `0` for no limit).

### Keeping and Resetting the Read Position

The connector saves the read position of each endpoint in Connectware, under the service ID, the connection name, and the endpoint name. The position stays when you disable or update the service, or move it to another agent. After a move, the other agent continues where the first one stopped, as long as it reads the same files, for example from the same share. An endpoint that you remove in a service update also keeps its position, and continues from it if you add it again with the same name.

[Deleting the service](https://docs.cybus.io/2-7-0/data-flows/services/managing/deleting.md) keeps the read positions of its endpoints too, so a service that you install again with the same service ID continues from them. In `followMany` mode, a file that is renamed while the service is deleted is published a second time after you install the service again.

To discard the saved position of an endpoint, including the list of files that a `batch` endpoint has read, increase its `checkpointGeneration` property (default 0) by one and update the service. The endpoint then starts as described in [Where Reading Starts](#where-reading-starts). Rows that it reads again get new `dedupKey` values, so consumers that drop duplicates do not drop these rows. To read existing data again, also set the following properties in the same update:

* For `follow` and `followMany`, set `startAt.type: beginning`.
* For `batch` and `followMany`, set `maxFileAgeMs: 0` or a value that covers the files.
* For `followMany`, set `inactiveMs` longer than the age of the files, because files that were not modified within `inactiveMs` are not read.

## Parsing CSV

The `parse` endpoint property defines how the connector reads the content of a file. The most common settings are the following:

* `delimiter`: Field separator (default `,`), for example `;` or `\t`.
* `encoding`: Character encoding of the file (default `utf8`), for example `windows-1252` or `iso-8859-1`.
* `decimalSeparator`: Set to `,` for numbers like `1.000,5`. The dot is then a thousands separator.
* `skipRows`: Number of lines to ignore at the top of the file, such as title or unit lines.

For the full list of parser properties and their defaults, see [File Endpoint Properties](https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file/fileendpoint.md).

### Column Names

`parse.header` defines whether the first line (after `skipRows`) is a header line, and `parse.columns` lets you declare the column names yourself:

| `header`         | `columns` | Result                                                                                               |
| ---------------- | --------- | ---------------------------------------------------------------------------------------------------- |
| `true` (default) | Not set   | The names from the header line become the keys of each row.                                          |
| `true`           | Set       | The header line is skipped, and your names become the keys.                                          |
| `false`          | Set       | Every line is data, and your names become the keys.                                                  |
| `false`          | Not set   | Every line is data, and each row is an array. Refer to columns by their position, starting at `"0"`. |

### Column Types

Without a type, every value is published as a string. Use `parse.types` to convert columns:

| Type                | Accepts                                                                                                                                  | Publishes                 |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `string`            | Any value.                                                                                                                               | The value as a string.    |
| `number`            | Decimal numbers, including scientific notation.                                                                                          | A number.                 |
| `integer`           | Whole numbers.                                                                                                                           | A number.                 |
| `boolean`           | `true`/`false`, `1`/`0`, `yes`/`no`, and `on`/`off`, in any capitalization.                                                              | `true` or `false`.        |
| `datetime`          | ISO 8601 timestamps.                                                                                                                     | Milliseconds since epoch. |
| `datetime:<format>` | Timestamps in a [Luxon format](https://moment.github.io/luxon/#/parsing?id=table-of-tokens), for example `datetime:dd.MM.yyyy HH:mm:ss`. | Milliseconds since epoch. |
| `datetime:X`        | Seconds since epoch.                                                                                                                     | Milliseconds since epoch. |
| `datetime:x`        | Milliseconds since epoch.                                                                                                                | Milliseconds since epoch. |

An empty cell in a typed column becomes `null`. A value that cannot be converted makes the row invalid. For timestamps without a time zone offset, set `parse.timezone` to an IANA time zone such as `Europe/Berlin` (default `UTC`).

To use a column as the timestamp of the message, set `timestampColumn` to its name and give it a `datetime` type. Without `timestampColumn`, the timestamp is the time when the connector read the row.

{% code lineNumbers="true" %}

```yaml
subscribe:
  mode: follow
  path: quality.csv
  timestampColumn: ts
  parse:
    delimiter: ';'
    decimalSeparator: ','
    timezone: Europe/Berlin
    types:
      ts: 'datetime:dd.MM.yyyy HH:mm:ss'
      thickness: number
      ok: boolean
```

{% endcode %}

### Invalid Rows

The `onParseError` property defines what happens when a row is invalid, for example because a value cannot be converted:

* `skipAndReport` (default): The connector skips the row, logs a warning, and continues.
* `stop`: The connector stops reading for this endpoint at the first invalid row.

Whatever `onParseError` is set to, the connector also stops reading when it cannot parse a file at all. This happens for a control character other than tab, carriage return, or line feed in the first 512 bytes of a file, which usually means a binary file, for a header line that contains the same column name twice, for a byte order mark that does not match `parse.encoding`, and for a row or header line larger than `parse.maxRecordBytes` (default 1 MiB), which usually means that `parse.recordDelimiter` or `parse.encoding` does not match the file.

When reading stops, the endpoint keeps its **Enabled** status, and the Protocol Mapper or agent log names the cause, but not the file or the line. The `agent_endpoints_halted` metric counts these endpoints, see [Metrics Reference](https://docs.cybus.io/2-7-0/monitoring/connectware-metrics/metrics-reference.md#protocol-mapper). To continue, fix the file or the configuration, then disable and enable the service. The endpoint also starts again when its connection reconnects, and stops again at the same row if the file is unchanged. Increasing `checkpointGeneration` also starts a stopped endpoint again, from the position described in [Where Reading Starts](#where-reading-starts).

## Published Messages

Each row becomes one MQTT message on the topic of the endpoint. With the default `payload: full`, the message contains the row and where it comes from:

{% code lineNumbers="true" %}

```json
{
  "timestamp": 1791454200000,
  "value": { "ts": 1791454200000, "temperature": 22.6, "pressure": 1.05 },
  "file": {
    "path": "/mnt/line3/process.log",
    "identity": "head:150:10af34dba8da92a02e5905284b71d7c162897ede",
    "line": 5,
    "byteOffset": 150
  },
  "dedupKey": "head:150:10af34dba8da92a02e5905284b71d7c162897ede:150"
}
```

{% endcode %}

* `timestamp`: Milliseconds since epoch, from `timestampColumn` or the time when the connector read the row.
* `value`: The row, as an object keyed by column names or as an array (see [Column Names](#column-names)).
* `file.path`: The file that the row comes from.
* `file.identity`: How the connector recognizes the file.
* `file.line`: The line on which the row ends, counted from 1. It is `null` when the connector started at the end of the file (`startAt.type: end` or `lastRows`).
* `file.byteOffset`: The position in the file right after the row, in bytes.
* `file.epoch`: How often the file was truncated or overwritten. Only present after the first time.
* `file.generation`: The `checkpointGeneration` of the endpoint. Only present when it is above 0.
* `dedupKey`: An identifier of the row, in the form `<identity>:g<generation>:e<epoch>:<byteOffset>`. The `g` and `e` parts are only present when `file` contains `generation` and `epoch`, so without them the key is `<identity>:<byteOffset>`, as in the example. It stays the same when the connector publishes the row again, with one exception for small files with content identity. See [Delivery and Duplicates](#delivery-and-duplicates).

With `payload: minimal`, the message has the same shape as messages of other connectors:

{% code lineNumbers="true" %}

```json
{
  "timestamp": 1791451936933,
  "value": { "ts": "2026-10-08 11:32:00", "spindle_speed": 12100, "load": 0.55 }
}
```

{% endcode %}

With `includeRaw: true`, every message also contains `raw`, the row exactly as it is written in the file, without its line break.

In both payload modes, [rules](https://docs.cybus.io/2-7-0/data-flows/rule-engine.md) can read `$context.file`, `$context.dedupKey`, and, with `includeRaw: true`, `$context.raw`. The following rule adds the source file and `dedupKey` to a minimal message:

{% code lineNumbers="true" %}

```yaml
dailyLogs:
  type: Cybus::Endpoint
  properties:
    protocol: File
    connection: !ref line3Files
    topic: logs
    subscribe:
      mode: followMany
      path: logs
      payload: minimal
    rules:
      - transform:
          expression: |
            {
              "timestamp": timestamp,
              "value": value,
              "sourceFile": $context.file.path,
              "dedupKey": $context.dedupKey
            }
```

{% endcode %}

## Delivery and Duplicates

The connector saves its read position in Connectware, by default every 100 rows or every second. To change this, set `checkpoint.flushEveryRows` and `checkpoint.flushIntervalMs` on the endpoint. After a crash of the Protocol Mapper or agent, it publishes the rows since the last saved position again. When the connection to the MQTT broker drops, it stops reading. Once the connection has been back for 5 seconds without interruption, it continues from the saved position. To change this wait, set the `CYBUS_FILE_DOWNSTREAM_STABLE_MS` environment variable of the Protocol Mapper or agent, in milliseconds.

If the network fails without an error, for example because of a pulled cable or a firewall that drops packets, the Protocol Mapper or agent notices the failure only when the keep-alive interval of its MQTT connection runs out. Until then, it keeps reading and publishing. The rows at risk are therefore all rows that the connector reads in one keep-alive interval, that is, the interval in seconds times the number of rows per second.

With the default `qos: 0` on the endpoint, the broker does not confirm messages, and rows that the Protocol Mapper or agent publishes on the failed connection are lost. To have the broker confirm every row, set `qos: 1` on the endpoint. The Protocol Mapper or agent then keeps each row until the broker confirms it, and sends unconfirmed rows again after the reconnect. If the Protocol Mapper or agent stops or crashes before the reconnect, these rows are lost even with `qos: 1`, because the connector saves its read position without waiting for the broker to confirm the rows.

A row that is published again has the same `dedupKey`, with one exception. Content identity is built from the first 4096 bytes of a file, so while a file is smaller than that, its identity changes as it grows. A row of such a file that is published again after a crash can therefore get a new `dedupKey`, and a consumer that drops duplicates keeps it. This applies to `identity: content` and to the default identity on `sftp`. To process each row exactly once, have the consumer drop messages whose `dedupKey` it has already seen, for example by using `dedupKey` as a unique key in a database. With `payload: minimal`, add it with a rule as shown in [Published Messages](#published-messages).

Two endpoints that read the same file publish the same keys. If you combine the messages of several endpoints, make the topic part of the unique key. With `identity: content`, two identical files, or two files whose first 4096 bytes are identical, get the same `file.identity`, so their rows get the same keys. In that case, also make `file.path` part of the unique key.

## Agents with Custom Roles

The connector saves read positions and learned SFTP host keys in the `resourceCheckpoints` bucket of Connectware, which needs NATS permissions. The built-in `protocol-mapper-agent` and `connectware-admin` roles include them. If the user of an agent has a custom role instead, add the following permissions to that role, each with the operation `write` and the context `control_plane`:

| Resource                                                                               |
| -------------------------------------------------------------------------------------- |
| `$JS.API.STREAM.INFO.KV_resourceCheckpoints`                                           |
| `$KV.resourceCheckpoints.*.*.*`                                                        |
| `$JS.API.DIRECT.GET.KV_resourceCheckpoints.$KV.resourceCheckpoints.*.*.*`              |
| `$KV.resourceCheckpoints.hostkeys.{agent}.>`                                           |
| `$JS.API.DIRECT.GET.KV_resourceCheckpoints.$KV.resourceCheckpoints.hostkeys.{agent}.>` |
| `$KV.resourceCheckpoints._probe.{agent}`                                               |
| `$JS.API.DIRECT.GET.KV_resourceCheckpoints.$KV.resourceCheckpoints._probe.{agent}`     |
| `$JS.API.CONSUMER.CREATE.KV_resourceCheckpoints.*.$KV.resourceCheckpoints.*.*.*`       |
| `$JS.API.CONSUMER.INFO.KV_resourceCheckpoints.*`                                       |
| `$JS.API.CONSUMER.DELETE.KV_resourceCheckpoints.*`                                     |
| `$JS.FC.KV_resourceCheckpoints.>`                                                      |
| `$JS.ACK.KV_resourceCheckpoints.>`                                                     |

Enter `{agent}` as written, including the braces. Connectware replaces it with an encoded form of the agent user name, so that each agent can access only its own host keys. Read positions are shared by all agents, so that another agent can continue where one stopped.

The Admin UI does not add permissions with the `control_plane` context. Add them with the `POST /api/roles/${ROLE_ID}/permissions` request of the [Auth Server API](https://docs.cybus.io/2-7-0/reference/api-reference/auth-server-api.md), one request per permission, where `${ROLE_ID}` is the ID of the role. `GET /api/roles` lists the roles with their IDs. For example, the request body for `$KV.resourceCheckpoints.*.*.*` is the following:

{% code lineNumbers="true" %}

```json
{
  "resource": "$KV.resourceCheckpoints.*.*.*",
  "operation": "write",
  "context": "control_plane"
}
```

{% endcode %}

If a permission is missing, File endpoints on the agent stop reading. Depending on the permission, the endpoints show **Deviated**, SFTP connections stay in **Reconnecting**, or the endpoints stay **Enabled** and the `agent_endpoints_halted` metric shows the reason `checkpointPermission`. Add all permissions from the table, then restart the agent.

## Service Commissioning File Example

This example reads a process log, job files, and daily log files from an SMB share that is mounted at `/mnt/line3`.

{% file src="/files/dPYfE5Z9nd68Uq94sSsM" %}

{% code title="file-connector-example.yml" lineNumbers="true" expandable="true" %}

```yaml
description: |
  File connector example. Reads CSV files from an SMB share that is mounted at /mnt/line3.

metadata:
  name: File Connector Example
  provider: cybus
  homepage: https://www.cybus.io
  version: 1.0.0

resources:
  line3Files:
    type: Cybus::Connection
    properties:
      protocol: File
      connection:
        transport:
          type: local
          basePath: /mnt/line3

  # One growing file without a header line. Publishes every row that is appended.
  processLog:
    type: Cybus::Endpoint
    properties:
      protocol: File
      connection: !ref line3Files
      topic: process
      subscribe:
        mode: follow
        path: process.log
        identity: content # SMB/CIFS mounts have no stable inode numbers
        startAt:
          type: beginning # read the existing rows once, then follow
        timestampColumn: ts
        parse:
          delimiter: ';'
          header: false
          columns: [ts, temperature, pressure]
          timezone: Europe/Berlin
          types:
            ts: 'datetime:yyyy-MM-dd HH:mm:ss'
            temperature: number
            pressure: number

  # One file per job. Publishes each file once and then moves it to jobs/processed.
  jobFiles:
    type: Cybus::Endpoint
    properties:
      protocol: File
      connection: !ref line3Files
      topic: jobs
      subscribe:
        mode: batch
        path: jobs
        identity: content
        pattern: 'job_*.csv'
        afterProcessing:
          action: move
          moveTo: jobs/processed
        parse:
          delimiter: ';'
          types:
            quantity: integer

  # One log file per day. Follows every matching file that was written to within the last hour.
  dailyLogs:
    type: Cybus::Endpoint
    properties:
      protocol: File
      connection: !ref line3Files
      topic: logs
      subscribe:
        mode: followMany
        path: logs
        identity: content
        pattern: 'log_*.csv'
        payload: minimal
        startAt:
          type: beginning
        parse:
          delimiter: ';'
          decimalSeparator: ','
          types:
            spindle_speed: integer
            load: number
```

{% endcode %}

## Troubleshooting

| Symptom                                            | Cause and solution                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| After enabling, the service shows **Deviated**.    | A setting is invalid. Some settings are only checked when you enable the service, for example a `basePath` outside `CYBUS_FILE_LOCAL_ROOTS` or a mode that the transport does not support. The Protocol Mapper or agent log names the setting. You can read the Protocol Mapper log in the [Container Logs View](https://docs.cybus.io/2-7-0/monitoring/system-status/system-container-status.md#container-logs-view). On an agent with a custom role, a permission can be missing, see [Agents with Custom Roles](#agents-with-custom-roles). |
| The connection stays in **Reconnecting**.          | The server is unreachable, the credentials are wrong, the share is not mounted, the SFTP host key does not match, or the Protocol Mapper or agent cannot read the file of a `follow` endpoint or the directory of a `batch` or `followMany` endpoint. On an agent with a custom role, the SFTP host key permissions can be missing, see [Agents with Custom Roles](#agents-with-custom-roles).                                                                                                                                                 |
| The endpoint is enabled, but nothing is published. | By default, `follow` starts at the end of the file and `batch` ignores files older than one day. See [Where Reading Starts](#where-reading-starts).                                                                                                                                                                                                                                                                                                                                                                                            |
| A file in a `batch` directory is not read.         | It is still changing (`settleMs`), it does not match `pattern`, it matches `ignorePattern`, it is in a subdirectory, or it is older than `maxFileAgeMs`. If the Protocol Mapper or agent cannot read the file, the log shows `File not readable (permission denied)`, the other files are read as usual, and the connector tries the file again until it can read it. Copying a file can keep its original modification time, which can make a new copy older than `maxFileAgeMs`.                                                             |
| Reading stopped for one endpoint.                  | The connector found a row or file that it cannot read. See [Invalid Rows](#invalid-rows).                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| Special characters are wrong.                      | `parse.encoding` does not match the file, for example for a file in `windows-1252`.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
