> For the complete documentation index, see [llms.txt](https://docs.cybus.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file/fileendpoint.md).

# File Endpoint Properties

## `format` (string, enum)

File format. `csv` reads delimiter-separated text and publishes one message per row. The CSV options are in `parse`.

This element must be one of the following enum values:

* `csv`

Default: `"csv"`

Example: `"csv"`

## `mode` (string, enum, required)

How the connector reads files. `follow` reads one file and publishes rows as they are appended. `batch` reads each complete file in a directory once. `followMany` follows every matching file in a directory, each with its own read position, and stops checking a file after `inactiveMs` without growth. `follow` and `followMany` need the `local` or `sftp` transport. Properties that apply only to other modes are rejected unless they are set to their default value.

This element must be one of the following enum values:

* `follow`
* `batch`
* `followMany`

Example: `"batch"`

## `path` (string, required)

The file to read in `follow` mode, or the directory to watch in `batch` and `followMany` mode. Files in subdirectories are not read. Relative paths resolve against `transport.basePath` on `local`, and against the login directory on `sftp` and `ftp`. On `local`, absolute paths must be inside `basePath`.

Example: `"/line3/jobs"`

## `pattern` (string)

Modes `batch` and `followMany`. Glob pattern for the names of the files to read. Matching is case-sensitive, so the default `*.csv` does not match `DATA.CSV`, and `*` does not match names that start with a dot.

Default: `"*.csv"`

Example: `"job_*.csv"`

## `ignorePattern` (string)

Modes `batch` and `followMany`. Glob (minimatch syntax) of file names to ignore even if they match `pattern`, typically temporary names a writer uses before renaming the finished file.

Example: `"*.tmp"`

## `markerSuffix` (string)

Mode `batch` only. Suffix of a marker file that signals that a data file is complete. A file such as `job.csv` is read as soon as `job.csv<markerSuffix>` exists, without waiting for `settleMs`. Files without a marker are still read after `settleMs`. Set a high `settleMs` to make the marker effectively required. Marker files are never read as data, and `afterProcessing` moves or deletes them together with their file.

Example: `".done"`

## `settleMs` (integer)

Mode `batch` only. A file is considered complete once its size and modification time have not changed for this many milliseconds. Protects against reading files that are still being written. `0` processes files as soon as they appear.

Default: `3000`

## `afterProcessing` (object)

Modes `batch` and `followMany`. What happens to a file once the connector is done with it. In `batch` mode, that is after all its rows are published. In `followMany` mode, it is after the file has not grown for `inactiveMs`. Files that are already inactive at the first start are never moved or deleted. If moving or deleting fails, the error is logged, and the file stays in place without being read again.

Properties of the `afterProcessing` object:

#### `action` (string, enum)

Action for a finished file. `none` leaves the file in place. The connector remembers it and reads it again only if it changes. `move` moves the file to `moveTo`. `delete` deletes the file.

This element must be one of the following enum values:

* `none`
* `move`
* `delete`

Default: `"none"`

#### `moveTo` (string)

Directory to move files to. Required when `action` is `move`. The connector creates it if it does not exist. Relative paths resolve like `path`, not against the watched directory. On `local`, it must be inside `basePath`. If a file with the same name already exists there, the moved file gets a timestamp suffix.

Example: `"/line3/jobs/processed"`

Default:

```
{
  "action": "none"
}
```

## `maxFileAgeMs` (integer)

Modes `batch` and `followMany`. Maximum age of a file in milliseconds, based on its modification time. Older files are not read. `0` reads files of any age.

Default: `86400000`

## `maxProcessedEntries` (integer)

Mode `batch` only. Number of read files that the endpoint remembers. Above it, the endpoint forgets files that are no longer in the directory, oldest first. Files that are still in the directory are always remembered, so they are never read twice. If the directory holds more files than the endpoint can remember, it stops reading new files and logs an error. For directories that receive many files, set `afterProcessing` to `move` or `delete`.

Default: `2000`

Additional restrictions:

* Minimum: `100`
* Maximum: `5000`

## `processExisting` (boolean)

Mode `batch` only. If `false`, files that are in the directory at the first start are skipped, and only files that arrive later are read. A skipped file is read in full if it changes later. If `true`, existing files are read, except files older than `maxFileAgeMs`.

Default: `true`

## `startAt` (object)

Modes `follow` and `followMany`. Where to start reading when the endpoint has no saved read position: at the first start, after you increase `checkpointGeneration`, or after you change `path` or `mode`. In `followMany` mode, it applies to each file that exists at the first start. Files that appear later are read from their beginning.

Properties of the `startAt` object:

#### `type` (string, enum)

Start position. `end` publishes only rows that are appended after the start. `beginning` publishes the whole file. `lastRows` publishes the last `rows` rows. `since` publishes rows whose `timestampColumn` value is at or after `since`. Rows without a valid timestamp are skipped.

This element must be one of the following enum values:

* `end`
* `beginning`
* `lastRows`
* `since`

Default: `"end"`

#### `rows` (integer)

Number of trailing rows to publish on cold start. Required when `type` is `lastRows`.

Example: `100`

Additional restrictions:

* Minimum: `1`

#### `since` (string)

ISO 8601 timestamp; rows with a `timestampColumn` value before it are skipped on cold start. Required when `type` is `since`, which also requires `timestampColumn`.

Example: `"2026-09-01T00:00:00Z"`

Default:

```
{
  "type": "end"
}
```

## `inactiveMs` (integer)

Mode `followMany` only. Time in milliseconds without growth after which a file counts as finished. The connector then stops checking the file and applies `afterProcessing`. If a finished file grows again, reading continues where it stopped. The connector remembers up to 2000 finished files, and a forgotten file that grows again is read from its beginning. Files that have not changed for this long at the first start are not read. Set it longer than the longest pause between two writes to the same file.

Default: `3600000`

Additional restrictions:

* Minimum: `1000`

## `maxTrackedFiles` (integer)

Mode `followMany` only. Maximum number of files followed at the same time. If more files are active, the newest are followed, and the others wait until a followed file finishes. A waiting file is then read from its beginning, regardless of `startAt`.

Default: `200`

Additional restrictions:

* Minimum: `1`
* Maximum: `2000`

## `timestampColumn` (string)

Column whose value becomes the message `timestamp`. Give the column a `datetime` type in `parse.types`, or `number` or `integer` if it holds milliseconds since epoch. Use the column name, or the zero-based index as a string for array rows. If the value is empty or invalid, or `timestampColumn` is not set, the timestamp is the time when the row was read. Required for `startAt.type: since`.

Example: `"ts"`

## `backfillRateLimitRowsPerSecond` (integer)

Maximum rows per second that the endpoint publishes from files that existed when it started. In `batch` mode, the limit covers all rows of those files. In `follow` and `followMany` mode, it covers the rows up to the end of each file at the start. Files that appear later are not limited. `0` disables the limit.

Default: `500`

## `checkpointGeneration` (integer)

Counter that resets the saved read position of the endpoint. Increase it to discard the position, including the list of files that a `batch` endpoint has read. Deleting the service does not discard the position: a service installed again with the same ID continues from it. The endpoint then starts as configured by `startAt`, `processExisting`, and `maxFileAgeMs`. A value above `0` is also part of every `dedupKey` (`<identity>:g<generation>[:e<epoch>]:<offset>`) and appears as `file.generation`, so rows that are read again after you increase it get new keys and a consumer that drops duplicates does not discard them. With `0` the keys have no generation part.

Default: `0`

## `checkpoint` (object)

How often the endpoint saves its read position. After a crash, rows published since the last save are published again. Consumers detect them by `dedupKey`, which the message includes only with `payload: full`.

Properties of the `checkpoint` object:

#### `flushEveryRows` (integer)

Persist the checkpoint after this many emitted rows. Lower values shrink the replay window at the cost of more writes to the store.

Default: `100`

Additional restrictions:

* Minimum: `1`

#### `flushIntervalMs` (integer)

Persist the checkpoint at least this often (in milliseconds) while rows are pending, whichever of the two limits is reached first.

Default: `1000`

Additional restrictions:

* Minimum: `100`

Default:

```
{}
```

## `identity` (string, enum)

How the connector recognizes a file after it is renamed, truncated, or replaced. `inode` uses the inode number and works only on `local`. `content` uses a hash of the first 4096 bytes of the file and works on `local` and `sftp`. Use it for SMB/CIFS mounts, where inode numbers can change. `auto` uses `inode` on `local`, `content` on `sftp`, and the path, size, and modification time on `ftp`. Changing `identity` makes the endpoint read its files again.

This element must be one of the following enum values:

* `auto`
* `inode`
* `content`

Default: `"auto"`

## `parse` (object)

How the file content is decoded into rows.

Properties of the `parse` object:

#### `delimiter` (string)

Character (or character sequence) separating fields, for example `,` `;` or `\t`. Must not contain the quote or escape character and no control character other than TAB (control bytes mark a record as binary content, see `onParseError`).

Default: `","`

Example: `";"`

#### `quote` (string)

Character enclosing fields that contain the delimiter, quotes or line breaks. Exactly one character, not a control character.

Default: `"\""`

#### `escape` (string)

Character that escapes a quote inside a quoted field. RFC 4180 doubles the quote, so the default equals `quote`. Exactly one character, not a control character.

Default: `"\""`

#### `encoding` (string, enum)

Character encoding of the file. A byte order mark that matches the encoding is removed, and a UTF-8 byte order mark is also removed for the single-byte encodings. A byte order mark of another encoding stops the endpoint with `CSV_ENCODING_MISMATCH`.

This element must be one of the following enum values:

* `utf8`
* `utf16le`
* `utf16be`
* `windows-1252`
* `iso-8859-1`
* `iso-8859-2`
* `iso-8859-3`
* `iso-8859-4`
* `iso-8859-5`
* `iso-8859-6`
* `iso-8859-7`
* `iso-8859-8`
* `iso-8859-9`
* `iso-8859-10`
* `iso-8859-11`
* `iso-8859-13`
* `iso-8859-14`
* `iso-8859-15`
* `iso-8859-16`

Default: `"utf8"`

Example: `"windows-1252"`

#### `recordDelimiter` (string, enum)

Line ending between records. `auto` accepts CRLF, LF, and CR. In `follow` and `followMany` mode, `auto` treats a file that ends with a lone CR as still being written, so set `\r` for files with CR-only line endings. With `\n`, a CR before the line break stays in the last field.

This element must be one of the following enum values:

* `auto`
*
*
*

Default: `"auto"`

#### `header` (boolean)

If `true`, the first line after `skipRows` is a header. The header line is not published, and without `columns` its names become the keys of each row. An empty name becomes `col<index>`. A header with duplicate names or a parse error stops the endpoint. If `false`, the first line is data. Use `columns` to name the columns.

Default: `true`

#### `columns` (array)

Column names to use as the keys of each row, regardless of `header`. With `header: true`, the header line is skipped, its names are ignored, and a different number of columns is logged once per file. With `header: false`, every line is data. Without `columns` and with `header: false`, rows are published as arrays, and `types` and `timestampColumn` refer to columns by their zero-based index as a string. When `columns` is set, `types` keys and `timestampColumn` must be names from it.

The object is an array with all elements of the type `string`.

Example: `["ts","temp","pressure"]`

Additional restrictions:

* Minimum items: `1`
* Unique items: `true`

#### `skipRows` (integer)

Number of lines at the start of the file to ignore before the header line (or the first data row), e.g. title or unit lines written by the exporting machine.

Default: `0`

#### `relaxColumnCount` (boolean)

If `true`, rows with fewer or more fields than columns are still published. Missing columns are left out. Extra fields are dropped from rows with named columns and kept in array rows. If `false`, such rows are parse errors handled by `onParseError`.

Default: `true`

#### `relaxQuotes` (boolean)

If `true`, a quote inside an unquoted field, such as the inch mark in `5"`, or a closing quote followed by other characters, is kept as data. If `false`, strict RFC 4180 quoting applies, and such rows are parse errors handled by `onParseError`. With either value, a field that opens with a quote that is never closed runs to the end of the file, or until it exceeds `maxRecordBytes`.

Default: `true`

#### `trim` (boolean)

Remove leading and trailing whitespace from every unquoted field.

Default: `false`

#### `maxRecordBytes` (integer)

Upper bound in bytes for a single record (row including quoted line breaks and its line terminator; blank lines in front of the record do not count). A record exceeding it — typically a file without the configured line ending or in a different encoding — halts ingestion for this endpoint with the error `CSV_MAX_RECORD_SIZE` regardless of `onParseError`, instead of buffering the whole file in memory.

Default: `1048576`

Additional restrictions:

* Minimum: `1024`

#### `types` (object)

Type per column, keyed by column name, or by the zero-based index as a string for array rows. Allowed values are `string`, `number`, `integer`, `boolean`, `datetime`, and `datetime:<format>`. `datetime` accepts ISO 8601. `datetime:<format>` takes a Luxon format such as `datetime:yyyy-MM-dd HH:mm:ss`, and `datetime:X` and `datetime:x` read seconds and milliseconds since epoch. Datetimes become milliseconds since epoch. `boolean` accepts `true`, `1`, `yes`, `on`, `false`, `0`, `no`, and `off` in any case. Empty values become `null`, except in `string` columns. A row with a value that cannot be converted is a parse error handled by `onParseError`. Columns not listed stay strings.

Example: `{"ts":"datetime:yyyy-MM-dd HH:mm:ss","temp":"number"}`

#### `decimalSeparator` (string, enum)

Decimal separator of `number` and `integer` columns in `types`. With `,` a dot is accepted only as a well-formed thousands separator (first group of 1-3 digits not starting with 0, then groups of exactly three: `1.000,5` becomes 1000.5, `1.500` becomes 1500); any other dot such as `1.5` or `0.500` is rejected as not a number, and a comma needs digits on both sides (`1,` and `,5` are rejected); with `.` a comma is not accepted. Epoch timestamps (`datetime:X`) always use `.`.

This element must be one of the following enum values:

* `.`
* `,`

Default: `"."`

#### `timezone` (string)

IANA time zone, for example `Europe/Berlin`, for `datetime` values without a UTC offset, including ISO 8601 values.

Default: `"UTC"`

Example: `"Europe/Berlin"`

Default:

```
{}
```

## `onParseError` (string, enum)

How the endpoint handles a malformed row. `skipAndReport` logs it with file, line, and byte offset, skips it, and continues. `stop` stops the endpoint at the first malformed row. A stopped endpoint restarts when it is enabled again or its connection reconnects, and it stops again at the same row unless the file was corrected. Rows with values that do not match their `types` are malformed too. Regardless of this setting, the endpoint stops on a header with duplicate names, a byte order mark that does not match `parse.encoding`, a record larger than `parse.maxRecordBytes`, and a control character other than TAB, CR, or LF in the first 512 bytes of a file.

This element must be one of the following enum values:

* `skipAndReport`
* `stop`

Default: `"skipAndReport"`

## `payload` (string, enum)

Shape of the published message. `full` publishes `{ timestamp, value, file: { path, identity, line, byteOffset }, dedupKey }` (`file` also holds `epoch` after a truncation or rewrite and `generation` when `checkpointGeneration` is above `0`; both are part of `dedupKey`), and `minimal` publishes `{ timestamp, value }`. In both modes, rules of this endpoint can read `file` and `dedupKey` from the Rule Engine context. Consumers that drop duplicates by `dedupKey` need `full`, or a rule that adds `dedupKey` to the message.

This element must be one of the following enum values:

* `full`
* `minimal`

Default: `"full"`

## `includeRaw` (boolean)

If `true`, every message and the Rule Engine context carry `raw`, the record as written in the file, decoded with `parse.encoding`, without byte order mark and line ending. Logged parse errors include the raw record, truncated to 500 characters.

Default: `false`


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.cybus.io/2-7-0/connectors/shop-floor-connectors/file/fileendpoint.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
