Apache Kafka is a software framework for stream-processing of large amounts of data. It is an open-source software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
This protocol implementation allows to send the data from the Connectware to an Apache Kafka broker for message processing or to subscribe to a particular topic to receive data.
Currently, the protocol supports authentification by means of SASL and compression of produced messages by using GZIP.
Message assembly
The endpoints in this protocol are available for the write and subscribe operations only. To transmit messages to the Kafka broker, a JSON message with the payload as value needs to be sent to the respective endpoint. The message may look as follows when producing messages to the broker:
The Endpoints for the Kafka connections may only be of the type write or subscribe.
Special care must be taken when configuring the connection properties since connections can only take all write endpoints or all subscribe endpoints. But not both at the same time.
To choose the type of connection you are planning to use, you will need to configure the property clientType on the connection resource.