What is Kafka Connect?

Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems (for example, a connector to a relational database can capture changes to tables). You configure connectors: source connectors bring data from an external system into Kafka topics (their tasks use producers), and sink connectors export data from Kafka topics to an external system (their tasks use consumers). A connector is split into tasks, up to tasks.max, for parallelism. Connect runs in standalone mode (all work in a single process, suited to development and small deployments) or distributed mode (a cluster of workers that balances work, can be scaled up or down, and stores configs, offsets and status in Kafka topics). Converters (key.converter, value.converter) control the serialized format written to Kafka independently of the connector, and transformations make lightweight message-at-a-time changes. Connect manages offset commits automatically, and connectors are managed through a REST API.

Expected key concepts: connectors, source connectors, sink connectors, tasks.max, standalone, distributed, workers, converters, transformations, offset commits, REST API