What is Kafka's wire protocol and how does versioning work?

Kafka uses a binary protocol over TCP. All APIs are request/response pairs of size-delimited messages, and a broker processes the requests on one connection in the order they were sent. Every request carries an API key and an API version (two 16-bit numbers) that together identify the schema of the message that follows; for example Produce is key 0, Fetch is 1, Metadata is 3 and ApiVersions is 18. Kafka has a bidirectional compatibility policy: new clients can talk to old brokers and old clients to new brokers, which allows rolling upgrades of either side without downtime. After connecting, the client sends an ApiVersionsRequest, the broker returns the API keys and version ranges it supports, and the client uses the highest version supported by both sides. A broker rejects versions it does not support. Compatibility has a floor: Kafka 4.0 removed many deprecated APIs, and clients must be on version 2.1 or higher to work with 4.0 brokers. The protocol is documented in the Kafka protocol guide on kafka.apache.org.

Expected key concepts: binary protocol, TCP, API key, API version, bidirectional compatibility, ApiVersionsRequest, rolling upgrades, Produce, Fetch, Metadata, highest version