What is Kafka's wire protocol and how does versioning work?

Kafka uses a custom binary TCP protocol over plain sockets. Every request/response pair has an API key (e.g., Produce=0, Fetch=1, Metadata=3) and an API version. Kafka is backward-compatible: newer brokers support older API versions for older clients. The client negotiates the highest mutually supported version on connect via ApiVersionsRequest. This allows rolling upgrades (old clients talk to new brokers and vice versa). The protocol is well-documented at kafka.apache.org/protocol.

Expected key concepts: binary TCP, API key, API version, backward-compatible, ApiVersionsRequest, rolling upgrades, Produce, Fetch, Metadata, negotiate