How do you safely perform a rolling upgrade of a Kafka cluster?

Apache's upgrade notes (the 4.0.1 section, which the 4.1.0 notes say also applies to 4.1.x; the 4.2 section itself lists no separate upgrade steps) describe a rolling upgrade as: upgrade the brokers one at a time, shutting each broker down, updating the code and restarting it. Read the release notes first. Stop each broker gracefully rather than with a hard kill: with controlled.shutdown.enable (default true) the broker migrates the partitions it leads to other replicas before shutting down, and this only succeeds if all partitions it hosts have replicas. A common approach is to wait until the UnderReplicatedPartitions metric (normally 0) is back to 0 before moving to the next broker. Replication is what makes this possible: the docs note that a replication factor of 2 or 3 lets you transparently bounce machines without interrupting data consumption. Once you have verified behavior and performance, the 4.0 notes finalize the upgrade with kafka-features.sh upgrade --release-version, and warn that cluster metadata downgrade is not supported.

Expected key concepts: rolling upgrade, one at a time, controlled.shutdown.enable, UnderReplicatedPartitions, kafka-features.sh, restart, release notes, replication factor, gracefully, hard kill