How do you prevent consumer lag from growing under high load?

Consumer lag is LOG-END-OFFSET minus CURRENT-OFFSET, counted in records, so it grows when consumers process records more slowly than they are produced. Common approaches: 1) Add consumers to the group, up to the number of partitions. 2) Add partitions so more consumers can work in parallel. 3) Profile and speed up the processing itself. 4) Tune the fetch and poll settings: max.poll.records is the maximum number of records returned by one poll() (default 500), and fetch.max.bytes (default 50 MiB) and fetch.min.bytes bound how much data a fetch returns. 5) Keep the time between poll() calls under max.poll.interval.ms (default 5 minutes). 6) If you hand work to a thread pool, track offsets carefully so offset commits only cover records that have actually been processed.

Expected key concepts: consumer lag, LOG-END-OFFSET, CURRENT-OFFSET, partitions, max.poll.records, max.poll.interval.ms, fetch.max.bytes, thread pool, offset commits