Overview
Apache Kafka is the industry standard for high-throughput event streaming. Modern Kafka (v3.3+) has officially deprecated ZooKeeper in favor of KRaft (Kafka Raft), an internal consensus protocol that significantly simplifies deployment and improves scalability. Configuring server.properties correctly is essential for data durability and cluster stability.
Data Durability
Configure min.insync.replicas and unclean leader election to prevent silent data loss during node failures.
KRaft Consensus
Configure process roles (broker/controller) and quorum voters to eliminate the need for ZooKeeper.
Listener Security
Map protocol interfaces to ensure internal broker traffic is separated from external client traffic.
Production Checklist
- Advertised Listeners: The most common Kafka connection error is misconfiguring
advertised.listeners. It MUST resolve for the client machines, not just the broker. - Format Storage: When using KRaft, you cannot just start the broker. You must first generate a UUID and format the log directories using
kafka-storage.sh. - Replication Factor: Ensure your default topic replication factor is 3, and your
min.insync.replicasis 2. This allows you to lose one broker without downtime or data loss. - Hardware: Kafka is highly dependent on sequential disk I/O. Use fast NVMe SSDs or provisioned IOPS EBS volumes for the
log.dirs.