Redpanda in 2026: an architecture note for teams evaluating Kafka-compatible streaming without ZooKeeper or JVM overhead

Redpanda founder and CEO Alexander Gallego on the Streamfest speaker roster, a topic-grounded visual anchor for the platform's architecture and operator ecosystem.

The case for Redpanda starts with two removals. No ZooKeeper. No JVM. That framing is useful as a shortcut, but it lands differently depending on whether a team's real pain is operational coordination overhead, GC-induced latency spikes, or something more mundane — like not wanting to explain to a newcomer why their event broker requires a separate distributed coordination service that itself needs operational discipline.[1][3]

The fuller picture is an architecture built around a thread-per-core model via the Seastar framework, Raft consensus embedded directly into the broker, and Kafka protocol compatibility implemented at the API surface rather than through shared code. Whether that combination is the right tradeoff depends on what a team is actually giving up and what they are buying.

The thread-per-core model and why it changes latency shape

Seastar assigns each application thread to exactly one CPU core. That thread owns its local memory, its event loop, and its I/O queue.[1] When two cores need to share work, they communicate through explicit message passing rather than shared mutable state. The result is a design with no lock contention across threads and no thread context switches within a partition's serving path.

The operational consequence is tail latency behavior. JVM-based brokers carry GC pauses that are bounded but not zero — and GC pressure tends to become visible under bursty ingestion or large partition counts, where heap churn is highest. A C++ broker with a shared-nothing core cannot have GC pauses by construction. Whether that matters for a given workload depends on whether p99 or p999 latency is actually a constraint, not on abstract performance claims.[1][3]

One tradeoff that is less advertised: Seastar's shared-nothing model means that cross-core work requires explicit scheduling through its actor-like messaging layer. Under pathological access patterns — large fan-out consumer groups pulling from many partitions simultaneously — that inter-thread coordination shows up in CPU profiles. It does not look like lock contention; it looks like scheduling overhead on the message-passing path.

Raft per partition and what replaces ZooKeeper

Each partition in Redpanda is backed by a dedicated Raft group: one leader and zero or more followers.[1] Leader election and log replication happen through heartbeats (default 150ms) and follower timeouts (default 1.5 seconds). A Raft group tolerates f failures with 2f+1 nodes — standard quorum math.

A controller partition holds cluster-level metadata and creates snapshots at configurable intervals (default 60 seconds).[1] There is no separate coordination service. The system that replicates your data is the same system that manages cluster membership and topic configuration.

The architectural significance is not just deployment simplification. ZooKeeper-era Kafka required operators to maintain a second distributed system with its own failure modes, session timeout tuning, and znodes hygiene. That system could drift out of sync with broker state in ways that were hard to observe. Redpanda's controller partition is visible through the same observability tooling as any other partition — there is no second debugging surface to understand.

The failure mode to watch is controller partition quorum loss. If a majority of nodes hosting the controller lose connectivity, metadata writes stall. Partition leadership can still serve existing data from surviving nodes, but operations that require metadata progression — topic creation, configuration changes, rebalancing — block until quorum is restored. Teams that run 3-node clusters should model this scenario explicitly; a 3-node quorum tolerates exactly one simultaneous failure.

Kafka compatibility: real but not total

Redpanda implements the Kafka wire protocol, and standard Kafka clients — librdkafka-based clients in C/C++, Python, Rust, and .NET; the Apache Kafka Java client; franz-go for Go; KafkaJS for Node.js — work against Redpanda brokers without modification.[2] Protocol-level compatibility starts from Kafka 0.11.

The incompatibilities worth knowing before production migration:

SASL/SCRAM: Redpanda supports SCRAM-SHA-256 or SCRAM-SHA-512 per user, but not both simultaneously for the same user. Configurations that rely on multiple mechanisms per principal need adjustment.[2]
Per-user quotas: Bandwidth and API rate quotas per individual user are not available. Per-client-ID and per-client-group quotas work.[2]
HTTP Proxy (Pandaproxy): The built-in HTTP/REST proxy supports data production and consumption. Administrative operations — topic creation, ACL management — require the native Kafka API or Redpanda Admin API, not the HTTP proxy.[2]
Kafka Streams and Connect: These are Kafka JVM-ecosystem components that talk to a broker but run as separate processes. They are generally compatible, but integration testing against a real Redpanda cluster is necessary before treating compatibility as assumed.

The Schema Registry that ships with Redpanda is compatible with the Confluent Schema Registry HTTP API, storing and managing Avro, Protobuf, and JSON schemas and supporting server-side schema ID validation.[4] Teams already using Confluent Schema Registry clients do not need to swap client libraries.

Tiered Storage and the operational model it changes

Tiered Storage offloads log segments to object storage (S3, GCS, Azure Blob) near real time while keeping recent segments on local disk.[1] Consumers interact with both tiers through the same API — the offset range they are consuming determines which tier serves the data transparently.

The operational implication is that local disk is no longer the retention limit. A cluster sized for active throughput can hold years of event history in object storage without proportionally growing broker disk. The tradeoff is that fetches from the cold tier carry object-storage access latency and egress cost. Teams running consumer workloads that replay large historical ranges should profile that cost path before treating infinite local retention as the alternative.

License and governance posture

Redpanda is licensed under the Redpanda Business Source License (BSL/BUSL). BSL source is available and modifiable for non-competing use; the license converts to a permissive open-source license (Apache 2.0) after four years from each release.[3] The practical meaning for most self-hosted production teams is that the source is readable and auditable, and the operational model is standard open-source for internal deployment. Teams building a competing streaming service on top of Redpanda or redistributing it as infrastructure need to read the license terms specifically.

Where this changes the operational surface

The genuine wins are:

One binary to deploy and version; no separate coordination service to operate.
Predictable tail latency shape without GC tuning.
Kafka API compatibility that covers the majority of existing client code without migration cost.

The genuine cautions are:

Controller partition quorum sizing is now your metadata reliability story, not someone else's.
SASL and quota gaps require explicit verification against your auth requirements before migration.
C++ debugging surface is different from JVM tooling; thread-dump-style diagnostics do not apply.

Teams coming from Kafka with ZooKeeper who have already migrated to KRaft are in a different position than teams still carrying ZooKeeper. For the latter, Redpanda's operational model is clearly simpler. For the former, the comparison narrows to performance profile and ecosystem depth.

cronfeed.work