Kafka vs Apache Pulsar Storage Architecture: PageCache Zero-Copy vs Disaggregated BookKeeper

In high-throughput event streaming infrastructure (Uber, LinkedIn, Netflix, DoorDash), message brokers process millions of events per second while ensuring high availability and zero data loss.

To handle massive write workloads, real-time message engines utilize two radically different architectural paradigms: Monolithic Coupled Storage (Apache Kafka) and Disaggregated Cloud-Native Storage (Apache Pulsar).

Apache Kafka achieves exceptional single-broker network throughput by leveraging the Linux OS PageCache and kernel sendfile() Zero-Copy I/O.

Conversely, Apache Pulsar decouples stateless message brokers from persistent storage nodes using Apache BookKeeper, delivering instant cluster scaling and zero-copy partition rebalancing.

This article details Kafka sequential disk log segments, OS PageCache mechanics, Linux sendfile() zero-copy transfers, Apache Pulsar disaggregated architecture, and BookKeeper quorum ledger writes.

Storage Architecture Comparison: Kafka vs Apache Pulsar

How Kafka's coupled PageCache Zero-Copy model compares to Apache Pulsar's disaggregated BookKeeper ledger architecture:

Core Streaming Storage Mechanics

1. Apache Kafka Storage Engine Architecture:
   • Coupled Compute + Storage: Kafka broker nodes handle both client TCP connections and physical disk log partition storage (/var/lib/kafka/data/topic-0/0000.log).
   • OS PageCache Primacy: Kafka avoids allocating large JVM heap buffers for message caching, relying instead on the Linux kernel PageCache. This avoids JVM GC pause overhead and maximizes available RAM utilization.
   • sendfile() Zero-Copy Transfer:
     • Traditional I/O (4 Context Switches, 2 CPU Copying Loops): Read disk to kernel buffer → Copy to JVM user buffer → Copy to socket buffer → Write to NIC.
     • Linux sendfile() Zero-Copy (2 Context Switches, 0 CPU Copy Loops): Transfers data directly from kernel PageCache to NIC DMA buffer using Direct Memory Access (DMA), cutting CPU context switching overhead in half!
2. Apache Pulsar Disaggregated Storage Architecture:
   • Decoupled Architecture: Pulsar separates stateless Broker Nodes (which handle pub/sub routing and protocol parsing) from Storage Nodes (Bookies) running Apache BookKeeper.
   • Segment-Centric BookKeeper Ledgers:
     • Topics in Pulsar are broken down into small, immutable Ledger Segments.
     • Ledgers are striped across multiple Bookies using Quorum Write parameters:
       • $Q_w$ (Write Quorum Size): Number of Bookies storing each entry.
       • $Q_a$ (Ack Quorum Size): Number of Bookie acknowledgments required before completing a write.
3. Cluster Auto-Scaling & Partition Rebalancing Trade-Offs:
   • Kafka: Adding a new broker requires copying gigabytes or terabytes of partition log data over the network from old brokers to the new broker (Heavy Partition Rebalancing).
   • Pulsar: Adding a new Broker or Bookie node requires zero data migration! New ledger segments are immediately allocated on the new Bookie, allowing instant elastic scaling.

Python Implementation: Kafka Zero-Copy vs Pulsar BookKeeper Engine

Here is a production-grade Python implementation of a Kafka Zero-Copy PageCache Replicator and a Pulsar BookKeeper Ledger Quorum Engine Simulator:

import time
from typing import Dict, List, Tuple
from pydantic import BaseModel

class StreamMessage(BaseModel):
    offset: int
    payload: str
    timestamp: float

class KafkaZeroCopyBrokerEngine:
    """
    Simulates Apache Kafka PageCache & Linux sendfile() Zero-Copy Transfer.
    """
    def __init__(self):
        # OS PageCache Simulation: { offset -> StreamMessage }
        self.page_cache: Dict[int, StreamMessage] = {}
        self.next_offset = 0

    def append_message(self, payload: str) -> int:
        offset = self.next_offset
        self.next_offset += 1
        msg = StreamMessage(offset=offset, payload=payload, timestamp=time.time())
        self.page_cache[offset] = msg
        print(f" 📥 [Kafka Append] Wrote Offset #{offset} ('{payload}') to Linux OS PageCache")
        return offset

    def sendfile_zero_copy_fetch(self, start_offset: int, max_messages: int = 2) -> List[StreamMessage]:
        """
        Simulates Linux sys_sendfile() Zero-Copy: PageCache -> DMA -> Network.
        Zero CPU copy loops into JVM user memory!
        """
        print(f"\n⚡ [Linux sendfile() Zero-Copy] Transferring offsets [{start_offset}..{start_offset + max_messages - 1}] directly from PageCache -> NIC")
        fetched = []
        for o in range(start_offset, start_offset + max_messages):
            if o in self.page_cache:
                fetched.append(self.page_cache[o])

        print(f" 🚀 [Zero-Copy Success] Sent {len(fetched)} messages over socket with 0 CPU user-space copies!")
        return fetched

class PulsarBookKeeperEngine:
    """
    Simulates Apache Pulsar Disaggregated BookKeeper Quorum Ledger Storage.
    """
    def __init__(self, write_quorum: int = 3, ack_quorum: int = 2):
        self.write_quorum = write_quorum
        self.ack_quorum = ack_quorum
        # Bookie nodes storage: { bookie_id -> { ledger_id -> [entries] } }
        self.bookies: Dict[int, Dict[int, List[str]]] = {1: {}, 2: {}, 3: {}}

    def write_ledger_entry_quorum(self, ledger_id: int, entry_data: str) -> bool:
        """Writes entry across Bookies using Quorum Write parameters (Qw=3, Qa=2)."""
        print(f"\n🧱 [Pulsar BookKeeper Write] Writing Ledger #{ledger_id} Entry: '{entry_data}' across Qw={self.write_quorum} Bookies...")
        ack_count = 0

        for bookie_id in range(1, self.write_quorum + 1):
            if ledger_id not in self.bookies[bookie_id]:
                self.bookies[bookie_id][ledger_id] = []
            
            self.bookies[bookie_id][ledger_id].append(entry_data)
            ack_count += 1
            print(f"   • Bookie #{bookie_id} ACKed write entry")

        if ack_count >= self.ack_quorum:
            print(f" 🎉 [Pulsar Quorum Met] Received {ack_count} ACKs (>= Qa={self.ack_quorum}). Write Committed!")
            return True
        return False

# Demonstration Execution
if __name__ == "__main__":
    print("🚀 Demonstrating Kafka Zero-Copy vs Pulsar BookKeeper Engines...")
    print("=" * 75)

    # 1. Demonstrate Kafka PageCache Zero-Copy
    kafka = KafkaZeroCopyBrokerEngine()
    kafka.append_message("user_signup_event_101")
    kafka.append_message("payment_processed_event_102")
    kafka.sendfile_zero_copy_fetch(start_offset=0, max_messages=2)

    # 2. Demonstrate Pulsar BookKeeper Quorum Ledger Write
    pulsar = PulsarBookKeeperEngine(write_quorum=3, ack_quorum=2)
    pulsar.write_ledger_entry_quorum(ledger_id=5001, entry_data="order_fulfilled_event_99")

Messaging Storage Gotchas & Best Practices

When choosing a streaming storage architecture:

Real-World Enterprise Impact

Streaming storage architectures (such as Kafka PageCache Zero-Copy and Pulsar BookKeeper) report:

• Over $4\times$ Higher Network Transfer Rates via Zero-Copy: Eliminating CPU user-space copying loops with Linux sendfile() allows Kafka brokers to saturate 100Gbps network interfaces.
• Instant Elastic Auto-Scaling: Pulsar's disaggregated BookKeeper architecture enables cluster scaling without moving gigabytes of historical log data.