Linux I/O Models

How applications wait for data — the foundation of every high-performance server.

Each section below ends with a quick check — try to answer before revealing:

The Five I/O Models

graph TD
    classDef blocking  fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef nonblock  fill:#e67e22,stroke:#d35400,color:#fff
    classDef mux       fill:#3498db,stroke:#2980b9,color:#fff
    classDef async     fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef signal    fill:#9b59b6,stroke:#8e44ad,color:#fff

    BLK["Blocking I/O read() blocks until data arrives Process sleeps in S state Simple but wastes a thread per connection"]:::blocking

    NB["Non-blocking I/O read() returns EAGAIN if no data Process must poll in a loop (busy-wait) Wastes CPU"]:::nonblock

    SEL["I/O Multiplexing select/poll/epoll Monitor many fds at once Block until any fd is ready One thread handles N connections"]:::mux

    SIG["Signal-driven I/O SIGIO signal when data ready Rarely used in practice"]:::signal

    AIO["Async I/O io_uring / aio Kernel does I/O in background App gets completion notification True async: no blocking ever"]:::async

Blocking I/O — What Actually Happens

sequenceDiagram
    participant APP as Application Thread
    participant KERN as Kernel
    participant NIC as NIC / Disk

    APP->>KERN: read(fd, buf, 1024)
    Note over APP: Thread blocked (S state) Scheduled out by kernel No CPU consumed while waiting
    NIC->>KERN: Data arrives (DMA into kernel buffer)
    KERN->>KERN: Copy data from kernel buffer to user buf
    KERN-->>APP: read() returns (thread woken up)
    Note over APP: Thread running again

Step through the same wait, one moment at a time:

The problem: One thread blocked = one thread wasted. For 10,000 concurrent connections, you need 10,000 threads. Each thread costs ~8MB stack → 80GB RAM just for stacks. That's the C10K problem.

select and poll — The Old Way

graph LR
    classDef app   fill:#3498db,stroke:#2980b9,color:#fff
    classDef kern  fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef limit fill:#e67e22,stroke:#d35400,color:#fff

    APP["App passes fd_set (bitmap of fds)"]:::app
    KERN["Kernel scans ALL fds in set O(N) scan every call"]:::kern
    COPY["Kernel copies entire fd_set back to userspace"]:::kern
    APP2["App must scan full set to find which fds are ready"]:::app
    LIMIT["select: max 1024 fds (FD_SETSIZE) poll: no limit but still O(N)"]:::limit

    APP --> KERN --> COPY --> APP2
    APP2 -.- LIMIT

select/poll problems:

• O(N) scan of all fds on every call — scales poorly past 1000 fds
• select hard limit: 1024 fds
• Full fd set copied kernel↔userspace on every call
• App must re-scan entire set to find ready fds

epoll — The Modern Way

epoll is O(1) for event notification regardless of how many fds you're watching. Used by nginx, Node.js, Redis, Go's netpoller.

sequenceDiagram
    participant APP as Application
    participant KERN as Kernel epoll instance

    APP->>KERN: epoll_create() — create epoll fd
    APP->>KERN: epoll_ctl(EPOLL_CTL_ADD, fd1, EPOLLIN)
    APP->>KERN: epoll_ctl(EPOLL_CTL_ADD, fd2, EPOLLIN)
    APP->>KERN: epoll_ctl(EPOLL_CTL_ADD, fd3, EPOLLIN)
    Note over KERN: Kernel registers interest list internally No repeated fd set copies

    APP->>KERN: epoll_wait(epfd, events, maxevents, timeout)
    Note over KERN: Thread blocks, kernel monitors all fds
    Note over KERN: fd2 becomes readable (data arrives)
    KERN-->>APP: returns 1 event: {fd2, EPOLLIN}
    Note over APP: Only process fd2, not all fds

One iteration of the event loop, step by step:

Why epoll is O(1):

• Interest list stored in a red-black tree inside kernel — epoll_ctl is O(log N)
• Ready list is a separate linked list — when an fd becomes ready the kernel adds it directly
• epoll_wait returns only ready fds — app never scans unready ones

Edge-triggered (EPOLLET) vs level-triggered (default):

• Level-triggered (default): epoll_wait returns as long as data is available. Safe but can cause many wakeups.
• Edge-triggered: epoll_wait returns only when state changes (new data arrives). More efficient but you MUST read until EAGAIN or you'll miss data.

// Adding fd with edge-triggered mode
struct epoll_event ev;
ev.events = EPOLLIN | EPOLLET;   // edge-triggered
ev.data.fd = client_fd;
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

Go's Runtime Netpoller (Built on epoll)

Go doesn't expose epoll directly. Its runtime wraps it transparently — Go code looks blocking but is actually non-blocking underneath.

graph TD
    classDef go    fill:#00add8,stroke:#007d9c,color:#fff
    classDef run   fill:#e67e22,stroke:#d35400,color:#fff
    classDef kern  fill:#2c3e50,stroke:#1a252f,color:#fff

    GOCODE["Go code: conn.Read(buf) looks like blocking I/O"]:::go
    RUNTIME["Go runtime: 1. Sets fd to non-blocking 2. Calls read() — gets EAGAIN 3. Registers fd with epoll 4. Parks goroutine (not OS thread)"]:::run
    EPOLL["Linux epoll waits for fd to be readable"]:::kern
    NETPOLL["Go netpoller goroutine calls epoll_wait when fd ready: unparks goroutine"]:::run
    RESUME["Goroutine resumes read() succeeds returns to Go code"]:::go

    GOCODE --> RUNTIME --> EPOLL
    EPOLL --> NETPOLL --> RESUME

The key insight: One OS thread runs many goroutines. When a goroutine would block on I/O, the runtime parks it and switches to another goroutine on the same OS thread. The OS thread never actually blocks — it's always running some goroutine. This is how Go handles 100,000 concurrent connections with far fewer OS threads than connections.

sendfile(2) and splice(2) — The Classic Zero-Copy Syscalls

The problem with naive I/O forwarding. Serving a static file over a socket the naive way looks like this:

ssize_t n = read(file_fd, buf, sizeof(buf));   // page cache -> userspace buffer
write(socket_fd, buf, n);                       // userspace buffer -> socket buffer

That's two data copies for bytes the application never actually needed to look at: read() copies the file's data from the kernel's page cache into a userspace buffer, then write() copies that same data straight back into a kernel-side socket buffer for transmission. Two copies, plus two syscall round trips into and out of the kernel — all to move data the app just passes through unchanged.

graph TD
    classDef app  fill:#3498db,stroke:#2980b9,color:#fff
    classDef kern fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef bad  fill:#e74c3c,stroke:#c0392b,color:#fff

    subgraph Naive["Naive read plus write: 2 copies"]
        direction TD
        PC1["Page cache file data"]:::kern
        UB["Userspace buffer read lands here"]:::bad
        SB1["Socket buffer write lands here"]:::bad
        NIC1["NIC"]:::kern
        PC1 -->|"copy 1: read()"| UB
        UB -->|"copy 2: write()"| SB1
        SB1 --> NIC1
    end

    subgraph ZeroCopy["sendfile / splice: 0 userspace copies"]
        direction TD
        PC2["Page cache file data"]:::kern
        SB2["Socket buffer kernel-internal transfer"]:::kern
        NIC2["NIC"]:::kern
        PC2 -->|"sendfile or splice, kernel only"| SB2
        SB2 --> NIC2
    end

sendfile(2) — fd to fd, entirely in kernel space

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

sendfile() copies data directly from one file descriptor to another entirely within kernel space — no userspace buffer ever gets involved. The classic call is sendfile(socket_fd, file_fd, &offset, count): the kernel reads from the file (page cache) and writes into the socket buffer itself, in one syscall, with zero copies into userspace. This is exactly what nginx does when you leave sendfile on; set (the default) to serve static assets — the file's bytes go straight from disk cache to network card without ever passing through the worker process's address space.

splice(2) — the general case, via a kernel pipe buffer

ssize_t splice(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out, size_t len, unsigned int flags);

sendfile() only covers one specific shape: read from this fd, write to that fd. splice() generalizes it — it moves data between two file descriptors via a kernel-only pipe buffer, without the restriction that one side has to be a regular file the way sendfile() effectively requires. That's what lets you splice socket → pipe → socket: proxying data between two TCP connections with the same zero-copy property, something sendfile() alone can't do since neither end there is a file. HAProxy uses splice() for exactly this — forwarding bytes between the client and backend connections it's proxying without ever copying the payload into its own userspace buffers.

How this fits next to io_uring

This file already covers io_uring's own zero-copy send/receive (IORING_OP_SEND_ZC, registered buffers, and so on) — so what's the difference? sendfile/splice are older, narrower, synchronous syscalls purpose-built to solve one problem: avoid the userspace round-trip when moving data between two kernel-visible endpoints. io_uring generalizes the same zero-copy idea across arbitrary async I/O operations — reads, writes, sends, receives, accepts — all through the same submission/completion ring, rather than one fd-to-fd forwarding call. In practice, which one a modern high-performance server reaches for depends on what it's already built on: a server already running an io_uring event loop stays in that ring and uses its zero-copy send/receive opcodes; a simpler blocking or thread-per-connection server (nginx's traditional worker model, HAProxy) reaches for sendfile/splice directly, since that gets the zero-copy win without restructuring the whole I/O loop around io_uring.

io_uring — True Async I/O (Linux 5.1+)

graph LR
    classDef sq   fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef cq   fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef kern fill:#e74c3c,stroke:#c0392b,color:#fff

    APP["Application"]
    SQ["Submission Queue (SQ) App writes I/O requests ring buffer shared with kernel no syscall needed"]:::sq
    KERN["Kernel processes SQ entries does I/O asynchronously"]:::kern
    CQ["Completion Queue (CQ) Kernel writes results App polls for completions no syscall needed"]:::cq

    APP -->|"writes request"| SQ
    SQ --> KERN
    KERN -->|"writes result"| CQ
    CQ -->|"app reads result"| APP

One submission/completion cycle, step by step:

Why io_uring is faster than epoll:

• Zero-copy between app and kernel (shared ring buffers)
• Batched submissions — submit 100 I/O operations with one syscall (or zero with SQPOLL)
• Works for files, not just sockets (epoll doesn't work on regular files — O_NONBLOCK on files is a lie)
• No context switches in SQPOLL mode — kernel thread polls SQ continuously

In Go: Go's runtime netpoller is still epoll-based (as of Go 1.23) — the standard library does not use io_uring. io_uring is available only through third-party libraries (e.g. iceber/iouring-go). Tokio (Rust) can use it heavily. For most workloads Go's epoll-based netpoller is excellent and io_uring is unnecessary.

Comparison

The one-line version of each model, side by side — the table below has the full breakdown: