NGINX stream Module: TCP and UDP Load Balancing

The http block only speaks HTTP. Databases, DNS, syslog, MQTT, SMTP and SIP all need something lower-level, and that is what the stream module is for: a TCP/UDP proxy and load balancer built from the same configuration language. This guide builds a working configuration from an empty file — listeners, upstream groups, balancing methods, timeouts and buffers — and covers the details that catch people out during a rollout.

stream vs http: What Changes

The stream context is a top-level block, sibling to http. TCP is the default protocol, so there is no tcp parameter on listen; UDP must be declared explicitly. Inside a stream server you use proxy_pass, not location blocks, and the directives are connection-oriented rather than request-oriented. Make sure the module is compiled in (nginx -V 2>&1 | grep -- --with-stream); on distro packages it usually ships in the same binary.

Minimal Proxies: One TCP Port, One UDP Port

stream {
    server {
        listen     12345;
        proxy_pass backend.example.com:12345;      # TCP by default
    }

    server {
        listen     53 udp;
        proxy_pass dns_backend;                    # UDP needs the udp parameter
    }
}

Choosing a balancing method

The stream module offers several algorithms, and the right one depends on how long a connection lives rather than on how much traffic it carries.

Two nuances matter in practice. With least_conn, a backend holding one very slow connection looks busy and stops being chosen, so a single long analytics query can pin a server out of rotation for minutes. The hash method gives affinity but knows nothing about load: a client that opens a hundred connections with the same key lands all hundred on one backend, which is fine for a session-oriented protocol and terrible for a fan-out one.

stream {
    upstream db_pool {
        hash $remote_addr consistent;    # stick a client to one backend
        server db1.example.com:5432 weight=3;
        server db2.example.com:5432;
    }

    server {
        listen     5432;
        proxy_pass db_pool;
    }
}

Long-lived TCP is where the choice really shows. A storage protocol such as NVMe over TCP or a database replication stream holds one connection for hours: round robin will hand the next connection to a backend that is already saturated, while least_conn keeps the two sides comparable.

Upstream Groups and Balancing Methods

Once more than one backend exists, define an upstream and choose how connections are spread. Round-robin is the default; least_conn is usually the better default for long-lived database or application sessions because it accounts for connections already open:

stream {
    upstream stream_backend {
        least_conn;
        server backend1.example.com:12345 weight=5;
        server backend2.example.com:12345 max_fails=2 fail_timeout=30s;
        server backend3.example.com:12345 max_conns=3;
        server backup1.example.com:12345 backup;      # used only when others fail
    }

    upstream dns_servers {
        least_conn;
        server 192.168.136.130:53;
        server 192.168.136.131:53;
        server 192.168.136.132:53;
    }

    server {
        listen              12345;
        proxy_pass          stream_backend;
        proxy_timeout       3s;      # idle timeout for the proxied connection
        proxy_connect_timeout 1s;    # how long to wait for a backend to accept
    }

    server {
        listen     53 udp;
        proxy_pass dns_servers;
    }
}

Note that active health checks (health_check) are an NGINX Plus feature. In open-source NGINX you rely on passive checks (max_fails/fail_timeout), so size fail_timeout against how quickly you need a dead backend out of rotation.

Timeouts, buffers and connection limits

Most stream problems in production come from the defaults rather than from the algorithm. Know what the defaults are before you tune anything.

Connection limits belong in the stream context too. limit_conn is the cheapest protection you can give a fragile backend: it caps how many simultaneous sessions one client may hold open, which is exactly the pattern a runaway application or a compromised host produces.

stream {
    limit_conn_zone $binary_remote_addr zone=perclient:10m;

    server {
        listen          6379;
        proxy_pass      redis_pool;
        limit_conn      perclient 20;
        proxy_connect_timeout 1s;
        proxy_timeout   5m;
        proxy_socket_keepalive on;
    }
}

A single proxy_timeout value for a 1-second DNS query and a 30-minute database session is a design bug waiting to be reported as an outage. Give each server block the timeout its protocol actually needs.

Binding, Buffers and Client IP Preservation

stream {
    server {
        listen            127.0.0.1:12345;
        proxy_pass        backend.example.com:12345;
        proxy_bind       127.0.0.1:12345;    # source address for outbound connections
        proxy_buffer_size 16k;               # or disable with proxy_buffering off
        proxy_protocol    on;                # forward original client address to the backend
    }
}

Two of these matter more than they look. proxy_buffer_size affects the first chunk of data from the backend — too small and you add latency on every connection. proxy_protocol on is how the backend learns the real client IP; without it, every session appears to come from the load balancer, which breaks access logs, per-IP rate limits and auditing.

Routing by SNI without terminating TLS

The stream module can read the TLS ClientHello and route on the server name while leaving the encrypted session untouched, which is why it is used in front of mail, database and IoT endpoints where certificates must stay on the backend, and in front of ingress controllers that terminate TLS themselves.

stream {
    map $ssl_preread_server_name $backend_by_sni {
        db.example.com    db_backend;
        mail.example.com  mail_backend;
        default           default_backend;
    }

    upstream db_backend      { server 10.0.0.11:5432; }
    upstream mail_backend    { server 10.0.0.12:993;  }
    upstream default_backend { server 10.0.0.13:443;  }

    server {
        listen 443;
        ssl_preread on;
        proxy_pass $backend_by_sni;
        proxy_timeout 30m;
    }
}

ssl_preread needs the stream_ssl_preread module, which is present in distribution packages but not in a minimal custom build; confirm with nginx -V before designing around it. If you would rather terminate TLS in nginx, the stream module's ssl_certificate and ssl_certificate_key directives work inside the server block, and certificate rotation then means a reload. Verify the chain before that reload rather than after - our notes on diagnosing TLS certificate chains with OpenSSL cover the checks that catch a missing intermediate ahead of your users.

UDP-specific behaviour

UDP has no connection, so nginx tracks a session keyed on the client address and port. Three consequences follow.

• There is no handshake to fail. The first datagram is proxied immediately, so proxy_connect_timeout rarely comes into play and passive health checking is weaker than for TCP.
• Responses must find their way back. By default the number of datagrams nginx expects in response is unlimited, which is what lets a DNS server send a truncated answer and be retried over TCP. Setting proxy_responses 1 fits single-datagram protocols such as RADIUS and syslog and keeps session state small.
• A reload discards session state. A client mid-conversation with an authentication server sees a timeout rather than a clean error, so schedule reloads outside peak authentication windows.

stream {
    upstream dns_pool {
        least_conn;
        server 10.0.0.53:53;
        server 10.0.0.54:53;
    }

    server {
        listen          53 udp;
        proxy_pass      dns_pool;
        proxy_responses 1;      # one answer per question through nginx
        proxy_timeout   5s;     # keep UDP session state short
    }
}

For DNS specifically, test with dig repeatedly and read which backend answered. If clients need large answers, remember that a stream listener on 53 and one on 53 udp are two independent servers: TCP fallback has to be proxied separately. Server side of that setup: BIND9 zone configuration.

Deploy and Verify

nginx -t                                  # syntax check before reload
nginx -T | sed -n '/stream {/,/^}/p'      # show the effective stream config
systemctl reload nginx

ss -ltnp | grep -E '12345|:53'            # listeners exist and belong to nginx
tail -f /var/log/nginx/error.log          # upstream connect errors appear here

Then test end-to-end rather than trusting the config: connect through the proxy to each backend, stop one backend and confirm connections still succeed (failover), and for UDP use dig @proxy-ip example.com repeatedly to see responses from more than one server.

DNS names in upstreams and re-resolution

When an upstream is written as a hostname, nginx resolves it once and keeps that address for the life of the worker. On a container platform where backend addresses change on every deploy, that is a slow-motion outage rather than an obvious failure.

stream {
    resolver 127.0.0.53 valid=10s ipv6=off;

    upstream app_pool {
        zone app_pool 64k;                    # needed when the upstream uses variables
        server app1.internal:8080 resolve;    # follow DNS for this name
        server app2.internal:8080 resolve;
    }

    server {
        listen     8080;
        proxy_pass app_pool;
    }
}

Three pieces have to be in place together: a resolver in the stream context, a zone declared on the upstream, and the resolve parameter on each server that should follow DNS. Shorten valid during a cutover and lengthen it afterwards, because every re-resolution is a query against your resolver - and the same problem, solved a layer up, is what makes a service discovery integration such as MetalLB address pools interesting.

Pitfalls

• Port conflicts: a stream listener on 443 and an http listener on 443 cannot coexist — decide which layer terminates TLS.
• Proxy timeouts too aggressive: a 3-second proxy_timeout is right for short queries and wrong for a database session that idles.
• No client IP at the backend: add proxy_protocol and configure the backend to expect it.
• Plaintext by default: stream proxying is transparent — if the protocol is not encrypted, neither is the hop through nginx.

Logging proxied sessions

Stream connections produce no access log lines by default, and the reason is practical: a stream server has no request and no URI to log. The variables that do exist are connection-oriented, which is enough to record who connected, where they were sent and how long it lasted.

stream {
    log_format proxy '$remote_addr:$remote_port [$time_local] '
                     '$protocol -> $upstream_addr '
                     'status=$status sent=$bytes_sent recv=$bytes_received '
                     'session=$session_time';

    server {
        listen     12345;
        proxy_pass stream_backend;
        access_log /var/log/nginx/stream.log proxy;
        error_log  /var/log/nginx/stream_error.log warn;
    }
}

Two fields in that log settle most arguments. $upstream_addr shows which backend actually accepted the connection, which turns "the load balancer is not balancing" into an observation. $status reports the session outcome, and a rising count of non-zero statuses next to a stable connection count is the earliest warning that a backend is degrading.

Sizing for thousands of concurrent connections

A stream proxy holds one file descriptor per direction per session, so the realistic ceiling is set by the operating system before nginx is the limiting factor.

# nginx.conf
worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
}

# /etc/security/limits.conf
# nginx soft nofile 65535
# nginx hard nofile 65535

Multiply worker_connections by the number of workers to get the theoretical session ceiling, then halve it, because each proxied session consumes two descriptors - one to the client, one to the backend. The error log message worker_connections are not enough is a clear signal to raise limits, but check the system-wide value first: raising nginx without raising the OS limit only moves where the failure appears. The kernel side of that arithmetic is covered in Linux network tuning with sysctl and MTU, MSS and PMTUD troubleshooting.

Troubleshooting checklist

Whichever protocol comes next, the pattern is identical: choose the algorithm for the session length, set the timeouts for the protocol, make the backend aware of the real client, and prove failover by stopping a server instead of by reading the configuration. For the HTTP side of the same server see nginx reverse proxy configuration, and for where stream proxying fits in a container platform see Kubernetes Gateway API versus Ingress.

相关阅读：NGINX 反向代理配置：location 与 proxy_pass、Linux 网络内核参数调优：TCP 缓冲与 backlog 以及 TCP MSS 钳制与 PMTUD 排障。

原文链接：NGINX Documentation - TCP and UDP Load Balancing