TCP tuning and BBR
On a 10 Gbit port, the kernel defaults are conservative. Four settings get you most of the available gain; the rest of the internet's advice is cargo cult.
Congestion control
The algorithm that decides how fast to send. The default on most systems is cubic, which treats packet loss as congestion — a reasonable assumption on a wire, a poor one on a path with any loss that is not congestion.
BBR models bandwidth and round-trip time instead, and on long or lossy paths it is substantially better:
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
Put that in /etc/sysctl.d/99-net.conf and
sysctl --system. The fq queueing discipline is not
optional — BBR relies on its pacing.
For a server delivering large files to distant clients, this is frequently the single most effective change available. See streaming and high-bandwidth delivery.
Buffers
At 10 Gbit over a long path, a single connection needs a large window to keep the pipe full:
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
Leave autotuning on; these set the ceiling it may grow to, not a fixed allocation. On a machine with modest memory and tens of thousands of connections, do not raise them this far — memory is per socket.
Accept queues
A busy server drops connections before your application ever sees them if the queues are too short:
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
And raise the matching limit in your application — nginx's
listen ... backlog=65535, your framework's equivalent. The kernel
setting alone does nothing if the application asked for 128.
Check whether it is even the problem
ss -ti shows per-connection congestion window, round-trip time and
retransmissions. Retransmissions near zero means the path is clean and tuning will
gain you little.
netstat -s | grep -i retrans for the totals, and
ip -s link for interface errors and drops. Interface errors are a
hardware matter and therefore ours — open a ticket.
What not to do
- Do not disable
tcp_timestamps. Old advice; it breaks window scaling behaviour you want. - Do not enable
tcp_tw_recycle. It no longer exists on current kernels, and it broke clients behind NAT when it did. - Do not paste a forty-line sysctl block from a forum. Change one thing, measure with fio and iperf3, keep what helped.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base