Network cards and offloads
A modern card does most of the work in hardware. When it does not reach line rate, the cause is usually one of a small number of things.
See what is on
ethtool -k eno1
You want these enabled, and they usually are:
- GRO and GSO — batch segments so the kernel handles fewer, larger units
- TSO — the card does the segmenting
- Checksum offload — the card computes checksums
- RSS — spread incoming flows across several queues and cores
If someone tells you to disable offloads, ask why. The legitimate reasons are narrow: packet capture that needs to see real frame boundaries, and a specific driver bug. Turning them off costs real throughput.
Ring buffers
ethtool -g eno1
If the preset maximum is far above the current setting, raising it reduces drops under burst:
ethtool -G eno1 rx 4096 tx 4096
Check whether you need it first — ethtool -S eno1 | grep -i drop.
Zero drops means leave it alone. Larger rings add latency, so this is a trade, not a
free win.
Interrupts across cores
If one core is at 100% in softirq while the others idle, all interrupts
are landing on it and that core is your ceiling.
cat /proc/interrupts | grep eno1
Install and enable irqbalance, or pin queues to cores by hand for a
latency-sensitive workload. On NUMA machines, pin them to the node the card is
attached to — crossing sockets for every packet is a measurable cost. See
choosing a CPU.
Queue lengths
For a high-rate server:
net.core.netdev_max_backlog = 16384
And watch the second column of /proc/net/softnet_stat — non-zero
means packets were dropped because the kernel could not keep up.
Measure properly
A single TCP stream will not fill a 10 Gbit port over any distance. Use parallel streams:
iperf3 -c host -P 8 -t 30
and test against something nearby first, so you are measuring the machine rather than the path. See benchmarking and speed testing.
When it is not you
Interface errors, CRC errors or drops reported by ip -s link are not a
tuning matter. They are cabling, an optic or a port — our hardware, our problem.
Send us the counters and the machine identifier and we will check it from the switch
side. See hardware failure.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base