Benchmarking a new server
Benchmarks are easy to run and easy to get wrong. Most of the figures people quote at each other are measuring page cache, a warm-up period or their own laptop's connection.
Before anything: decide what you are testing
“Is this server fast?” is not a question with an answer. “Can this disk sustain 20,000 random write operations per second at a queue depth my application actually produces?” is. Start from the workload, pick the measurement, then run it.
Disks: use fio, not dd
dd measures a single sequential stream through the page cache. It is
the least representative test possible for a server. Use fio, with a
dataset larger than RAM so you are measuring the disk rather than memory:
fio --name=randread --ioengine=libaio --direct=1 --rw=randread --bs=4k
--size=20G --numjobs=4 --iodepth=32 --runtime=120 --time_based --group_reporting
Swap randread for randwrite or randrw to
match your workload. The numbers to read are IOPS and the latency percentiles —
particularly the 99th. An average latency of 1 ms with a 99th percentile of 400 ms
is a disk that will make your application feel broken even though the average looks
excellent.
Note: --direct=1 bypasses the cache, which is the
point. And do not run a destructive write test against a filesystem holding anything
you want to keep.
Network: use iperf3, and know what you are measuring
A single TCP stream to a distant endpoint measures the path, not your port. For a 10 Gbit port, use parallel streams:
iperf3 -c <host> -P 8 -t 30
And test against something nearby first. If you cannot fill the port against a well-connected host in Amsterdam, the problem is local; if you can, and a distant destination is slow, the problem is the path. See speed testing for the longer version and routing problems for what to do next.
CPU: match the test to the work
Synthetic CPU scores are nearly useless for choosing between machines, because real workloads are bound by different things — memory bandwidth, single-thread clock, or instruction set extensions. Compile your own software, run your own query set, encode your own file. Fifteen minutes of your own workload tells you more than any score out of ten.
The five mistakes
- Testing a dataset that fits in RAM. You have benchmarked memory.
- Running for ten seconds. Flash drives have write caches and thermal behaviour; run for minutes, not seconds.
- Testing a filesystem that is nearly full. Allocation behaviour changes badly above about 85%.
- Benchmarking from a laptop over wifi and concluding something about the server.
- Running one test once. Run it three times. If the numbers move by 30%, something else is going on — find that before you trust any of them.
Benchmark before you go live
The useful moment for all of this is the first hour, while the machine is empty and you can be destructive. Record the figures somewhere. When someone claims six months later that the server “got slower”, that record is the difference between a diagnosis and an argument.
If your numbers look wrong to you, send them to us with the exact commands. We will run the same test from our side. If the hardware is underperforming, that is ours to fix.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base