ZFS on a dedicated server
ZFS is a filesystem that assumes your hardware will lie to you, and checks. On a machine with ECC memory and real disks, it is hard to beat.
What you get
- Checksums on everything. Silent corruption is detected, and with redundancy it is repaired automatically.
- Snapshots that cost nothing until the data diverges. Take one before every upgrade.
- send/receive — incremental replication to another machine at block level, which is the best backup transport there is.
- Compression that usually makes things faster, because the disk moves fewer bytes.
Pool layout
Decide this before you create the pool; you cannot change a vdev's shape afterwards.
- Mirrors
- Pairs of disks. Best IOPS, fastest resilver, simplest to expand — add another pair whenever you like. The right default for anything with a database on it.
- raidz1
- One disk of parity. Use only with small disks; with modern large drives the rebuild window is long enough to be uncomfortable.
- raidz2
- Two disks of parity. The sane choice for bulk storage of six disks or more.
Always address disks by /dev/disk/by-id/, never /dev/sdb.
Device names move between boots and a pool that references them will eventually
confuse itself.
Memory, and ECC
ZFS uses free memory as its cache (the ARC) and gives it back under pressure. Plenty of memory makes it fast; a shortage makes it ordinary but not broken. On a machine that also runs a database, cap it so the two do not fight:
options zfs zfs_arc_max=8589934592 in
/etc/modprobe.d/zfs.conf — 8 GB
ECC memory is strongly advisable, because ZFS trusts what is in RAM to be what it checksums. Every machine we build has it; see ECC memory and memory errors.
Settings worth changing
zfs set compression=lz4 tank — on by default on modern versions,
and it is nearly always a win
zfs set atime=off tank — stop writing on every read
zfs set recordsize=16K tank/mysql — match the database page size;
the default 128K causes read amplification on random IO
Leave dedup off. It costs enormous memory and almost never pays for
itself. Compression gives you most of the benefit for none of the cost.
Snapshots and replication
zfs snapshot tank/data@before-upgrade
zfs rollback tank/data@before-upgrade — and you are back
Incremental replication to another machine:
zfs send -i @yesterday tank/data@today | ssh backup zfs recv
tank/data
Only the changed blocks cross the wire. Use a tool such as syncoid or
zrepl to schedule and prune it rather than writing your own. Remember
that a snapshot is not a backup until it is on another machine — see
backups.
Keep it healthy
zpool status -v — the first command when anything looks
wrong
zpool scrub tank — monthly, to read and verify everything
Alert on a pool that is not ONLINE and on checksum errors rising. A
disk reporting checksum errors is a disk on its way out; tell us and we will replace
it, as hardware failure describes. Keep the pool below
about 80% full — ZFS slows down markedly above that.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base