High availability, honestly
Every layer you add to survive a failure is a layer that can fail. The goal is fewer minutes of downtime per year, not more boxes.
Decide what you are protecting against
Different failures need different answers, and nothing covers all of them:
- A disk fails
- Mirrored disks. Cheap, effective, no second machine needed. See RAID and disks.
- A component fails
- Hardware replacement. With parts on site and engineers in the building, this is measured in minutes to hours — see hardware failure.
- The machine is unreachable
- A second machine, and a way to move traffic to it.
- You deployed something broken
- Neither of the above. This is the most common cause of downtime, and the answer is deployment practice and backups, not hardware.
That last row deserves emphasis. Most outages are self-inflicted. A cluster does not protect you from a bad migration; it runs it on every node.
A floating address with keepalived
Two machines share a service address. VRRP decides which holds it, and it moves within a couple of seconds if the holder stops responding.
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
virtual_ipaddress { 198.51.100.10 }
}
Two things to get right. Add a health check script so the address moves when your application fails rather than only when the machine dies. And make sure the two machines can always see each other — if they cannot, both claim the address, which is worse than the failure you were preventing.
Talk to us before you set this up: a floating address has to be routed to both machines, and that is arranged on our side.
Databases are the hard part
Everything above is easy because application servers are interchangeable. Databases are not.
- Asynchronous replication is simple and loses the last few transactions on failover. Acceptable for a great many businesses.
- Synchronous replication loses nothing and makes every write wait for the second machine.
- Automatic failover is where people get hurt. A false positive promotes a replica while the primary is still writing, and now you have two divergent databases. Manual failover with good monitoring is often the better engineering decision, and almost always the better one for a small team.
Count the honest numbers
One well-built machine with redundant disks, on a network with redundant carriers, with spare parts in the same building, is unavailable for a very small number of minutes a year. A two-node cluster run by people who are not sure how failover behaves can easily do worse, because it fails in ways nobody has practised.
If you do build it, test the failover on purpose, during the day, on a schedule. An untested failover is a theory, exactly like an untested restore.
What we provide underneath
Redundant carriers and routing, redundant power with UPS and generator, monitoring at the rack and engineers on site. That covers the layers below your operating system. Above it, the choices are yours — and we are happy to look at a design before you build it. Ask in a ticket; an engineer will read it.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base