Prometheus and Grafana
Monitoring you built yourself beats a dashboard you rent, because you can ask it questions nobody anticipated.
The pieces
- Prometheus
- Scrapes metrics over HTTP, stores them as time series, evaluates alert rules.
- node_exporter
- Runs on every machine and exposes CPU, memory, disk, network, filesystem and systemd state. This one exporter answers most questions.
- Alertmanager
- Takes firing alerts and decides who gets told, with grouping and silences.
- Grafana
- The dashboards. Useful, and not where the value is — the value is in the alerts.
Add blackbox_exporter to probe your own services from outside, and the
exporter for whatever you run: nginx, postgres, mysqld, redis.
Keep it private
Prometheus and Grafana have no business being on a public address. Bind them to a private interface and reach them over WireGuard, or firewall them to your own addresses. An exporter endpoint is a detailed map of your infrastructure for anyone who can read it.
Put the monitoring on a different machine from the things it watches. A monitoring system that dies with the server it monitors has told you nothing.
Retention and disk
--storage.tsdb.retention.time=90d
Prometheus is compact — roughly one to two bytes per sample — but hundreds of series per host adds up. Watch its own disk usage and alert on it, which is a pleasing kind of recursion.
Alerts worth having, in order
- The service does not answer — a blackbox probe against the real URL. This is the only alert that maps directly to what a customer experiences.
- Disk will be full within 24 hours, using
predict_linear, rather than a flat 90% threshold. It warns you before it is urgent. - Certificate expires within 14 days — see TLS certificates.
- A systemd unit has failed.
- Memory available is low and swap is in use.
- Unusual outbound traffic, which is your earliest sign of a compromise — see understanding your traffic.
- SMART errors or a degraded array — disk health.
- A reboot is pending — automatic security updates.
Use for: 5m on nearly everything, so a single scrape failure does not
wake anybody.
Alert on symptoms, not causes
High CPU is not a problem; a slow site is. Alert on the things a user would notice, and use the dashboards to find out why afterwards. An alert nobody acts on gets ignored, and then the one that mattered is ignored too.
Have something outside your own infrastructure
If your whole estate is unreachable, your own monitoring cannot tell you. One external uptime check against your main service, from somewhere else entirely, covers the case your stack cannot. See monitoring your server.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base