Running your application as a service
If your application is started with nohup inside a screen session, it will be down the first time the machine reboots and nobody will know why.
A minimal unit
Put this in /etc/systemd/system/myapp.service:
[Unit]
Description=My application
After=network-online.target
[Service]
User=myapp
WorkingDirectory=/srv/myapp
ExecStart=/srv/myapp/bin/server
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
Then:
systemctl daemon-reload && systemctl enable --now myapp
That is the whole difference between a process that happens to be running and a service that starts at boot, restarts when it dies and can be managed like everything else on the machine.
Its own user, always
Create an unprivileged account that owns the application and nothing else:
useradd --system --home /srv/myapp --shell /usr/sbin/nologin myapp
An application running as root turns any remote code execution into a complete compromise. Running as its own user turns the same bug into a contained nuisance. See users and sudo.
Sandboxing, cheaply
Four lines in the [Service] section that cost nothing and remove whole
categories of damage:
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/myapp/data
ProtectSystem=strict makes the entire filesystem read-only for that
service except the paths you name. If your application is compromised, it cannot
write anywhere you did not explicitly allow.
Check how you are doing: systemd-analyze security myapp scores the
unit and tells you what else is available.
Logs go to journald
Anything your application writes to stdout and stderr is captured automatically:
journalctl -u myapp -f — follow it live
journalctl -u myapp --since "1 hour ago" -p err — only errors
That means your application does not need to manage log files, rotation or permissions. See log management for keeping the journal from filling the disk.
Restarts that do not make things worse
Restart=always with a crash loop will restart a broken service
hundreds of times a minute. Bound it:
StartLimitIntervalSec=300
StartLimitBurst=5
After five failures in five minutes systemd gives up and leaves the service failed, which is what you want — a failed unit is visible in monitoring, an endless loop just burns CPU and fills the journal.
Timers instead of cron
For scheduled work, a systemd timer gives you the same logging, the same dependency handling and the same sandboxing as a service. See cron jobs and timers.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base