DEV Community

INTFRAME
INTFRAME

Posted on Originally published at intframe.com

One box is enough

Everyone we meet assumes we run a fleet of cloud instances. We run one dedicated box in a Falkenstein data center, and it hosts every production site, database and bot we operate.

This is not nostalgia. It is a forcing function. When everything lives on one machine, you cannot hide sloppy engineering behind autoscaling. A memory leak is a neighbor problem within the hour. A runaway cron shows up in everyone's latency. So the discipline has to live in three places, and all three are enforced by config, not by promises.

1. Every service gets a ceiling

# systemd override, applied to every unit, no exceptions
[Service]
MemoryMax=512M
CPUQuota=80%
Restart=on-failure
RestartSec=5
Enter fullscreen mode Exit fullscreen mode

The ceiling is not there to be hit. It is there so that when something does go wrong, the kernel kills one service instead of the OOM killer choosing a victim at random among forty.

2. Every scheduled job gets a lock, a timeout, and a heartbeat