Servers

Linux server monitoring: what to watch, when to alert

Most server problems don't appear out of nowhere. A disk fills up over weeks, memory usage grows day by day and a certificate expires on a fixed date. The problem is that nobody is looking. The first sign is then a customer's call: "your site won't open".

Monitoring means the server itself tells you before something breaks. Good monitoring doesn't send you hundreds of notifications a day, only the ones that require you to do something.

What to monitor

Availability from the outside

The most important question is simple: does the site open? This has to be checked from outside the server, because if the server is down it can't report it itself. The check requests the page every few minutes and raises an alert if there is no response or the response is an error code.

Check not only the home page but also what matters to the business: the online store checkout, login, form submission. The home page may be served from cache even when the database is down.

Disk space

A full disk is one of the most common reasons a server suddenly stops working. The database can't write, logs don't fit, a backup is left half-finished. A disk doesn't usually fill up overnight but gradually, through logs, backups and uploaded files.

A good threshold is a warning at 80% and an alert at 90%. Even better is to watch the growth rate: if at the current pace the disk will be full in two weeks, that is a warning today.

Memory and CPU

A short CPU spike is normal. It's worth worrying when the load stays high for a longer time, or when memory usage keeps growing and never drops. The latter often points to a memory leak in the application, which eventually leads to the system starting to kill processes.

Services

Web server (Nginx or Apache), database (PostgreSQL, MySQL, MariaDB), application server, Docker containers. Are all the required services running, and did they come back up on their own after a restart?

Certificates and domain

An expired SSL certificate makes the site unusable in the browser, even if the server is working. Automatic renewal can silently break. Watch the expiry date and alert at least two weeks before. The same applies to the domain.

Backups

Monitoring has to tell you not only that a backup was made, but also that it is a reasonable size and arrived at its destination. A backup that has been an empty file for the last three months is worse than no backup at all, because it gives a false sense of security. Why a backup also has to be tested by restoring it is covered in more detail in the post on server backups and restore testing.

Security

The number of failed logins, overlooked security updates, unexpectedly open ports. These usually don't need an immediate reaction at night, but they must reach the daily overview.

Alert or warning?

The most frequent monitoring mistake is too many notifications. If the phone beeps twenty times a day, people stop reading the notifications and miss the one that matters.

So split notifications into three groups:

  • Alert: a service is down or will go down within the next few hours. Needs immediate action.
  • Warning: something is moving in a bad direction (disk at 80%, certificate expires in 14 days). Deal with it during working hours.
  • Overview: trends and statistics. Look at it once a week.

Every alert should make it clear what to do. If an alert never requires any action, it should be deleted or turned into a warning.

Automatic monitoring is not the same as an on-call phone

An important difference worth asking about when buying the service: automatic monitoring runs around the clock, but that doesn't mean someone responds around the clock. Monitoring notices the problem at 3 a.m., but whether someone fixes it at 3 a.m. is a matter of agreement.

My management package includes automatic 24/7 monitoring and alerts. Faults are resolved on working days, and the response time is set out in the contract. If a business needs round-the-clock on-call cover, we agree on that separately. That way both sides know what to expect. What else a management contract covers is described in the post on Linux server maintenance.

Summary

Good monitoring watches availability from the outside, disk space, services, certificates and backups, and raises an alert only when action is needed. Everything else becomes a warning or goes into the overview.

If your server has no monitoring, or its alerts are driving you mad, see Linux server management or describe your server. I'll put monitoring and backups in order and document what is being watched.

Keep reading

Related services

Who looks after your server?

Setup, updates, backups and monitoring — a Linux server with someone behind it you can actually call.