How much RAM does a server need? Memory, swap and the page cache

A week after a new server goes live, someone runs free -h, sees 312 MiB free out of 8 GiB, and starts pricing a bigger plan. The server is fine. Pages load quickly, nothing is swapping, and the same command showed more than 5 GiB free on the day the machine booted. Nothing leaked in between. The kernel spent the week keeping copies of files in memory, which is exactly what it should do with memory that no program wants.

Memory gets misread in both directions. People buy RAM they do not need because one column looked small, and they miss a real shortage until the database vanishes at nine in the evening. Both mistakes go away once you know what the kernel does with memory, and how to measure what your programs really use.

Reading free -h: the column that matters is available

free measures nothing itself. It parses /proc/meminfo, the kernel's running account of memory.³ Here it is on an 8 GiB server a week into production:

$ free -h               total        used        free      shared  buff/cache   availableMem:           7.7Gi       2.6Gi       312Mi        98Mi       4.9Gi       5.1GiSwap:          2.0Gi       148Mi       1.9Gi

Each column comes from a field in /proc/meminfo. With -h, Mi and Gi are binary units, so 1 Gi is 1,024 Mi.³

The kernel works MemAvailable out from free memory, reclaimable kernel caches, its lists of cached file pages and the low watermarks explained further down.¹ Here 5.1 GiB is available although only 312 MiB is free: the difference is cache the kernel can reclaim when a program needs the memory. Used is the same estimate seen from the other side.

Why available replaced "free plus cached"

For years the usual estimate of usable memory was free plus cached. When Rik van Riel added MemAvailable to the kernel in 2014, his commit message said that approach "was fine ten years ago, but is pretty much guaranteed to be wrong today".² Cached includes memory that cannot be freed like ordinary cache, such as shared memory and tmpfs, and it leaves out reclaimable kernel caches.²The free manual lists MemAvailable from kernel 3.14 on.³ The estimate is also deliberately cautious: it assumes the system needs some page cache to keep working, and that not all reclaimable kernel memory can be reclaimed while it is in use.¹

In the illustration at the top, free is the same on the healthy server and on the one running short; only available differs.

Why a healthy server has almost nothing free

Whenever a file is read, the kernel keeps the data in memory, in the page cache, so the next read of the same data does not have to go to the disk. Writes land there too, and reach the disk a little later.⁴A page that has been changed but not yet written is counted as Dirty in /proc/meminfo.¹

A server rereads the same files all day: programs, libraries, templates, the database's files. So the cache grows into whatever memory programs leave, and free falls after boot and stays low.

Two kinds of memory, and only one has a copy on disk

The kernel documentation names the page cache and anonymous memory as the two main kinds of memory it can reclaim,⁴ and it reclaims them differently. A clean cached page is a copy of something already on disk, so the kernel can drop it; the only cost is a disk read if the data is wanted again. Anonymous memory is the heap and stack of your programs, with no file behind it, so to repurpose such a page the kernel has to write it to swap first.⁴ Without swap, anonymous memory cannot be reclaimed at all. That difference decides everything that happens under pressure.

What the kernel does when memory runs short

The kernel keeps thresholds on free memory, called watermarks, and acts in stages as free memory falls through them. Physical memory is divided into zones, ranges the kernel manages separately,⁴ and each zone has three watermarks: min, low and high.⁵ Nobody notices the first stage. The later ones are what people call "the server is slow" and "the database crashed".

When free pages fall to the low watermark, the next request for memory wakes kswapd, a kernel thread that reclaims in the background. It frees pages whose data exists elsewhere, such as clean file cache, and writes the rest out first: dirty file pages to their files, anonymous pages to swap.⁴It keeps going until enough memory is free for it to sleep again, a level above low.⁵

If kswapd cannot keep up and free memory reaches the min watermark, requests for memory trigger direct reclaim: the program that asked is stalled until enough pages have been reclaimed.⁴ This is the stage people feel. A PHP worker that wanted a few more kilobytes does the kernel's housekeeping instead, and the request it was serving waits.

Push further and the kernel evicts pages that are needed again almost at once: read back in, pushed out, read back in. The kernel's pressure accounting calls a workload that spends a long time with all its non-idle tasks stalled on memory thrashing.¹⁴ When reclaim cannot free enough for the system to keep running, the kernel invokes the OOM killer, which ends one process so the rest can continue.⁴

The OOM killer: who it picks, and where it leaves a trace

The OOM killer does not look for the process that caused the shortage. By default it scans every task and picks one by heuristics, normally a memory-hungry task whose death frees a lot.⁵ The kernel rates each candidate with a badness score, roughly the share of its allowed memory (RAM plus swap) the task uses: all of it scores 1000, half scores 500.⁹ The highest score is the likeliest victim, and the score is based on memory use.⁸ The figure in /proc/<pid>/oom_score is that score rescaled: current kernels add 1000 and multiply by two thirds, so a process using almost no memory reads about 666.⁷ Compare processes with each other rather than reading it as a percentage.

On a server running a web application and its database, the biggest process is very often the database. A leaking PHP script can push the machine over the edge, and the kernel may well kill the database, because that is where the memory is. oom_score_adj, from -1000 to +1000, steers the choice, and -1000 exempts a process entirely.⁹ That only moves the kill to another process; the memory is still missing.

The victim is sent SIGKILL,⁶ which a process cannot catch, so it never writes a last line to its own log: from the application's side, the log just stops. The evidence is in the kernel log, which journalctl -k shows on its own, for the current boot unless you ask for another.¹⁰

$ journalctl -k | grep -iE 'invoked oom-killer|out of memory'Sep 24 21:07:12 web1 kernel: php-fpm invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0Sep 24 21:07:12 web1 kernel: Out of memory: Killed process 1184 (mariadbd) total-vm:2395116kB, anon-rss:1398212kB, file-rss:0kB, shmem-rss:0kB, UID:110 pgtables:3284kB oom_score_adj:0

A PHP-FPM worker asked for memory that could not be found, and the kernel killed MariaDB, which held about 1.3 GiB of anonymous memory (1,398,212 kB divided by 1,048,576). For the previous boot, add -b -1, the journalctl manual's own example for kernel logs.¹⁰ A line reading Memory cgroup out of memory means a container's or a service's memory limit was hit, not the whole machine's.⁶

Swap parks idle memory on disk

Swap exists for anonymous memory. Programs allocate memory at start-up that they touch once and never again, and idle daemons hold pages nobody reads for days. Swap lets the kernel park those pages on disk and use the RAM for cache that is actually read. Without swap, the only thing the kernel can shrink under pressure is the file cache, including the parts your programs keep rereading.

Swap cannot stand in for memory you need. Red Hat's documentation says swap helps small machines but does not substitute for RAM,¹¹ and the kernel notes that filesystem I/O under memory pressure tends to be more efficient than swap's random I/O.⁵ A server whose working set lives in swap runs at the speed of the disk.

What vm.swappiness really controls

The name vm.swappiness suggests a percentage, as in "start swapping at 60% full". What it actually sets is the rough relative I/O cost of swapping against dropping and rereading file pages, from 0 to 200. At 100 the kernel treats the two as equally expensive and presses equally on the page cache and on swap-backed pages; lower values make swap look dearer, so the kernel drops cache first. The default is 60.⁵

Nor does 0 turn swap off: the kernel still swaps once free and file-backed pages fall below the high watermark in a zone.⁵ Values above 100 suit swap faster than the filesystem, such as compressed swap in RAM: for swap twice as fast, the documentation works out 133 (x + 2x = 200).⁵ On an ordinary server, leave the default unless measurements say otherwise.

Swap traffic is the warning sign

A few hundred MiB of swap used with nothing moving is healthy: cold pages, parked. The warning is traffic: the si and so columns of vmstat, memory swapped in from and out to disk per second.¹² Numbers in both, interval after interval, mean the server is short of memory now.

There is no universal swap size, and twice the RAM is not a rule. Red Hat's guideline for its own systems runs from twice the RAM below 2 GiB to at least 4 GB from 8 to 64 GB, and the same page ties swap to the workload rather than to the amount of memory.¹¹Treat swap as a buffer for idle pages and a few minutes of warning. If you need a lot of it, you need RAM.

Measuring real memory use over time

One run of free describes one moment, usually a quiet one. Sizing needs the busiest hour of the busiest day, the nightly backup and the report someone runs on the first of the month.

vmstat: memory and swap traffic, every few seconds

vmstat 5 prints a line every five seconds. Ignore the first: it gives averages since boot, and only the lines after it describe each interval.¹²

$ vmstat 5 4procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st 1  0 151552 319488 186312 4951712    2    3   41   88  612  930  7  2 90  1  0 3  0 151552 301220 186320 4960104    0    0    0  212 1840 2710 21  4 74  1  0 2  0 151552 297804 186320 4961880    0    0    0  176 1702 2544 18  3 78  1  0 4  1 151552 288112 186328 4963300    0    0    4  240 1911 2803 24  5 70  1  0

swpd shows the same 148 MiB of swap in use that free reported, and nothing is moving: si and so are zero after the first line. Healthy.

sar: the history you will want later

sar, from the sysstat package, reads the activity files sysstat saves through the day, so it can answer questions about last night: by default today's file, and with -1 yesterday's. It can only show what was saved,¹³ so turn sysstat's collection on the day the server goes live.

$ sar -r -1     # memory, from yesterday's file$ sar -W        # pages swapped in and out per second$ sar -B        # paging, including major faults per second

sar -r reports kbavail, the same estimate as MemAvailable, and kbmemused, which is total memory minus kbavail.¹³ The week's lowest kbavail is the figure to plan around. sar -W gives swap pages in and out per second, and sar -B gives major faults, the ones that had to load a page from disk.¹³ When cached files or swapped pages are read back constantly, majflt/s climbs.

sar -r also prints %commit, an estimate of the memory needed to guarantee the machine never runs out, which can pass 100% because the kernel usually overcommits.¹³ It counts memory promised to programs whether or not they touch it (a process that allocates 1 GB and touches 300 MB counts as 1 GB),¹ so a figure above 100% is normal and says nothing about sizing.

Pressure stall information: is memory costing you time?

Available says how close the edge is. Pressure stall information (PSI) says whether you are already paying for it. In /proc/pressure/memory, some measures how much of the time at least one task was waiting on memory, and full how much of the time every task with work to do was waiting at once. Each is a percentage averaged over the last 10, 60 and 300 seconds.¹⁴

$ cat /proc/pressure/memorysome avg10=0.00 avg60=0.12 avg300=0.08 total=1839211full avg10=0.00 avg60=0.03 avg300=0.02 total=611042

Figures like these are background noise. A some figure that climbs every evening means requests are waiting on reclaim. A full figure that stays in whole percentage points for minutes at a time means every task is stalling together, which the kernel documentation calls thrashing.¹⁴

Per process: PSS is the figure that adds up

ps reports RSS, the non-swapped physical memory a process uses, in KiB.¹⁵ It counts every page the process maps, including pages shared with other processes, such as code and libraries, so adding up RSS across 24 PHP-FPM workers counts each shared page 24 times.

PSS, the proportional set size, divides each shared page among the processes sharing it. In the kernel documentation's example, a process with 1,000 pages of its own and 1,000 shared with one other process has a PSS of 1,500.¹ PSS figures therefore add up to what is really in memory.

Each process's /proc/<pid>/smaps_rollup totals its memory across all of its mappings, and splits PSS into anonymous, file and shared-memory parts (Pss_Anon, Pss_File, Pss_Shmem).¹The file part is page cache: program code, libraries and mapped files, which the sizing below budgets as cache. Leave it out here, or it is counted twice. Reading another user's processes usually needs root. This sums the anonymous and shared-memory PSS across PHP-FPM at the peak, when the workers are busiest:

# for p in $(pgrep php-fpm); do awk '/^Pss_(Anon|Shmem):/ {print $2}' /proc/$p/smaps_rollup; done | awk '{s+=$1} END {printf "%.0f MiB of Pss_Anon + Pss_Shmem\n", s/1024}'1571 MiB of Pss_Anon + Pss_Shmem

That covers the pool's master and its 24 workers.

Reading the signals together

Available is healthy when its lowest point at the peak still covers the cache the workload rereads plus a margin; the worked example below puts numbers on both.

Where the memory usually goes

On most servers one or two programs decide the memory, and each has a setting that works as a multiplier or a ceiling. Find it and sizing becomes arithmetic.

PHP-FPM serves each request in a worker process. pm.max_children is the worker count in a static pool and the maximum in a dynamic or on-demand one, and it caps how many requests are served at once.¹⁶ Peak pool memory is roughly that number times a busy worker's PSS, so it is the multiplier you control: fewer workers use less memory and queue requests sooner. The web server stack explained shows where the pool sits.

A Java application has its heap capped by -Xmx. Without it the JVM picks a cap at runtime: MaxRAMPercentage, 25 percent by default, of the memory available to the JVM or of 128 GB, whichever is lower.¹⁷ On a 16 GiB server that is a heap of about 4 GiB (16 × 0.25). The heap is not the whole process either: class metadata lives outside it, unlimited by default, and every thread has its own stack, 1,024 KB by default on Linux x64.¹⁷ So budget from the process's measured PSS at the peak.

A database keeps its own cache of table and index pages, sized in its configuration, and is usually the largest process on the machine. Sizing that cache is covered in sizing a database server; for the memory budget, measure its PSS at the peak like any other process.

The kernel needs a share too: unreclaimable kernel memory (SUnreclaim), kernel stacks (KernelStack) and page tables (PageTables) are all listed in /proc/meminfo.¹ Game servers vary too much by game, map and player count for a rule of thumb; the server calculator sizes them per game.

A worked example: sizing from measurements

A shop runs nginx, PHP-FPM and MariaDB on a 4 GiB server. At the evening peak pages crawl, and twice in one week the database has vanished, with the OOM kill shown earlier in the kernel log. The owner enables sysstat's collection and, for a week, records the anonymous and shared-memory PSS of every process at the peak. The figures below are that server's, an example: yours will differ, which is why you measure.

The first four rows are anonymous, shared or kernel memory, none of which the kernel can reclaim without swap: 1,536 + 1,400 + 350 + 250 = 3,536 MiB. With MemTotal a little below 4,096 MiB, that leaves a few hundred MiB for the files the site rereads, so they are evicted at every peak, and one spike leaves reclaim nothing to take. The cache row is the roughest number, a guess taken when nothing was short. The arithmetic:

• Measured at the peak: 3,536 + 1,200 = 4,736 MiB.
• Margin: 25 percent, a judgement meant to cover a deploy, a backup that runs into the peak and a busier week than the one measured. 4,736 × 1.25 = 5,920 MiB, which is 5,920 / 1,024 = 5.8 GiB.
• On 4 GiB, 4,096 MiB: 5,920 − 4,096 leaves the server 1,824 MiB short, the week's trouble in one number.
• On 8 GiB, 8,192 MiB: 8,192 − 5,920 = 2,272 MiB spare.

The spare 2,272 MiB will not sit idle: the kernel fills it with page cache, and more of the site is served without touching the disk. A 6 GiB server (6,144 MiB) clears 5,920 by 224 MiB on paper. The margin is already inside the 5,920, so the real test is MemTotal. If it sits about 0.3 GiB below the nominal size, as on the 8 GiB server earlier (7.7 of 8), 6 GiB gives about 5.7 GiB, roughly 5,840 MiB, which is 80 MiB under. So this shop goes to 8 GiB.

The other lever is the multiplier. With pm.max_children at 16, PHP-FPM needs 16 × 64 = 1,024 MiB, and the total becomes (1,024 + 1,400 + 350 + 250 + 1,200) × 1.25 = 5,280 MiB: still more than 4 GiB, and requests would now queue at the peak. Cutting workers trades throughput for memory, and here even the cut leaves the shop short. The database died each time because it was the largest process; exempting it would only have moved the kill.

Before there is a server to measure

For a first server, measure a copy. Run the application on a small instance or a local machine, drive it with a load test against that copy, and read Pss_Anon and Pss_Shmem for a busy worker and for the database as above. Multiply the worker figure by the worker count you plan to run, add the rest and apply the same formula, then measure again once real traffic peaks. The server calculator gives a starting point to check against.

After the change: check that the numbers hold

A resize is a hypothesis until the next peak confirms it. The same week of measurements should now show the lowest available at the busiest hour at or above the hot cache you budgeted plus your margin (1,200 + 1,184 = 2,384 MiB in the worked example), si and so at zero, majflt/s low and memory pressure near zero. If the lowest available at the peak exceeds the hot cache plus margin by more than the difference to the next size down, test that size. If it falls a little each week, the working set is growing, and you now know how fast, which is the number the next decision needs.

On a VPS, all of this happens inside your virtual machine. Whether the host has promised its guests more memory than it has is a separate question, covered in what is a VPS.