NVMe vs SATA SSD vs hard disk for servers

The number printed on a drive's box is how fast it reads one long file from start to finish. Very little of what a server does looks like that. A database answering a query, a web server opening its scripts and templates, a mail server, a game server saving player data: each asks for a few kilobytes here and a few kilobytes there, from all over the disk, and often has to wait for one answer before it knows what to ask for next.

Measured that way, the gaps between storage types change shape. A hard disk that streams more than 100 MB/s manages well under 1 MB/s of small random reads, and the difference between a SATA SSD and an NVMe SSD is less about megabytes per second than about how many requests the drive can be handed at once, and how long they wait in line. To see why, follow one small read through each kind of drive.

Three drives, two separate choices

"NVMe vs SSD" is a common search and a false choice. The three drives differ in two independent ways, and most of the confusion comes from mixing them up.

The first is the medium, the thing that holds your data. A hard disk stores it magnetically on spinning platters and reads it with a head on a moving arm. An SSD, whether SATA or NVMe, stores it in flash memory chips with no moving parts.

The second is the interface: the link to the computer and the command protocol spoken over it. Desktop and capacity hard disks use SATA, whose 3.0 revision runs at six gigabits per second.⁹ Operating systems usually drive a SATA disk through AHCI, the standard set of registers they use to talk to a SATA controller, and SATA's command queue, called Native Command Queuing, is the single queue of 32 commands described below. Many server hard disks use SAS, Serial Attached SCSI, instead; the textbook's 15,000 RPM example drive further down connects via SCSI.⁶NVMe is a newer host controller interface, designed for solid-state storage attached over PCI Express, the kind of lanes a graphics card uses.² Its authors give a PCIe Gen3 link four lanes wide as delivering 4 GB/s.² Six gigabits per second is 0.75 gigabytes per second before encoding overhead, so even that older PCIe link carries more than five times as much.

A SATA SSD is flash behind a hard disk's SATA interface, built to drop into machines made for hard disks, and it inherits that interface's limits. An NVMe SSD is the same kind of flash with an interface designed for it. So the useful comparisons are hard disk against flash (a question of physics) and SATA against NVMe (a question of how requests are queued and carried).

What a hard disk does for every random read

Inside a hard disk, platters spin at a constant speed while a head on an arm floats just above the surface over one circular track. A read has three parts, and its time is their sum: seek, rotation and transfer.⁶

The seek moves the arm to the right track in four phases: the arm accelerates, coasts, decelerates, and then settles while the head locks onto the track, and settling alone often takes 0.5 to 2 ms.⁶ The rotation is the wait for the wanted sector to come round under the head. The textbook gives 7,200 to 15,000 revolutions per minute as the typical range, and on average a read waits half a turn.⁶ The transfer is reading the data as it passes, the only part that looks like the speed on the box.

The arithmetic of one read

The textbook Operating Systems: Three Easy Pieces works this through for two real drives. The first, a SCSI drive, spins at 15,000 RPM, 250 turns a second, so one turn takes 4 ms and the average half-turn wait is 2 ms. Its average seek is 4 ms, and moving 4 KB at its 125 MB/s peak takes about 30 microseconds. A random 4 KB read therefore costs about 6 ms, of which the transfer is half of one per cent.⁶

Six milliseconds per read means one second holds about 1 / 0.006, or 166, of them. Multiply by 4 KB and the drive delivers about 0.66 MB/s on random reads, against 125 MB/s when it streams one long file: a gap of almost 200 times. The second drive, a 7,200 RPM SATA model with a 9 ms average seek, does worse. Half a turn at 7,200 RPM is 60,000 ms / 7,200 / 2, or about 4.2 ms, so a random read takes about 13.2 ms, roughly 76 reads a second and 0.31 MB/s, more than 300 times below its 105 MB/s sequential rate.⁶ Database systems are the textbook's example of software that reads this way.⁶

One random 4 KB read, 15,000 RPM disk

6ms

What that allows, one at a time

166reads/s

Random 4 KB reads, against 125 MB/s streaming

0.66MB/s

Look at where the time goes and you can see why no faster cable would help. Transfer is the sliver; seek and rotation are mechanics, and the only ways to shorten them are to spin faster, move the arm less, or pick the order of the reads cleverly. Linux's original block layer did the last of these, sorting requests by where the head happened to be, because the moving parts were the bottleneck.³ SATA's Native Command Queuing lets the drive itself reorder the commands it holds: in SATA-IO's own illustration, four commands that need two and three quarter turns in arrival order need only one and a quarter when the drive picks the order.⁸ That is a real gain, but reordering only shortens the trips between reads and never removes them.

Flash: no arm, but rules of its own

Flash-based SSDs have no mechanical parts. They are random-access devices, more like memory than like a disk, and where a hard disk manages a few hundred random operations a second at most, an SSD does far more.⁷ Reading a page of flash takes tens of microseconds wherever the page is.⁷ There is no seek and no rotation.

Writing is where flash has rules. Data lives in pages of a few kilobytes, grouped into erase blocks (128 KB or 256 KB in the textbook's account), and a page cannot simply be overwritten: its whole block has to be erased first. Programming a page takes hundreds of microseconds and erasing a block a few milliseconds, slow by flash standards.⁷

So every SSD carries a controller running a flash translation layer. It maps the logical blocks the operating system asks for onto physical pages, and spreads writes evenly so no block wears out early, which is called wear levelling.⁷ Most translation layers today are log structured: a rewritten block goes to the next free page, the old copy becomes garbage, and the drive later copies live pages out of mostly dead blocks and erases them. The textbook calls the cost of that cleaning write amplification: bytes written to flash divided by bytes you asked the drive to write.⁷

Every modern SSD is also built from many flash chips that it uses in parallel.⁷ One read keeps one chip busy for tens of microseconds; thirty reads that land on different chips can mostly proceed at once. A hard disk has one arm, a flash drive has many independent places to work, and that is why queueing matters so much more for flash than it ever did for disks.

A flash read sits about two orders of magnitude below a hard disk's random read, so once storage became flash, the time spent carrying a request to the drive and the answer back started to matter. In the words of the kernel documentation, "the bottleneck of the stack had moved from the storage device to the operating system."³ NVM Express wanted a protocol that allows end-to-end latency under 10 microseconds, software included, for memory that may read in under a microsecond.²

IOPS, throughput and latency: two lines of arithmetic

Spec sheets quote three kinds of number, and they are linked. Throughput is how many bytes move per second. IOPS is how many separate operations complete per second. Latency is how long one operation takes, from request to answer.

The first link is multiplication: throughput is IOPS times block size. A drive completing 10,000 reads a second of 4 KiB each moves 40,960 KiB/s, which is 40 MiB/s. The same 40 MiB/s made of 1 MiB blocks is only 40 operations a second. A throughput figure measured with large blocks tells you nothing about how many small reads a drive can do, and small reads are what a database or a directory full of small files asks for.

The second link explains queues. If N requests are always outstanding and each takes L seconds, then N of them complete every L seconds, so the device finishes about N / L a second. With one request at a time, IOPS is one over the latency: the 6 ms disk above gives 166. Take a flash read path of 100 µs, a round number chosen to show the arithmetic rather than a figure for any drive: one at a time, that is 1 / 0.0001, or 10,000 IOPS. Keep 32 in flight and, if the drive can serve them in parallel without each one slowing down, the same formula allows 320,000. In practice latency creeps up as the queue fills and the drive runs short of free chips, and that is where a real drive's ceiling comes from.

So queue depth, the number of requests outstanding at once, is the hidden variable in every IOPS figure. A program that issues one read and waits for it before deciding what to read next runs at a queue depth of one, however many queues the drive supports. A database following an index from page to page, or a script opening files one by one, lives there, and latency alone sets its speed. A busy database serving many connections, or a host running many virtual machines, keeps many requests in flight, and there the drive's parallelism counts.

A worked example: 200 small reads

Suppose one page of a web shop needs 200 random 4 KB reads that are not already in memory, issued one after another. The number is invented to show the arithmetic. That is 800 KB of data, which the 7,200 RPM disk could stream in 800 KB / 105 MB/s, about 7.6 ms. Read randomly, each costs about 13.2 ms, so the page waits 200 x 13.2 ms, or 2.64 seconds, for its disk. On the illustrative 100 µs flash path the same 800 KB takes 200 x 0.1 ms, or 20 ms: the data is identical, and only the waiting changed.

The textbook gathered published results for three SATA SSDs and a 15,000 RPM hard disk from around 2013: on sequential reads the four sit within a factor of two of each other, while on random reads the SSDs deliver 39 to 103 MB/s and the hard disk about 2 MB/s, rounded up.⁷

Flash removes the seek, but random access is still slower than sequential, enough that the textbook says file systems should still be designed to reduce random I/O.⁷ And the hard disk's sequential bar is why it has not disappeared: the same authors still recommend a hard disk when sequential speed is all you need.⁷

Queues: one line of 32, or a line for every core

A queue, here, is the list of commands the computer has handed to the drive and the drive has not finished yet. How many it can hold, and how many separate lists there are, is the real difference between SATA and NVMe.

NVM Express, the body that maintains the NVMe standard, sets out the older limits in its overview: 32 commands in one queue for a SATA device, and typically 256 for a SAS device. Enough for hard disks, it says, but not enough to get the full benefit of flash.² For a hard disk, 32 waiting commands are plenty: they give Native Command Queuing a choice of what to read next, and the arm can only be in one place anyway.

NVMe raises both limits by orders of magnitude: up to 65,535 I/O queues, each with up to 65,535 outstanding commands.¹ A queue is defined with up to 65,536 slots, one of which is always unusable because of how the head and tail pointers work, and a real controller reports its own maximum, which may be smaller, in a register field called CAP.MQES.¹ These are ceilings in the protocol, not features of any drive, and what they buy is separation more than depth. The specification's own example is a four-core machine with a queue pair per core, so that the cores never lock against each other and each queue's data stays in its own core's cache.¹

How a command reaches an NVMe drive

The mechanics are lean. The host writes a 64-byte command into a submission queue in its own memory, then writes the queue's new tail position to a register on the drive called a doorbell. The command counts as submitted once that write moves the tail past it, and as complete once the drive has posted an entry to the matching completion queue.¹ The specification's design points include no uncacheable register reads on that path, at most one register write to submit a command, and a 64-byte command that carries everything needed for a 4 KiB read.¹

How much does that shorter path save on one small read? Part of the path is the wire. A 4 KiB read carries 4,096 x 8 = 32,768 bits, which take 32,768 / 6,000,000,000 s, about 5.5 µs, on a 6 Gb/s SATA link before encoding overhead, and 4,096 / 4,000,000,000 s, about 1 µs, at the 4 GB/s of the overview's PCIe example. Both are a fraction of the 25 to 75 µs a flash chip needs to read the page in the first place.⁷ The rest is software and command handling, where NVM Express says its command set needs less than half the CPU instructions per request that SAS or SATA does.² So at a queue depth of one, the interface is a modest share of each read next to the flash itself, and NVMe's larger gain comes when many requests are in flight at once, which is what its queues are for.

The Linux side: one queue per core, not 65,535

The operating system had to change too. Linux's old block layer kept one queue of requests behind one lock, which did not scale on machines with many processors.³ Its replacement, blk-mq, keeps software staging queues per CPU or per memory node and maps them onto hardware dispatch queues that feed the device's own submission queues. There are never more hardware queues than cores, and every request carries a small integer tag that comes back on completion, so finding the finished request needs no search.³

So Linux gives a drive at most one hardware queue per core, whatever the protocol allows, and a SATA drive gets one because one is all it offers. What matters is that there is no single line: a busy database, or a host running dozens of virtual machines, can keep a flash drive's chips busy without every core waiting its turn at one 32-slot queue. On a VPS those streams belong to many customers at once, and what is a VPS follows a guest's disk request down to the host's drive.

A program that reads one thing at a time gains nothing from the extra queues. Against a SATA SSD it gains only the few microseconds of shorter path worked out above; against a hard disk it loses the seek and the rotation, milliseconds on every read, and for many small servers that is the difference that matters.

Which workloads feel the difference

The question is always the same: does the program wait on many small reads or writes that are not already in memory, and does it issue them one at a time or many at once?

For a database the first question is whether its working set, the data it reads often, fits in memory; if it does, most reads never reach the drive. That calculation has its own guide, sizing a database server. Once the working set outgrows memory, every miss is one of the small random reads this page is about.

Backups and archives sit at the other end: long sequential runs that need space more than speed. Asked why SSDs have not replaced hard disks, the textbook answers with cost per gigabyte: at its 2015 prices an SSD cost about 60 cents a gigabyte against about 5 cents for a hard disk, more than ten times as much.⁷ Making backups that actually restore is covered in server backups.

Measuring your own disk with fio

First check what you are measuring. The kernel reports whether a disk is rotational in /sys/block/<disk>/queue/rotational and lists its I/O schedulers in queue/scheduler, with the active one in square brackets.⁴ lsblk shows the same flag as ROTA. Inside a virtual machine these describe the virtual disk, which need not match the drive underneath: on a KVM guest, lsblk usually shows a virtio disk such as vda rather than nvme0n1.

$ lsblk -d -o NAME,ROTA,TRAN,SIZENAME    ROTA TRAN   SIZEsda        1 sata   3.6Tnvme0n1    0 nvme 476.9G$ cat /sys/block/nvme0n1/queue/scheduler[none] mq-deadline kyber bfq

Then measure what servers actually do: small random reads. The command below uses fio (install it with your package manager, for example apt install fio) to read 4 KiB blocks at random places in a 4 GB test file for 30 seconds, keeping exactly one request in flight. fio lays the file out once and then only reads it, so it is safe on a server in use if there is room for the file, though it will compete with real work while it runs.

$ fio --name=qd1 --filename=/root/fio.test --size=4G --rw=randread --bs=4k --direct=1 --ioengine=libaio --iodepth=1 --runtime=30 --time_basedqd1: (groupid=0, jobs=1): err= 0: pid=2211  read: IOPS=9850, BW=38.5MiB/s (40.3MB/s)(1154MiB/30001msec)    slat (nsec): min=1020, max=38410, avg=2870.12, stdev=512.40    clat (usec): min=61, max=1874, avg=98.40, stdev=16.95     lat (usec): min=63, max=1877, avg=101.33, stdev=17.02    clat percentiles (usec):     |  1.00th=[   78],  5.00th=[   82], 10.00th=[   85], 20.00th=[   89],     | 50.00th=[   97], 90.00th=[  113], 99.00th=[  141], 99.90th=[  247],     | 99.99th=[  619]  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

The options matter more than the tool. --rw=randread asks for random reads.⁵ --direct=1 uses non-buffered I/O, usually O_DIRECT, so the page cache cannot answer from memory and flatter the drive.⁵ --ioengine=libaio is Linux's native asynchronous I/O, which may only queue several requests at once when I/O is non-buffered, one more reason for --direct=1.⁵ --iodepth is how many requests fio keeps in flight, and --time_based makes it run for the full --runtime, looping if it finishes early.⁵

Reading the output

These numbers are an example of the layout, not a measurement of any drive. IOPS is the average number of operations per second, and BW is bandwidth in powers of two with the powers of ten in brackets, MiB/s and then MB/s. slat is submission latency, clat is completion latency, the time from submission to completion, and lat is the two added together.⁵ The IO depths line shows how many requests were actually in flight over the run.⁵

Check that the output agrees with itself. 9,850 operations of 4 KiB is 39,400 KiB/s, which is the 38.5 MiB/s reported. One request at a time taking 101.33 µs allows 1 / 0.00010133, about 9,870 a second: the N / L rule with N = 1. Now run the same command with --iodepth=32:

$ fio --name=qd32 --filename=/root/fio.test --size=4G --rw=randread --bs=4k --direct=1 --ioengine=libaio --iodepth=32 --runtime=30 --time_basedqd32: (groupid=0, jobs=1): err= 0: pid=2240  read: IOPS=171k, BW=668MiB/s (700MB/s)(19.6GiB/30001msec)    clat (usec): min=72, max=4210, avg=186.12, stdev=41.30    clat percentiles (usec):     | 50.00th=[  180], 90.00th=[  229], 99.00th=[  318], 99.90th=[  611],     | 99.99th=[ 1287]  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.5%, >=64=0.0%

With 32 requests in flight, IOPS rose about seventeen times while each request took almost twice as long: 32 / 186 µs is about 172,000 a second, which matches. A hard disk would show a few hundred IOPS at most in both tests, depth buying only what reordering can save. The first test shows what a one-request-at-a-time program will see; the second, how much parallelism the drive has in reserve.

Finally, read the percentiles, not only the average. fio's own documentation shows a sample run whose median completion latency is 416 µs while the 90th percentile is 12,911 µs, more than thirty times longer.⁵An average hides a tail like that, and the tail is what a slow page or a timed-out query is made of.

Endurance: what DWPD and TBW mean

Flash wears out as it is used. Each erase and program leaves a little extra charge in a block, until its cells can no longer be told apart as 0 or 1. Manufacturers have rated MLC flash blocks at 10,000 program and erase cycles and SLC at 100,000, though research has found real lifetimes to be longer.⁷ Cleaning adds writes you never asked for, and some drives keep extra flash in reserve, called over-provisioning, so that cleaning can wait and run in the background.⁷

Datasheets turn all of that into one rating. KIOXIA's CD8P-R data centre drive is rated for 1 DWPD, drive writes per day, with a five-year warranty, in sizes from 1,920 GB to 30,720 GB.¹⁰ Its sibling, the CD8P-V, uses the same TLC flash and PCIe 5.0 interface but is rated for 3 DWPD, in smaller sizes: 3,200 GB where the CD8P-R offers 3,840 GB, for instance.¹¹ Less space sold from the same flash is what the over-provisioning idea above would predict, more held back for cleaning, though KIOXIA's pages do not say how the two lines differ inside.

One drive write per day means writing the full capacity once a day. Over five years, 1 DWPD on the 3,840 GB drive comes to 3,840 GB x 365 x 5 = 7,008,000 GB, about 7,008 TB: the same promise stated as a total, which is what a TBW (terabytes written) figure is. The 3,200 GB drive at 3 DWPD comes to 3,200 x 3 x 365 x 5, or 17,520 TB. An illustrative server writing 50 GB a day uses 50 x 365 x 5 = 91,250 GB in five years, about 91 TB, or 1.3 per cent of the smaller rating. Endurance deserves attention on write-heavy databases, logging pipelines and caches that churn all day. On most web servers it is a footnote.

You do not have to guess, because NVMe drives keep score. The health log's Percentage Used field is the manufacturer's estimate of how much rated life has gone; 100 means the rated endurance is consumed, which does not necessarily mean failure, and the value may go past 100. Data Units Written counts what the host has written in thousands of 512-byte units, so multiply it by 512,000 to get bytes.¹ The nvme command below comes from the nvme-cli package and needs a physical NVMe drive to read. Inside a virtual machine the disk is a virtual device with no health log of its own; the host's drive is the provider's to monitor.

$ sudo nvme smart-log /dev/nvme0Smart Log for NVME device:nvme0 namespace-id:ffffffffcritical_warning                        : 0available_spare                         : 100%percentage_used                         : 3%data_units_read                         : 41,730,112 (21.37 TB)data_units_written                      : 58,112,405 (29.75 TB)power_on_hours                          : 9,412

Here 58,112,405 x 512,000 bytes is 29.75 TB, and 9,412 powered-on hours is about 392 days, so roughly 76 GB a day. Run the command a month apart and the difference is your real write rate, to set against the drive's rating instead of against a worry.

Mistakes people make when comparing drives

• Comparing sequential MB/s for a random workload. For a database or a tree of small files, the useful figures are random 4 KiB IOPS at a stated queue depth, and latency.
• Benchmarking the page cache. Copy a file twice, or run dd without direct I/O, and the second pass comes from memory.
• Quoting IOPS without a queue depth. A figure taken with 32 requests in flight says little about a program that reads one thing at a time.
• Reading only the average. The 99th and 99.9th percentiles are what slow requests are made of.
• Treating protocol limits as drive specs. Up to 65,535 queues of up to 65,535 commands is what NVMe allows, not what a drive or Linux uses.
• Writing off hard disks. For backups, archives and large files read end to end, their sequential speed is adequate, and cost per gigabyte is the reason the textbook gives for their survival.

Run both fio tests on a quiet server, keep the output, and rerun them when something feels slow.