Linux Disk I/O Optimization and Tuning with fio

Table of Contents

Overview: linux disk i/o optimization for production servers

Disk I/O is a common bottleneck on Linux servers, affecting database performance, virtual machines, and file services. This article focuses on practical diagnostics and tuning, using fio to generate reproducible workloads, and tools such as iostat and blktrace to find where latency and throughput are lost.

We cover step by step commands, recommended kernel and scheduler tweaks, filesystem mounting options, NVMe and RAID guidance, and a compact troubleshooting checklist you can run during incidents. The goal is measurable improvement in IOPS and latency, and predictable behavior under load.

Prepare the environment and install fio

Start on a maintenance window or isolated test host, not a production node. Install fio and tracing utilities on most distributions with the package manager. Example commands include the following:

Debian, Ubuntu: apt update, apt install fio sysstat blktrace
RHEL, CentOS, AlmaLinux: yum install fio sysstat blktrace or dnf install fio sysstat blktrace

Verify your kernel supports necessary tracing features, and confirm device names with lsblk or nvme list. If testing NVMe, prefer direct device paths such as /dev/nvme0n1 rather than mounted files, to avoid filesystem layering during raw device benchmarking.

Baseline diagnostics with iostat, vmstat and blktrace

Begin with lightweight counters to establish a baseline. Run iostat -xz 1 10 to capture utilization, await, and throughput, and vmstat 1 10 to see run queue and I/O wait. These outputs reveal whether the CPU or I/O subsystem is the bottleneck.

For deeper insight, use blktrace to capture per-request timing, then process with blkparse or blkparse and btt for block-level latency histograms. A typical command pair looks like: blktrace -d /dev/sdb -o – | blkparse -i – and later analyze with btt to get tail latency breakdowns.

Using fio for realistic workload tests

Create small job files that model your application, for example random read heavy or mixed read write with specified block size and queue depth. A basic random read test might be the following job file:

[randread]
ioengine=libaio
rw=randread
bs=4k
iodepth=32
numjobs=4
size=4G
direct=1
filename=/dev/sdb
runtime=60
group_reporting

Run fio against a spare device or a prepared file system, and vary bs, iodepth, and numjobs to map how throughput and latency respond. Keep tests reproducible, and avoid running competing benchmarks at the same time when establishing baseline numbers.

Interpreting fio results and key metrics

Focus on IOPS, bandwidth, average latency, and percentile latencies such as 95th and 99th. fio prints lat and clat fields: latency in submission, latency in completion, and IO completion time. Percentile numbers reveal tail latency that averaged metrics hide.

Watch for high CPU usage during fio runs, which indicates that the bottleneck may be CPU or driver overhead. Also examine iostat during the run to confirm whether util is saturated or average wait time is rising. Use blktrace output to correlate high latency with specific request patterns.

Kernel parameters and I/O scheduler tuning

Adjust kernel parameters when workloads show queueing or poor parallelism. Useful sysctl settings include vm.dirty_ratio and vm.dirty_background_ratio for write-heavy workloads, and elevator choice for older kernels. On modern kernels, mq-deadline, kyber, or none may perform differently depending on the storage.

Check current scheduler: cat /sys/block/sdb/queue/scheduler
Change scheduler: echo none > /sys/block/sdb/queue/scheduler
Tune read-ahead and rq_affinity: echo 128 > /sys/block/sdb/queue/read_ahead_kb

Make one change at a time and re-run fio to measure impact. Keep configuration changes in automation or config management to ensure repeatability across hosts.

Filesystem, RAID and NVMe recommendations

For raw performance, use XFS or ext4 tuned with noatime and appropriate inode settings. When using hardware RAID, ensure write cache is enabled and battery backup is functioning. For NVMe SSDs, use the native drivers and verify queue concurrency via /sys/class/nvme.

Mount options example: use noatime, nodiratime, and barrier settings depending on your storage. For databases consider placing data files on separate devices or partitions to reduce metadata contention. On RAID, stripe size should match typical I/O size for better throughput.

Queue depth, multi-queue, and NVMe tuning

Queue depth affects latency and throughput in a nonlinear way. Increase iodepth in fio to determine the saturation point, and then tune device queue limits under /sys/block//queue/nr_requests. For NVMe, adjust io_uring or block multiqueue settings as supported by your kernel.

Multi-queue devices scale better with multiple CPU cores, so confirm IRQ affinity and blk-mq settings. Align CPU interrupts with I/O processing using irqbalance or manual affinity to reduce lock contention and achieve better scaling under parallel workloads.

Troubleshooting checklist, automation and FAQs

When you see degraded performance, follow a compact checklist: collect iostat and vmstat, run a short fio test, capture blktrace if needed, check scheduler and queue settings, and validate filesystem mount options. Automate baseline collection so you can compare before and after tuning quickly.

Common questions and answers:

Q: Can I run fio on a mounted filesystem? A: Yes, but raw device tests avoid filesystem caching and metadata effects, giving clearer device-level results.
Q: How long should fio runs be? A: Use 30 to 120 seconds for quick checks, longer runs for steady state and thermal effects, depending on production tolerance.
Q: Will changing the scheduler break my services? A: It can, so test in staging and measure before rolling to production, especially for mixed workloads.
Q: When should I use blktrace? A: Use blktrace when average metrics are fine but latency spikes or tail latencies impact applications, to identify specific slow requests.

Conclusion: adopting a measurable optimization workflow

Linux disk I/O optimization is an iterative process: measure first, change one variable at a time, and verify impact with repeatable fio workloads. Use lightweight tools such as iostat and vmstat for quick checks, and blktrace for deep dives into request timing. Keep configuration changes tracked in automation and rollback plans ready when experimenting in production environments.

The most reliable improvements come from matching storage settings to workload characteristics, for example adjusting queue depth for NVMe, tuning read ahead for sequential workloads, and selecting a scheduler appropriate for your device and kernel. Prioritize percentile latencies as they affect user experience, not just averages. By building a standard test harness and a short troubleshooting checklist, you will be able to detect regressions early and tune servers confidently, delivering predictable I/O performance for databases, VMs, and file services.