Why tune the Linux TCP stack for servers
Linux TCP tuning matters when server network throughput, latency, or connection scaling becomes a bottleneck. Default kernel parameters are conservative to work across many environments, but data center servers, CDN edges, storage gateways, and database replicas often need different settings to saturate modern 10G or 25G links.
This guide focuses on practical kernel level tweaks, NIC feature configuration, testing with iperf3 and tcpdump, and safe rollback steps for production. Follow these steps to gain predictable throughput while keeping risk low.
Key kernel settings and what they do
Start by understanding the most impactful sysctl keys. Tuning focuses on buffer sizes, window scaling, autotuning ranges, and congestion control. Adjusting these can increase throughput, reduce packet loss under load, and improve latency for bulk transfers.
net.core.rmem_max,net.core.wmem_max: maximum socket buffer sizes for receive and send.net.ipv4.tcp_rmem,net.ipv4.tcp_wmem: min, default, max autotuning values used by TCP stacks.net.ipv4.tcp_congestion_control: selects congestion control algorithm, for example cubic or bbr.net.core.netdev_max_backlog,net.core.somaxconn: control kernel queue lengths for incoming packets and accept queue size.
Apply changes temporarily with sysctl -w for live testing, and persist them in /etc/sysctl.conf or a file in /etc/sysctl.d/. Always document the previous values before changing anything.
Adjusting buffer sizes and autotuning
Modern Linux kernels include autotuning for TCP windows. The usual approach is to raise the maximum allowed values while letting autotuning pick the working size. For high bandwidth delay product links, increase maximums so autotuning can scale upward as needed.
Example live commands to test and set higher ranges: sysctl -w net.core.rmem_max=536870912, sysctl -w net.core.wmem_max=536870912, sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864", sysctl -w net.ipv4.tcp_wmem="4096 65536 67108864". Monitor with ss -tin to see actual window sizes used by active sockets.
Selecting congestion control including BBR
Cubic is the default in many kernels, offering stable growth and broad compatibility. BBR can improve throughput and latency on links where loss based congestion control underestimates available capacity. Enable BBR only after validation on representative traffic patterns.
To enable BBR, ensure kernel supports it then set: sysctl -w net.ipv4.tcp_congestion_control=bbr. Confirm with sysctl net.ipv4.tcp_congestion_control and monitor retransmits and bandwidth during tests. If BBR causes regressions for short lived flows, revert to cubic.
NIC offloads and driver features to check
NIC features such as checksum offload, GRO, GSO, and TSO can reduce CPU usage and increase throughput by delegating segmentation and checksums to hardware. On the other hand, some offloads interfere with packet captures or certain software processing, so validate carefully.
Use ethtool -k to inspect offloads, and ethtool -K or off to change settings. Typical checks include ensuring hardware checksum is on, and validating GRO, GSO, and TSO behavior under load. Record default states so you can restore them.
Testing and benchmarking with iperf3 and tcpdump
Testing should be done with traffic that reflects production patterns. Use iperf3 for throughput baselines and direction specific tests. Example: run server with iperf3 -s and client with iperf3 -c server -P 10 -t 60 to use parallel streams.
Capture packets with tcpdump -i while testing to verify retransmits, window sizes, and packet pacing. Combine observations from ss -s, kernel logs, and perf counters to understand CPU or interrupt bottlenecks.
- iperf3 options to try: parallel streams, reverse mode, and different TCP window sizes.
- tcpdump filters to focus on problem flows, for example by IP or port to reduce noise.
Safe rollout and rollback in production
Never deploy a global kernel change without a rollback plan. Use canary hosts or a staged rollout so you can measure impact on realistic workloads. Apply settings on a test subset and run full traffic replay if possible.
Rollback steps include restoring backed up sysctl values, reverting NIC offload changes with ethtool, and switching congestion control back. Keep a small script that logs current values before applying changes, for example a file with sysctl -a output and ethtool states, to make reversal deterministic.
- Backup commands to run before changes and to restore values quickly.
- Use automation such as Ansible for consistent rollout and quick rollback steps.
Monitoring, verification, and ongoing maintenance
After changes, monitor connection retransmit rates, latency, CPU, and NIC statistics. Use existing observability stacks, or add probes that report socket metrics and interface counters. Watch for increased retransmits or unexpected latency spikes, which indicate misconfiguration.
Keep tuning decisions in version control, and schedule periodic revalidation after kernel, driver, or hardware upgrades. Network conditions and application mixes change over time, so what is optimal today may need adjustment later.
FAQ
What kernel version do I need for BBR? BBR is available in mainline Linux kernels from 4.9 onwards, with improvements in later releases. Check your distribution kernel and backport notes before enabling.
Will increasing buffer sizes hide packet loss? Increasing buffers can mask transient congestion but will not eliminate underlying loss. Watch retransmit counts and latency, and combine buffer changes with congestion control tuning.
Can NIC offloads break packet captures? Yes, certain offloads alter packets after capture or offload segmentation to hardware, which can make packet traces appear different. Disable specific offloads when deep packet inspection is needed.
How do I test settings without affecting production? Use canary servers, traffic replay tools, or a staging network that mirrors production. Apply settings to a limited subset, run iperf3 and real traffic, then compare metrics before wider rollout.
Conclusion
Tuning the Linux TCP stack for high throughput servers is a balance between increasing headroom for autotuning, choosing appropriate congestion control, and leveraging NIC features while managing risk. Start by raising maximum buffer sizes and autotuning ranges, then validate with controlled iperf3 tests and packet captures. Consider BBR for links where loss based algorithms limit throughput, but validate with representative flows because short lived connections can behave differently. Configure NIC offloads to reduce CPU work, and record original states so you can restore them quickly. Use staged rollouts, automated change management, and clear monitoring to detect regressions early.
Documentation and repeatable automation reduce human error and shorten rollback time. Keep tuning changes small and iterative, measure impact with the right metrics, and revalidate after kernel or driver upgrades. With careful testing and a clear rollback plan, Linux TCP tuning can unlock significant throughput on modern server hardware while maintaining stability for production services.