What Is Congestion Control in High-Performance Data Centre Networks?

When Too Much Network Traffic Becomes a Problem

Picture a data centre with hundreds of servers. Most of the day, everything runs smoothly. Applications respond quickly, backups finish on time, and storage systems keep pace with demand.

Then several things happen at once. A database replicates its data, a storage system rebuilds a volume, and an AI training job starts exchanging results between dozens of GPUs. Traffic from many servers heads for the same destination, and some network links suddenly carry more data than they can handle.

What Is Congestion Control in High-Performance Data Centre Networks?
Congestion control keeps data centre traffic moving when a shared network link gets overloaded.

Packets begin waiting in queues inside the switches. Some applications slow down. Others stall while they wait for data that is stuck in line. Response times become unpredictable, and the cause is not always obvious from the application side.

Your first instinct might be to buy faster links. That helps in many cases, but speed alone rarely solves the problem. A faster network can still develop hotspots, and bursts of traffic can still overwhelm a single port. Networks also need smart ways to detect congestion and manage traffic before it harms performance.

That is where congestion control comes in. So, what is congestion control in high-performance data centre networks, and how does it help maintain reliable communication when network traffic increases? This guide answers that question in plain language, explains techniques such as ECN, DCTCP, PFC and RoCEv2 congestion management, and offers practical advice for network teams.

What Is Network Congestion in a Data Centre?

Network congestion happens when more traffic arrives at a link than the link can carry. The extra packets cannot leave immediately, so switches hold them in memory queues until capacity frees up.

Take a simple example. Twenty servers send data through the same uplink to another part of the data centre. If together they send more than the uplink can carry, the switch queues the extra packets. The queue grows, and each packet waits longer. That wait is a major source of added data centre network latency.

If the queue fills, the switch has to drop packets. The sender then notices the loss and retransmits the missing data. Retransmissions use more bandwidth and add more delay. Servers, storage systems and applications that depend on that traffic all feel the effect.

A busy link is not the same as a congested one. A link running at high utilisation with short, stable queues may be working efficiently. Congestion becomes harmful when queues grow persistently, latency rises, or packets are dropped. That is why teams should watch queue behaviour, not only utilisation figures.

What Is Congestion Control?

Congestion control is a set of techniques that help a network respond when traffic demand approaches or exceeds available capacity. It typically involves five jobs:

  • Detecting congestion, for example through growing queues or lost packets.
  • Communicating congestion information to the devices that can act on it.
  • Adjusting the rate at which senders put traffic into the network.
  • Managing queues and packet delivery inside the switches.
  • Reducing unnecessary packet loss and excessive delay.

Think of a motorway at rush hour. Traffic lights on the slip roads limit how many cars enter at once, and signs warn drivers about slow traffic ahead. Without these controls, cars pile into the same lanes, and everyone stops. With them, traffic keeps moving at a steady pace.

Congestion control does not create extra bandwidth. A 100 Gbps link still carries 100 Gbps. Congestion control helps the network use that capacity more effectively by stopping senders from overwhelming it and by keeping queues under control.

Why Is Congestion Control Important in High-Performance Data Centres?

Modern data centres move far more data between servers than in and out of the building. Engineers call this east-west traffic. Distributed applications, virtualised workloads and cloud platforms generate it constantly.

Several workloads make congestion control especially valuable:

  • Storage traffic: Delays slow reads and writes for many applications at once.
  • Cloud and virtualised workloads: Many virtual machines share the same physical links.
  • AI training and GPU clusters: Many GPUs exchange data in synchronised bursts.
  • High-performance computing: Many nodes work together on one large job.
  • Latency-sensitive applications: Databases and analytics systems react quickly to added delay.

When congestion builds, one delayed transfer can hold up an entire job. In a GPU cluster, processors may sit idle while they wait for the slowest transfer. That stretches job completion times and wastes expensive compute capacity. Even in everyday enterprise networks, congestion hurts application responsiveness and overall network efficiency.

Not every workload needs the same approach, though. A general-purpose enterprise network and an AI training fabric have different requirements. The right congestion-management design depends on your traffic and your equipment.

How Does Congestion Control Work?

Congestion control follows a simple cycle:

  1. Servers generate and transmit network traffic.
  2. Switches receive the packets and forward them through network links.
  3. When traffic exceeds available capacity, queues begin to build.
  4. The network detects or signals congestion through a suitable mechanism.
  5. The sender or the network adjusts how traffic is transmitted.
  6. Congestion eases, and the network maintains efficient communication.

Congestion detection versus congestion response

These are two separate jobs. Detection asks, “Is the network getting overloaded?” A sender may detect congestion from lost packets, rising delay or a congestion mark that a switch added to a packet. Response asks, “What should we do about it?” The sender may slow down, or a switch may pause or reschedule traffic.

The key building blocks

  • Queue management: Switches decide when to mark or drop packets as queues grow.
  • Congestion detection: Drops, delays, or marks reveal trouble.
  • Feedback mechanisms: The network passes this information to the sender, usually through the receiver.
  • Rate adjustment: Senders reduce, and later increase, their transmission rate.
  • Packet scheduling: Switches decide which packets leave first, which helps protect important traffic classes.

A simple example

Imagine eight servers sending data to one storage server. The switch port facing the storage server can send only so much data per second, so its queue grows. The switch starts marking packets, the receiver reports the marks back, and each sender reduces its rate. The queue shrinks, and the storage server receives a steady stream of data instead of overloaded bursts.

Major Congestion Control Techniques Used in Data Centre Networks

No single mechanism solves every congestion problem. Each technique below has a different role.

A. TCP congestion control

TCP is the most widely used transport protocol, and it has congestion control built in. Each sender keeps a congestion window, which limits how much data it can send before it receives acknowledgements. The sender starts cautiously and grows the window while the network seems healthy.

In traditional TCP behaviour, packet loss is the main congestion signal. When the sender detects loss, it retransmits the missing data and shrinks its window. This helps avoid overwhelming the network, but it reacts only after queues have already overflowed. That can mean added delay and wasted bandwidth.

B. Explicit Congestion Notification (ECN)

ECN lets a switch tell endpoints about congestion without dropping a packet. When a queue grows past a configured threshold, the switch sets a congestion mark in the packet’s IP header. The packet still reaches the receiver. The receiver then echoes the information back to the sender, which reduces its rate.

Marking gives earlier and gentler feedback than loss. ECN does need support, though. The endpoints, the transport protocol and the switches must all be compatible and configured correctly for it to work end to end.

C. Data Centre TCP (DCTCP)

Conventional TCP can leave switch queues deep, and deep queues add latency. Many data centre applications need both low delay and high throughput, and traditional loss-based behaviour does not always deliver both.

Data Centre TCP (DCTCP) addresses this by using ECN feedback more precisely. Instead of cutting its rate sharply at the first sign of congestion, a DCTCP sender estimates what fraction of its packets were marked. It then reduces its rate in proportion to the extent of congestion. The goal is to keep queue occupancy low while maintaining high throughput.

DCTCP does not work automatically on every Ethernet network, and it is not universally better than other approaches. It needs ECN-capable switches with suitable marking settings, plus endpoints that support it.

D. Priority Flow Control (PFC)

Priority Flow Control (PFC) is an Ethernet mechanism defined in IEEE 802.1Qbb. When a switch buffer for a given priority is close to overflowing, the switch sends a pause frame to its neighbour. The neighbour stops sending traffic for that priority, while other priorities keep flowing.

Some loss-sensitive Ethernet environments use PFC to avoid dropping packets. That matters in certain RoCE deployments, where RDMA traffic performs poorly when packets are lost.

PFC has trade-offs. A paused priority blocks every flow in it, including flows that did not cause the congestion. This is called head-of-line blocking. Pauses can also spread backwards from switch to switch, a problem known as congestion spreading. Poorly designed PFC settings can even contribute to deadlocks.

PFC is not end-to-end congestion control. It works hop by hop to prevent buffer overflow. It does not tell the original senders to slow down across the whole path.

E. RoCEv2 congestion management

RDMA lets one server read or write another server’s memory with very little CPU involvement. RoCEv2 (RDMA over Converged Ethernet version 2) carries RDMA over routable Ethernet and IP networks. It appeals to storage and AI workloads that need low latency and efficient data movement.

RDMA traffic is sensitive to packet loss, so congestion management matters a great deal in RoCEv2 environments. Many designs use ECN marking in the switches, with rate-adjustment mechanisms on the network adapters so that senders slow down when they receive congestion feedback. Some designs also use PFC as a safety net.

The exact design depends on adapter capabilities, switch support, configuration and deployment requirements. Not every Ethernet data centre needs RoCEv2 or PFC.

Common Causes of Congestion in High-Performance Data Centre Networks

Congestion rarely has a single cause. These are the most common ones:

  • Oversubscribed links: When downlink capacity exceeds uplink capacity, heavy traffic can saturate the uplinks.
  • Traffic bursts: Many servers sending at the same moment can overflow one queue.
  • Uneven traffic distribution: Load balancing may place several large flows on one path while other paths sit idle.
  • Insufficient switch buffering: Shallow buffers may struggle with bursty traffic patterns.
  • Hotspots: A popular storage node or coordinating server attracts traffic from many sources.
  • Poor traffic engineering: Weak routing, load-balancing or traffic-class design concentrates traffic.
  • High-volume AI and storage workloads: Large, synchronised transfers create heavy, repetitive bursts.

Consider a practical example. Thirty servers read from the same storage node at the same time. Each flow is modest, but the switch port in front of the storage node must carry them all. The queue grows, no matter how fast the rest of the network is.

This is why congestion can occur even in networks with high-speed switches and links. Speed raises capacity, but traffic patterns decide whether that capacity is available where and when you need it.

Best Practices for Managing Congestion in Data Centre Networks

No single configuration suits every network. Use these practices as a starting point.

  1. Monitor utilisation and queue behaviour. Watch queue depth, drops and ECN marks, not only link usage.
  2. Identify hotspots and oversubscribed paths. Find the ports and links where traffic concentrates.
  3. Select suitable switch capacity and architecture. Match port speeds, buffering and topology to your traffic patterns.
  4. Configure ECN consistently where supported and required. Check that every switch and endpoint on the path supports it.
  5. Design PFC carefully. If a lossless Ethernet environment genuinely requires it, limit it to the priorities that need it and plan for its side effects.
  6. Evaluate congestion-control behaviour for RoCEv2 workloads. Check adapter, switch and configuration support before deployment.
  7. Test under realistic operating conditions. Use traffic that resembles your real workloads, including bursts.
  8. Review performance as workloads grow. New applications, more servers and AI projects change traffic patterns.

Choose congestion-management mechanisms based on application requirements, equipment capabilities and network design. A setup that suits an AI training cluster may not suit a general enterprise network. Document your settings, and revisit them whenever you change hardware, firmware or workloads.

Common Congestion Control Mistakes to Avoid

These mistakes are easy to make and costly to find later:

  • Treating PFC and congestion control as identical.
  • Assuming more bandwidth always eliminates congestion.
  • Enabling ECN without checking end-to-end compatibility.
  • Configuring PFC without considering head-of-line blocking and congestion spreading.
  • Ignoring switch queue behaviour.
  • Applying a configuration copied from another environment without testing it.
  • Assuming all workloads require lossless Ethernet.

Most of these mistakes share one root cause: applying a technique without understanding what it does and where it fits. Validate settings on a test setup first.

Conclusion

Congestion control in data centre networks plays a central role in keeping high-performance infrastructure reliable. As servers, storage and AI workloads exchange more data, queues build, latency rises, and packets can be lost.

Managing congestion takes more than faster links. Networks need suitable detection, feedback, traffic management and sound capacity planning. TCP congestion control, ECN, DCTCP, PFC and RoCEv2 congestion management each play different roles, and none suits every environment.

Start by understanding your workload patterns. Then choose techniques that match your applications and equipment, test them under realistic conditions, and review them as demand grows.

Comxus works with industrial Ethernet networking every day, and the same fundamentals of careful design, monitoring and testing apply across Ethernet environments.

Read Also: How to Improve Network Performance in a Data Centre

Read Also: Data Centre Network Infrastructure Costs: What Affects the Budget?

Written By:

Recent Posts:

CX-NA-24GS40I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-48GS4D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS4I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS40I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-16GS4I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-SPD01-40K2P DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-SPDP02-10KS1 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

Tell Us Your Requirement

CX-NMS DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-PS-485-D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-PS-487-R DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

Get A Quote

Please fill this form to get best price for projects.

CX-ND-0GSF24S20-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-PS-487-R DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-PS-482-D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-nPA-4GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS4I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-16GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-8GSF24S40-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-1GS1l-MC DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-0GSF12S0-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-24GS60D-L3DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-0GSF48S60-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-4GS2D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS2D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS4D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-16GS2D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPG-4GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPG-8GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPG-8GS4D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPG-16GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPG-24GS4D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NG-4GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NG-8GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NG-8GS4D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NG-16GS2D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NG-24GS4D DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-4GS2D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-16GS2D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-24GS4D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-24GS4S2-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-4GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

Selected Value: 0

CX-NA-8GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-8GS4I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-16GS2I-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS2D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-48GS40D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-48GS40D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-24GS40D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-8GS4D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS4D-L2 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS40D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-4GS2I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS40I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NR-8GSF2S10I DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-ND-0GSF24S20-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPD-24GS60D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS40I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NA-24GS40I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-48GS60D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-ND-0GSF48S60-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-8GS20I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-16GS40I-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NPA-24GS60D-L3 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-NE-5HS0I DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.

CX-SFP-1 DATASHEET DOWNLOAD​

Enter your details to receive the datasheet instantly in your email.

CX-SFP-10 DATASHEET DOWNLOAD

Enter your details to receive the datasheet instantly in your email.