Overlay vs Underlay in SD-WAN & DC- Easy Guide

The “Overlay vs underlay” discussion is important as overlays are everywhere in networks- in the Datacenters (DCs) and in SD-WAN.

And there are many tutorials out there explaining overlays and in particular difference between overlay and underlay. However, they are difficult to understand, and more importantly, they do not address the most important point – what problems overlays are trying to solve. What are the limitations in the underlays that triggered a need for the overlays?

Therefore this blog takes you step by step on the concept of underlay, its limitations, and how overlay solves it.

I will do so using SD-WAN as a reference, but the concepts are the same in the Datacenter i.e SDN in Datacenter also uses the concept of overlays.

Difference of Overlay and Underlay networks in summary

Overlay vs underlay in summary
An Underlay network is the physical network responsible for the delivery of packets like DWDM, L2, L3, MPLS, or internet, etc. while an overlay is a logical network that uses network virtualization to built connectivity on top of physical infrastructure using tunneling encapsulations such as VXLAN, GRE, IPSec.

Why overlay networks are important and worth discussing?

Do you know that traditional routers and switches are hardware-centric and perform packet forwarding based on the destination IP address, therefore network segmentation and multi-tenancy are not possible.

This creates the following inefficiencies and drawbacks: Network segmentation and network slicing are not possible – duplicate IP address ranges cannot traverse a single IP network natively; Scaling is hard. VPNs can solve these issues albeit with some inefficiencies as they are very transport medium centric and explained below.

On the other hand, the overlays bring the flexibility that underlay is not able to provide. Overlays have been there for a quite while but with the SD-WAN they have become very popular.

This blog uses the example of SD-WAN, but the concepts and advantages are exactly the same for the Datacenters, you will just replace the CE router with a virtual machine ( VM) that needs to communicate with Spine switch which is the CE a the HQ in this example. The underlay network is the MPLS here, it can be any routing protocol in Datacenter like OSPF, BGP, etc.

Let’s start to understand underlay before going to overlay

Let’s take the example of a common physical infrastructure for example a WAN network, In which a customer has three sites A, B, C with the C as the HQ which are all physical devices. There is a layer 3 IP/MPLS VPN between the 3 sites which is the underlay for this customer.

This is a Peer-based VPN, which means that the CE ( Customer Edge) peers with PE ( Provider Edge router), exchange routes, and then how the routes are advertised between the PE is the responsibility of the MPLS provider and not the customer who owns CE. Once CE handovers traffic to PE, it forgets on how it is transported inside the Provider network and which route it takes.

What problems underlays cannot solve?

While the above shows a simple example of a Layer 3 underlay. Things have changed beyond the use of just “MPLS” for transport. Think of it! internet is everywhere. High-speed data is cheaper than before. Exploiting the use of the internet for carrier transport is just a natural evolution then.

To understand the concept of overlay, I introduced two more networks. A broadband network and the 4G network. The need is to use these “low cost” transport networks to offload some of the data from the MPLS network ( if not all)

There are some issues here :

Issue 1: The Internet cannot be used as an underlay for private traffic

The first issue is that internet medium natively is insecure so it cannot be used as an underlay for private traffic. To protect it we will need to use security protocols like IPsec. IPsec is a tunneling protocol. Which is an overlay. ( but we wanted to solve the issue with underlay, isn’t it? )

Issue 2: Transport Dependence is too restrictive

Let’s for the moment assume that the internet is secure so we kill the first argument against the underlay. The customer wants to divide the applications between the different circuits ( for example YouTube on 4G, A SaaS app on broadband, and email on MPLS). Further, he wants to use the backup of MPLS as broadband. All this is possible but very complex and static which means if the customer wants to change the application’s distribution over the circuits. He needs to change his configurations which is operationally NOT easy. To put it simply the configurations are hardware-specific, which means they are specific to the physical underlay and complex to change.

Issue 3: Multi-Path forwarding

The customer wants load balancing by combining underlays of different types.

However, it is very complex ( if not impossible) to use multiple underlays which are of different types, simultaneously to forward traffic from source to destination.

Welcome to the Overlay Networks. The software instead of hardware is the solution

Until now we learned that if we stick with the underlay, it can restrict us in multiple ways. In order to get transport independence, overlays can come to our rescue.

Overlays are software-based and not dependent on transport. It’s like a virtual network ( virtual links) on top of the physical network. This virtual network needs to have underlay created first.

The trick is simple. Create an underlay once and forget it. Now create overlays on top of it and since they are software-based. changing the configurations is super easy and friendly. Underlay is a pipe on which overlays run. The intelligence is all in the overlay.

However, to have overlays in SD-WAN, we will need a special CPE called SD-WAN edge device. These overlay nodes are installed at all the sites.

How Overlays are formed?

The following example shows a GRE tunnel formed by SD-WAN Edge Device. The edge device adds a GRE tunnel header with a new IP header and masks the inner IP header from the MPLS domain, the MPLS forwards based on the outer IP header.

Once the packet reaches its destination, the SD-WAN edge removes the outer IP header and the Tunnel header and what we get is the original IP packet. During all this process the overlay is not aware of the underlay.

The same process can be done for the internet underlay but with the addition of encryption using IPSec. All this can be done seamlessly using the same SD-WAN Edge box.

The overlay is formed once encapsulation happens at the left SD-WAN box and ends when the header is removed at the destination. The overlay is not aware of the underlay on which it is riding. The underlay here is MPLS but it can be any other type of transport.

How Overlay networks solve the issues ( Advantages of Overlays)

1, the Security issue is solved by using encryption. A Public underlay such as the internet is not trusted. SD-WAN edge solved that issue through the use of IPsec ( This solves issue 1 above)

2. Transport Independence. The same SD-WAN overlay can seamlessly connect to multiple different media like the internet, MPLS, L2 media, etc. The overlay does not care about transport media as it is using tunnels. With this, it opens the world of innovation and flexibility in the services. The intelligence is now in the software and not in the hardware. The applications can be dynamically forwarded and distributed between the 4G, broadband, and MPLS. There can be multiple ways in which traffic can be protected between different transport media. ( This solves issue 2 above)

3. Considering overlays are transport agnostic, there is no limit on the number of multi-paths that can be used in overlays to carry traffic from source to destination for the purpose of load balancing. This makes overlays very scalable.

So in summary overlays wins in all cases and that’s why they have become popular these days.

Overlay vs Underlay in summary

Items	UNDERLAYS	OVERLAYS
Definition	Underlays networks are the physical networks responsible for the delivery of packets like DWDM, L2, L3, MPLS, or internet, etc.	while overlays are a logical network that uses network virtualization to built connectivity on top of the physical network using tunneling encapsulations such as VXLAN, GRE, IPSec.
Encapsulation	Does not require encapsulation	Requires encapsulation such as VXLAN, GRE, etc
Application Segregation	Very complex to divide applications across different underlays	Very easy to achieve this
Hardware or software	Hardware-based connectivity.	Software-based connectivity
Multi-path forwarding	Complex to set up multipath forwarding	Inherent support for multipath forwarding
Scalability	Limited scalability as hardware-dependent	More scalable as software-based and because of the multi-path capability
Security	Physical underlay such as the internet is not secure	Security can be easily added using encryption methods such as IPSec.
Deployment time	Long deployment cycle as hardware-dependent	Quicker deployment time. Adding and changing services is very quick

Reference: https://datatracker.ietf.org/doc/html/rfc4110