When revenue per bit starts falling and cost per bit starts increasing, it’s time to think about alternative strategies to expand Transmission/IP network.
As IP is predominant traffic in Transmission networks, therefore any expansion strategy on Transmission should be driven by IP demands. Indeed, both IP and Transmission departments need to work for a joint and synergetic expansion strategy, in this regard.
Hence, I intend to share with you some insights and tips that will enable you to plan the expansion of IP and Transmission network in a holistic and synergetic way. If you stay till the end, you will see how IP can live with lower layers of Transmission (DWDM, OTN, SDH, and L2) through a successful marriage.
Whether you belong to IP department (responsible for the IP/MPLS domain) or Transmission department (responsible to run SDH, DWDM, OTN and most recently Ethernet), there is some food for thought for you in this article. More specifically, I will show you how to put your traffic on the right layer that will reduce the TCO (Total Cost of Ownership) of your Core network.
So let’s see where is the problem?
The Problem#1 is that of treating both IP and Transmission as two different entities in terms of any expansion strategy. There is a reason for that: classically, both IP and Transmission are run by separate departments. Transmission (lower layer) is treated as a resource to serve IP layer (upper layer) without the need for Transmission to understand or help upper layer ( except connectivity).
I will give you one example:
Have you heard of the “Dumb Pipe” approach?
An approach, in which IP uses Transmission as transparent pipe. In other words, shifting complete aggregation, protection and intelligence to IP layer, thus keeping Transmission as dumb and non-intelligent pipe.
For obvious reasons, this strategy is simple and easy to implement. As Transmission does not need to provide any other service except connectivity, therefore there is little or no interaction needed between IP and Transmission folks.
But there is another side of this approach:
More CAPEX and OPEX spending!
Such operators need to consistently build up two layers in parallel: the IP layer and the Transmission layer.
I call it “Grow-Routers-Fast, Grow-Lower-Layers-Fast” (GRF-GLF) approach.
In GRF-GLF approach, Transmission is participating neither in aggregation, nor in protection or any intelligence for IP packets. Therefore, IP layer takes complete burden to achieve these objectives. More specifically, the bits, after entering into IP core, need to touch every core router/IP layer on its way to destination. All this because, IP is the only layer that can aggregate, protect or route the traffic.
As, all bits have to pass through the transit routers as well, so we end up building interfaces for three layers:
- Originating Core Router
- Terminating Core Router
- All Intermediate/Transit Core Routers
Isn’t it unfortunate that the intermediate router has to take the burden of the bits that are not dropped or added by it?
With every scale of routers, there would be a need to corresponding scale the Transmission resource. So indeed, we are constantly building routers and building Transmission with fast pace- a GRF-GLF Strategy!
Secondly and in contrast to above, there is another approach that calls for “Router Offload”. The term “Router Offload” has been abused a lot by opponents and proponents on both sides. “Router offload” in its simplest definition mean that any transit traffic at any intermediate core router should be offloaded to lower layer like SDH/OTN/ DWDM.
Therefore, core routers at any intermediate site do not need to be scaled for transit traffic. This will result in huge cost savings.
However, terms like “Router offload” and “IP offload” are pitched in a sense that gives out a massage that core routers are not needed at all. They go to the extent of even calling it “hollow core” designs with no core router in a network at all. Indeed, a wrong perception. Later on, I can show you that “Router offload” for some traffic does make sense but not for all kind of traffic.
It will also become apparent that Router offload is part of a solution but not a complete solution, if applied to all traffic.
What is the solution? How can IP and Transmission live through a successful marriage?
Having presented the problem, let’s see how we can solve these issues.
I would call upon some statistics that will support me to solve these problems.
Some stats form Cisco VNI-2013: (Annual report on future global data; following excerpts are from the updated report of June 2014)
- Fixed internet data will rise at CAGR of 20% annually up to 2018
- Mobile data (includes Mobile data and Internet traffic) will rise at CAGR of 61% annually up to 2018
- By 2018,Consumer Internet will represent two thirds of all IP traffic followed by managed IP at 20%
The key message here is that internet data will rise considerably (at CAGR 20%) more than any other data (e.g. enterprise data) in networks. And by 2018, consumer Internet will be two thirds of all data carried over networks.
I repeat the key message is, “Internet data” ( Also called HSI:” High Speed Internet”) is/will be the pre-dominant traffic in networks”.
The direct corollary is; “The nature of “Internet data/ HSI data” should drive any plan for network expansion”.
So what is the nature of the Internet data? The Internet data flows to fixed destination in network: towards internet gateways from originating routers. In fact this is a “point to point” data .This very nature is quite different than any other layer 3 VPN data that has to follow any to any/ mesh connectivity and needs to stop at every core router.
In other words, for the Internet data, the destination is fixed and known to an operator.
This nature of Internet data, as probably you have guessed, is similar to the point to point behavior of lower layers of Transmission. Wouldn’t it be appropriate to push this date to lower layers and take advantage of the lower layers?
This works on the principle that ““Cost to carry data on the lower layers is less than carrying it over IP layer”
By intelligently designing core network, routers can take help of the lower layers (L2, SDH, OTN or DWDM) to carry the traffic to the Internet Gateway site.
This means, the data does not need to touch IP layer at every intermediate core router because of transit router bypass.
Wouldn’t it result in slow down of core routers build up?
In fact, core routers do not need to grow as fast as lower layers need to, thus bringing significant cost savings for Operators. This strategy would result in “Grow-Routers-Slowly, Grow-Lower-Layers-Fast” (let’s call it GRS-GLF strategy). For Obvious reasons this will bring the TCO of the network lower.
Going to the earlier debate I can now say that GRS-GLF strategy neither makes the Transmission dumb as lower layers are helping the routers in aggregation (in addition to protection also but that is a subject for another day), neither makes a complete offload of core routers (as we are talking about the Internet data only) as the proponents of ” Router Offload” would sometimes want you to believe.
Having explained why the GRS-GLF strategy makes sense, let me now share some guidelines on how this can be implemented in networks.
(PS: While the discussion would refer to “OTN”. The data applies equally to SDH. Note that OTN here means OTN switch, not the OTN mapper in DWDM)
TIP #1 Know the nature of your Traffic very well
Meaning you should know what kind of traffic is carried over routers. What is the mix of layer 3 VPN versus Internet data? In other words, what is the ratio of traffic for enterprise/business connectivity versus internet data? Perhaps the biggest mistake the planners do is designing the network for one pattern and one type of traffic.
Let’s say the Internet data is 60% (which is a decent estimate) compared to other data. This means you can start zooming on this 60% of data to be offloaded to lower layers.
TIP #2 Know the Origin and Destination of your Internet Data
An easy way to look is the IP destination of the traffic. The IP destination will tell you whether the traffic is destined for Internet Gateways or not. The best point would be the aggregation points in your network where you are collecting the Internet data.
Here you can ask this question. Can I send this traffic directly to the Internet Gateway bypassing the intermediate core routers?
If the answer is yes, you need to answer the question in TIP 3.
TIP #3 Select Suitable Lower Layer to take the traffic to Destination (DWDM or OTN or L2 Switch?)
You need to take important decision here. Which layer to offload to?
If the up-link pipe of router is congested and already filled with internet data and up-link rate is same as DWDM lambda rate, the best would be to go to DWDM layer directly as OTN is not adding any benefit of its aggregation capabilities. Let the DWDM take the traffic directly to Internet Gateway.
If the uplink pipe of the router is partially filled, there would be a need for a lower layer that has aggregation/statistical multiplexing capabilities, both low layers like Layer 2 Switch and OTN can help here. (See next point also)
Interestingly these days, layer 2 switch and OTN are packaged in the same switch thus operators can take advantage of such switches for grooming capabilities. Remember we are talking about GRS-GLF strategy, so we are building lower layers fast but not the IP layer at same pace.
TIP #4 Keep Router Up-Link rate to OTN, lower compared to DWDM lambda rate!
(I am assuming that you already own OTN switch, if not you can still offload traffic to lower layers like DWDM and L2 Switch as TIP 3 tells).
Why would use an OTN switch ( in between router and DWDM) and then keep the up-link of the router at same rate as DWDM rate. This is in fact more CAPEX. We are now growing higher layers and lower layers at the same time.
If you are using an OTN switch, it is recommended to keep the uplink of the routers low compared to DWDM lambda rate. For example if you are using 100G lambdas, recommended uplink rate for router to OTN should be 10G. This would enable you to use the grooming functionality of the OTN in the intermediate sites. One of the mistakes I see is that some service providers keep both rates same. This would deprive the operator of using the grooming functionality of the OTN layer.
TIP #5 Use Color/DWDM Interfaces on OTN
If you are using OTN switch and DWDM from same vendor, it would make sense to explore the use of color DWDM interfaces that can give traffic directly over the DWDM instead of using a lot of back to back grey interfaces between OTN and DWDM. This will result in a lot of CAPEX/ OPEX saving.
If you are using OTN switch and DWDM system from different vendors, you would need to run interoperability tests to use the color interface of the OTN of one vendor over the DWDM system of the other.
TIP #6 The IP and Transmission Departments need to Handshake
Last but not the least!
If the IP and Transmission departments are different ( as usually in Tier 1 Operators) , it would need a good coordination between the two in order understand from each other on how best to expand both IP and Transmission layer that can bring total cost savings for operators. They would need to plan together on how to use all layers intelligently bringing the Transmission cost per bit down.
If there is one department that plans both the above networks, this is ideal as there will be combined strategy for IP and Transmission expansion.
As you can see that all the above tips apply to using lower layers of Transmission to support efficient and cheapest transport of IP data. These apply in particular to Internet/HSI data and I repeat by using selective offloading of IP traffic, an operator can get considerable cost benefits.
I can bet that by exploring these out of box approaches, the IP and OTN/DWDM can live happily together through a successful marriage.
While this article is mainly looking at how Transmission pipes can help the upper layers in terms of cheap transport of data by using aggregation intelligently, I can write at same later time on how the lower layers protection options can further help the MPLS layer in bringing down the cost of protection at upper layers.
Now it’s your turn!
I would love to listen to what do you think about the arguments and Tips in this blog and further,how you are coping with the tremendous expansion in the IP data in your network.