What your DWDM vendor did not tell you about latency for your Data Center Interconnect

 

Are you concerned about  latency in your data center links running  over optical network ?

Or, are you planning for Data Center Interconnectivity in near future and want to know  the impact of delays involved. The read on, for some useful tips to know your options.

Chances are, you are already using DWDM to connect data centers but there are few things that you should be aware of as an operator regarding the sources of delays in your optical links in order to either mitigate them or at least know how to properly dimension your network for delay. As they say , NOT all DWDM systems are equal when it comes to latency in them so it would help to know what are the choices available when comparing DWDM systems.

In this era of Cloud networks, it would be naive to ignore the impact of delay in optical networks that run clouds. Impact of delay for financial markets that do High Frequency Transactions ( HFT) can be considerable and far reaching; fraction of millisecond can impact revenue and as per one estimate,these fractional delays can contribute to difference of revenues over the year as high as 100 Million USD.

Lets see, what are these factors that cause delay and what measures can be taken to deal with them. Perhaps your DWDM vendor did not tell them to you, yet.

 

1. Fiber Delay

 

Delay Impact = 5 micro seconds / KM

Speed of light in vacuum is 299,792,458 meters / seconds. This equates to 3.33 mirco seconds/KM. Since optical fiber has different refractive index compared to vacuum therefore  latency is more and equals 5 microseconds/ KM.

So if there is a 100 KM of fiber between two network elements the delay would be 500 micro seconds ( This does not include delay of the network elements themselves)

Not surprisingly, fiber delay  is the biggest culprit and contributes to the major delay  in optical network  and it would impact no matter which DWDM system you use. Long fiber link causes more delay than shorter fiber link.Little  can be done to mitigate the latency owing to fiber except selecting shortest route if it is an option.If there are two diversity routes available between two stations, operator is advised to use the shortest route as the primary path and use the longer one as secondary or backup path; if  new locations for data centers are being planned, the  locations can be strategically selected in order to have minimum latency owing to fiber itself. While calculating latency coming from fiber, optical lengths should be considered and not geographical length. Operators should have good record of fiber and optical distances to dimension the delay properly. A good OTDR trace is very handy to know the optical length of the route.

Recommendation:

wherever possible use shorter distance for connectivity. Avoid using longer routes.

 

2. Transponding/Muxponding including FEC

 

Delay Impact = 10 micro seconds to few hundred micro seconds ( depending on model and make)

Also called “color conversion”. There are several processes involved in transponding/muxponding. Some of them are •Data encapsulation •Forward Error Correction ( FEC) •Performance Monitoring

Of these forward error correction is perhaps the biggest culprit. Delay due to it ranges from 15 to 100 microsecond.

A word about FEC:

Low OSNR is directly related to poor BER performance in DWDM networks. FEC or Forward error connection is a method used to achieve coding gain for higher bit rates. It is a method of encoding optical signal with extra error detection and correction overhead bytes enabling optical receivers to detect errors and correct them. Thus FEC can reduce BER and effectively increases distances reachable by high speed signals without regeneration.

Forwards Error Correction is a must have feature when going on longer distances ( several hundred kilometers)  but it does not bring considerable value at shorted distances ( metro links). There are some models of transponders/muxponders that can switch off the FEC option . There are other models specially designed without FEC option and optimized for low latency. Some vendors are offering transponders with “No-FEC” option for shorter distances that can give ultra low latency of few nano seconds. They claim that they can carry  signal in its native form without processing and  recommend them for data center connectivity.

Recommendation:

Whenever possible use  latency optimized transponder/muxponder from your vendor. Check with the vendor if it provides any such model. For green field deployment,do make this a benchmark for evaluating, which vendors to go with by comparing all variants of transponders/muxponders, the vendors have to offer. Not all vendors provide these kind of variants. If possible, go without FEC for shorter distances.

 

3. Native Optical

 

Always remember a simple rule of thumb : The lesser  the processing of an optical signal, the better it is for latency. Said in another way, the lower we are in the OSI layers the better it is for latency.

So when it comes to latency

• Layer2 switches perform better than routers
• OTN/ SDH performs better than switches
• DWDM performs better than switches ( DWDM is also called L0)

Simple: Higher layers have to process overhead at their respective layers that adds to latency.

This does not mean that we start thinking of removing switches and routers. They are not replaceable. However if  traffic is already processed at layer 2 or layer 3 at the point of origin and there is no need to do re-processing in the middle of the network , why not pack the traffic in ODU ( OTN)  and transport it to the destination. Better still if one can avoid processing at OTN layer, use native DWDM altogether.( considering OTN also adds overheads). Few vendors offer latency optimized short reach transponders without OTN processing.

Recommendation :

Be as close to L0 as possible

 

  4. Dispersion Compensation:

 

Delay due to DCF based Dispersion Compensation Module (DCM) = 5 to 1oo micro seconds

Delay due to Fiber Bragg Grating ( FBG) based DCM = Few nano seconds.

Are you obliged to use dispersion compensation ?

Perhaps you are running 10G lambdas, so you are already using some sort of dispersion compensation in your network.

Check for Dispersion compensation modules using Fiber Bragg Gratings. They are the latest type used to compensate for dispersion related issues. There have smaller length compared to the earlier systems using  Dispersion compensation Fibers ( DCF). Which were long spools of fiber usually 20% of the length of the fiber it compensates. Delay due to FBG is negligible compared to that of DCF.

Recommendation :

Try to remain DCM free if deploying green field DWDM. If still DCMs are needed because of 10g lambdas, opt for Fiber Bragg DCM to reduce latency

 

Conclusion:

As can be seen there are multiple factors that contribute to latency. There might be other components in DWDM network that can cause some latency like amplifiers, WSS etc. But the impact of latency due to these factors is not as considerable as the factors listed above. A well engineered network has to take care of all these factors to reduce latency in networks. Here is a summary of recommendations for low latency data center Interconnect.

a. Optimize fiber routes. Use shortest path for main traffic  in order to have low latency.

b. Try to be as close to L0 as possible and as native as possible.The best is to use DWDM in order to remove delays caused by the extra overheads at upper layers.

c. Check for latency optimized transponders/muxponders by vendors that can give delay in the range of nano seconds. Disable FEC if not needed. FEC causes additional delay.

d. User Fiber Bragg based DCMs instead of DCFs.

Finally, I would love to hear on what steps you have taken to optimize your data center interconnect.