Honestly yes!
One router may kill the other router, sooner or later. The “Virtual Router” is showing the performance and power required for carrier grade networks.
So, if you are planning to acquire routers, read on, as you will get useful information on the options you have at hand, now.
In this blog, I question whether a commodity server/ x86 machine has the power and performance to host a virtual router application and if yes what are the advantages of virtual routers compared to the hardware ones.
But, before that, why virtualize a router in the first place?
Well, the NFV (Network Functions Virtualization) Use Cases document by ETSI has listed few use cases targeting different areas in provider’s space. Provider Edge routing is one of them. The throughput needs and services richness at the “Edge” makes provider edge routing an attractive target for NFV. (NFV is about virtualizing network element functions by using commodity servers)
OK so to clarify in the start, we are talking about “Provider Edge” router and NOT “Core Routers”.
Core Routers are “big beasts”. They need to process hundreds of Gbps at one time. They are not a target for virtualization, given the processing needs they have. At least, not yet.
But even then, let’s face it, is it really possible to virtualize a component as powerful as a router on commodity machine. Will it deliver the same throughput/performance needed for carrier grade routing? Are we compromising on some features/performance here?
There is no catch here, read on!
- One virtual router vendor demonstrates 80 Gbps throughput on a single processor.
- Another vendor follows by demonstrating 160 Gbps throughput on two processors running on a size as small as 2RU server.
In terms of line port throughput, we are talking about 10 Gbps per line interface; pretty impressive! Isn’t it?
While this throughput is certainly not sufficient for core routing needs, it is a definite “Yes” for Edge routing.
Before proceeding, we need to understand how a routing application is different compared to a normal IT cloud application.
How a routing application is different than an IT cloud application?
To run an application, a server needs both CPU resources and Network I/O ( input/output) resources.
A server can easily run IT cloud applications as cloud applications are CPU intensive but usually not Network I/O intensive.
On the other hand, a router has a control plane that requires high performance CPU and additionally, it has a data plane that needs faster Network interface. Therefore, routing, in general, is both CPU and Network I/O intensive application. This challenges performance of a server. A server that has powerful CPU would not help if it is not able to push packets across its interface as fast as a carrier grade router would do.
So let’s see what chipset vendors are doing to have a very powerful forwarding machine.
How chipset manufacturers are harnessing the power of processors?
Given the fast forwarding performance needed for routing applications, the processor manufacturers are doing innovations to help in that area.
So, I tried to probe more on how the chipset vendors are innovating to achieve this performance.
Just as an example, here are some details on how Intel is innovating to achieve high performance on processors
Intel®DPDK libraries and drivers
Intel DPDK gives libraries and drivers that enhance packet processing performance by 10 times. Some of the features of DPDK include
DRAM bypass
DPDK passes the packets from the line card to the code running in the userspace by completely bypassing the high-latency DRAM processing.
Buffer Manager
Reduces by a significant amount the time the operating system spends allocating and de-allocating buffers.
Flow Classification
Provides an efficient mechanism so that packets may be placed into flows quickly for processing, thus greatly improving throughput.
So 10Gbps line throughput is quite realistic today. Achievable and available!
How about throughput in future?
Chipset vendors are constantly innovating and improving. The throughput will increase accordingly, so for sure the line throughput will not stay at 10Gbps. It will scale as more innovation happens in processing technology.
Shall you only consider Virtual Router for Provider Edge?
It is important to consider your routing needs/scale and advantages of virtual routing, before making decision.
For sub 100G throughput needs, a virtual router can deliver everything a physical router can. Therefore, it can be an attractive and a better option you can exercise(See benefits below). This is the performance achievable today; but this will definitely increase as processor performance increases in future.
In any case, the investment you make in the server today is a future proof investment. You can always re-purpose it if you need to move to a higher performance platform in future.
However, I do see some reluctance by traditional hardware router vendors that have introduced virtual routers to position virtual routers clearly. This is for the obvious reason of protecting some of their hardware business. A Customer is left to decide between a virtual and a physical router as they recommend both hardware and virtual router with the choice left to the customer.
The fact is that a virtual router can do everything a hardware router can do and much more!
For example:
- Advanced IPV4 /IPV6 routing and IP unicast and multicast.
- Layer 3 VPNs
- MPLS ( LDP, RSVP, P2MP LDP and RSVP)
- Layer 2 VPNs
- Deep Packet inspection
- Stateful firewall
Some vendors go further and offer Route reflector as Virtual Router.
So not only, can one consider the virtual router for Edge applications but also for aggregation areas.
What are the advantages of Virtual routers compared to hardware routers?
There are many:
Avoiding vendor lock in and quick service innovation:
No need to depend on vendors’ road maps to develop customized hardware and interfaces. Even the forwarding and control functions can be scaled independently. The customer buys services by up scaling their licenses only. The customer can ever share a server for multiple other functions without dedicating it for specific routing functions only. This could bring significant hardware optimization. This would slow down hardware expansion. This would lead to space and power conservation.
Pay as you Grow Model:
Start gradual and expand as you grow quickly. No need to buy pool of hardware cards in order to cope with quick and urgent expansion needs. Further, no need to wait for long lead times from your vendor. Upscale by adding commonly available x86 hardware resources and just add licenses as you need them
Downscale as well as upscale:
While up scaling is an obvious advantage of virtual environment, down scaling is also a benefit. If the need for one service has gone down and need for another service has gone up, no need to dismantle or add new hardware; An operator can protect its investment by sharing the same server to remove some or add other virtual machines. The customer can flexibly assign the processor and NIC resources among different applications.
Cost effective redundancy:
Owing to many types of edge equipment’s like routers,firewalls, load balancers, it is quite expensive to have equipment redundancy. This could be easily achieved by providing redundancy through common multiple servers. One can achieve even more than 1:1 redundancy in such cases as the equipment become a pool of resources thus providing higher level of redundancy.
Proof of Concepts:
Run tests on test servers for any new feature before moving the features to production. Try before buy becomes easy both for vendors and customers. Vendors’ benchmarks , SQT (System Quality Testing) will become a lot easier compared to pure hardware environment.
In conclusion, a Virtual Provider Edge Router is viable option today and available by multiple vendors. It has obvious CAPEX and OPEX advantage and is a right step towards cloudificaiton of network. Service providers should carefully look into this option for their future edge routing needs.
How about telling me your views on whether you see a virtual router, a good investment. Leave a comment below.
Thanks Mr. Faisal for very valuable knowledge & information.
Thanks Tariq for stopping by !
Nice topic, just want to share QOS, Security, Traffic engineering are the main behaviors of Router, will it be easy for a virtual router to take over on ordinary routers ?
Thanks Javed for commenting,
For sure, without QoS, Security and Traffic Engineering a carrier cannot think of deploying an Edge router and Virtual router is no exception.
Faisal
Put me in the skeptic area, please,
(Note: While I am a SW guy, I work for a HW Router vendor now, and worked for Chips-for-routers before, so I am probably biased)
I think Virtual routers are great, and should be used/will be used for many use-cases, but I do not buy them replacing HW routers anytime soon.
1. I have seen many demos of getting 10Gbps throughput in SW per
Core, and occasionally much better (6wind had a demo with 200Gps
Forwarding on a “single intell Xeon” (with many cores).
However, I will bet money these are synthetic demos, where the
routing decision is Pre-arranged to fit in the processor”s internal
on-chip cache. If the router needs to search a nin-trivial routing
Table, or do some real packet-header-matching to a rule table for
Security and QoS, and count traffic attributes for statistics and
Diagnosis, or actually implement QoS polices that have to pass
Each packet through a token-bucket counter – you will find that
you are far from your 10Gbps/Core promised land. (Also, I’d bet
Many of these demos are done with large packets – bit moving is
Easier than per-packet decisions, so demos that show large Gbps
Numbers usually have lower PPS numbers, and vice versa)
2. 80Gbps/160 Gbps sound like a lot – if you are used to SW numbers.
For HW based routers, these numbers are not much. Common
merchant Silicon chips will easily go to 1TBps and above, and
turning on services like QoS, ACL, statistics etc. Does not cost you
any performance – you can still Expect to get wire speed. They are
Much less flexible – but usualy cheaper (in $ and power needs) and
MUCH faster.
so – I see both VR’s and Real routers coexisting – each filling different needs.
Thanks Micheal Orr,
As a whole I agree with your points.
The article does not ask for dismantling the existing PEs and replacing them with Virtual PEs.
It is emphasizing that virtual router has a place; A Provider should understand the scale and needs when making decisions. The flexibility gained with virtual router makes them good contender to be considered. As far as performance is considered, I have some test reports showing that the performance is good even when the packet sizes go as low as 256 bytes. Yes when the packet size start approaching 64, the throughput start become less. Further,it can learn million of routes and establish/maintain sessions with many BGP Peers.
Hello,
We are not there yet for Service Provider Router using Blade Servers, Of course you can do simple configuration with limited QOS, TM but i don’t think it is ready for deployment. Even building successful routers on Broadcom, EzChip,Marvell silicons which are customized for these applications is difficult . Look into any big vendors CISCO, JUNIPER, ALU, HUAWEI and they all use use home group ASIC or use either Ezchip\Broadcom to achieve router functionality.
When these big companies move and invest into Virtual routers then this may happen.
All this hype about NFV will be justified only when we see big deployment and successfully running in a service Provider.
2Cents
@ AG
Thanks for commenting.
The majority of the big vendors you mentioned, have introduced the virtual routers. Virtual router is a defnite way to go with. NFV is no longer a hype. There is a serious industry effort going on to standardize the NFV. Either we accept it or left behind in business. The virtual can do everything the hardware router can do, at its scale.
Faisal
Thanks Faisal for this excellent article , in deed we realize this kind of virtual routers and FW here in Korea as the government are using them for security filtering against DDOS attacks which blocked internet service severely over the past 4 weeks ,
Hi Mustafa,
Thanks for sharing the example of Korea where government is already using Virtual Router and FW. Good use case !
regards,
FAISAL
I think virtual network elements are nice ideas, and they will work for sure in a lab ore test pad, but I think it will last a long while, to see it live in operators networks. Maybee in special applicationes but not as a mass product.
1:) Why should an operator change the existing installation
2:) Why do should a SW router be more stable than hardware, in an operator stability is more important than price.
3:) I think Intel is pushing this to open a new market for them.
4:) I see also a big issue with the hardware it selves, to make a server carrier grade you need many changes and then it is not cheaper, if you compare price per port.
Dear Christian Frenz,
Thanks for sharing your thoughts. I dont think so that Intel is pushing anything here. It is the community of Service providers that is pushing to get these things virtualized. Even though the price of carrier grade server becomes expensive and price per port expensive, the flexibility of virtual platform still wins over its physical counterpart.
thanks,
Faisal
I don’t think that they will ‘kill’ the traditional router but it’s clear that they will become more prevalent and will eventually become the norm for a number of use cases. As an analogy I would point to the recorded music and film industries where hardware (CDs, DVDs etc.) has largely been replaced by software (iTunes, Netflix, Amazon etc.) For the majority of consumers they can get the functionality they require from the digital medium even if it offers lower performance or features. It will be the same with virtual routers – best used where they fit.
Of more interest is what will be the impact on pricing as virtual routers take over top spot. For many of the reasons you have mentioned I would expect to see a race to the bottom as effectively users would be able to replace their routed infrastructure an demand.
Luke
Dear Luke,
Thanks for sharing your views.
Lets see and wait if they will kill the traditional router:). My view is that NFV is no more hype and writing is on the wall for the Edge router to start with.
thanks,
Faisal
Faisal,
Thank you for your article and to others for their insight.
There are other uses of virtual routers such as using them for secure communications within clustered servers. Communication in this manner allows for east west communications versus utilizing the IP infrastructure and utilizing bandwidth.
All the points for using virtual routers as gateways or edge versus hardware for core routing are valid. However, one advantage of hardware is the number of ports available on a purpose built hardware platform versus a server based virtual router.
Another area of benefit for virtual routing is agility. Virtual Routers on virtual machines allow for use on an as needed basis. Management of these virtual machines and routers is supported through VSphere, KVM and other products.
Hello Jim,
You pointed out a very valid use case which is using virtual router for secure communications within clustered server, it is important for the east west communication as you pointed out.
thanks,
Faisal
As Cisco’s very first marketing hire and the guy who actually invented the router icon that you all use every day, I’ll add my two cents.
Virtual routers — at least at the edge — make all the sense in the world and are absolutely happening, but let’s not kid ourselves. Making a router virtual only attacks the economic side of the problem. Much like Arista came to market with a cheaper switch than Cisco and Cumulus is coming to market with a cheaper version of Arista on COTS, virtual routers will be far less expensive than anything built on ASICs. Neither Arista nor Cumulus has done a thing to change the way the network actually runs. Similarly, virtual routing keeps intact all of the horrible queue-based QoS mechanisms and random, chaotic packet dropping that is endemic to TCP sessions under heavy link loads.
There’s a new virtual software tech, called Network Performance Enforcement (NPE), that leverages Intel’s DPDK and “domesticates” TCP so that no flows ever time out or are forces to restart, regardless of load. Combine this with virtual routing and you have a game changer where forwarding decisions can be made based on user, application, geo-location and such. Saisei — http://www.saisei.com — is the first out with this. I suspect more will follow.
If you make it cheaper *and* make it better — like running links at 95% utilization with zero signs of congestion — then you have a compelling reason to change.
Thanks Jeff,
Good to know about NPE , Network Peformance Enforcement. So you mean to say that Combining NPE on top of COTs server increase packet efficiency multiple times ? Would look further into details of NPE. One of the point against packet processing stacks on top of COTs server is that it makes the product expensive rather than cheaper , how do you compare it to the price of hardware router then ?
Faisal
Saisei actually uses a SaaS model — more disruption is good, yes? A user purchases a recurring software license for aggregate bandwidth. So, say you purchase a 1G license, then you could deploy 10 100M instantiations, for example. In most cases we’re talking prices that are basically in the range of the yearly maintenance contracts for routers or WAN optimizers and such.
NPE runs on x86 cores an memory only. Small systems take about 3-4 cores and 16G of memory.
Thanks Jeff for some quick useful info on the products in market.
rgds,
Faisal
Definitely see the value in virtualizing the routing. Starting in PE and even internal networks (Enterprise) makes tremendous sense. For those just beginning the network virtualization journey, start slow to gain knowledge and minimize risk.
Thanks Michael for stepping in with your comments. I agree with you. You hit the nail by saying “start slow to gain knowledge and minimize the risk”
Faisal
Thank you so much Failsal,
Really, great article.
I Think that Virtual Routers have more scalability and flexibility than Hardware routers. But we cannot say that HwRs will kill VRs.
Question: What about licensing (FW, VPN …etc)?
Thanks Bouchaib,
For compliments.
You brought up a good point. Based on our experience Vendors are ready to neogitate more on the SW than HW. But of course the recurring licensing charges if any, need to be considered to compare the TCO of virtual versus physical network. I would expect some virtual router vendor would cast his opinion here.
Good subject to contemplate. Though this article covers a wide angle of subjects and does not cover the complete picture of most virtualized environments, I choose to focus on the concept of VR.
It is understandable why network engineers today view the concept of VR from the SP provider point view, however what gets overlooked in the discussions is the
view from the other side, the end user.
The concept of virtualization originated at the server level and has been the backbone of most global data centers that directly serve end user’s dynamic demands. When we come to the intersection of SP and VR as this article highlights, a more appropriate question to ask is where virtual routers can fit best in the SP network.
A picture of a bicycle gear comes to mind when reflecting on network layers functions and change dynamics. The front sprockets that are attached to the crank arm are called chainrings. Chainrings could represent the SP layers of core (inner, outer), provider edge aggregation, services layer, and provider edge. While the gear cluster on the rear wheel (called a freewheel or a cassette) could represent the various markets the SP is serving.
VR, IMHO, will play a massive role on the user end more so than the in the SP network itself. The economics drivers to commoditize the network, and consequently network virtualization, will drive a change in the SP infrastructure. However this change will be considered small relative to the change that is projected to drive at the consumer end. I argue that network virtualization and commoditization is targeting normalizing end users access and network capabilities more so than the SP infrastructure.
Thanks for the opportunity to comment.
Thanks Zaid for sharing your thoughts and bringing in a different perspective. I appreciate it. Can you tell us more with an example , perhaps, on how the VR is adding to the end user benefits more than service provider’s benefits. This brings a total new perceptive then perhaps anyone of us thinking.
Faisal
Many thanks Faisal. I am following the steps mentioned by Michael ” “start slow to gain knowledge and minimize the risk”. This is really required as it NFV is new entry into telecom industry.
I have found below use cases on NFV :-
Virtual Network Function as Service ( VNFaaS)
Virtualization of Mobile Core Network and IMS
Virtualization of Mobile Base Station
Virtualization of Home Environments
Virtualization of CDNs
Fixed Access Network Functions Virtualization
Virtual Network Platform as a Service (VNPaaS)
Virtual Network Infrastructure as a Service (VNIaaS)
Virtualization Forwarding Graphs
Apart from above, is there anything else . If something is missing I would be delighted if someone add more inputs on use cases.
Also, Do we have any roadmap on implementation of NFV in Carrier Network ? Not sure, if Service Providers are still thinking on implementing the same !!!!
Yes you are right with the use cases.However, Any thing in the edge would be a hot use case, because it is least disruptive. For example virtual CPE that is a CPE installed at customer site.
NFV as a complete standard is in development and standardization phase. As a service provider you can start virtualizing edge till implementing complete end to end NFV.
Yes, very true. Which organization is working on standardizing the same ? Is it ETSI ?
Yes Abdul Rauf, ETSI is defining the main baseline architecture of NFV!
Thanks Abdul Rauf….NFV is still new with use cases here and there…Like Virtual CPE is a hot use case. Shifting to a complete NFV for a service provider is a very long term project as service providers have invested heavily in legacy platforms. What service providers might do is to take some of the use cases and start implementing them gradually.
Hello faisal,
Really interesting article,Nice explanation, Good Effort.
Regarding Service Provider transformation for NFV/SDN, i would like to inform AT&T has plan to migrate 75% of its network to SDN in 2020, and they will complete almost 10% transformation this year.
Google internal wan(GWAN- used between google DC) is running on NFV/SDN .
Thanks Prabhakant,
You are right, AT&T is one of the leaders in taking initiative for virtualization.
Faisal
Hi Faisal,
Nice share for this information, but i have question.
how can you seamlessly move from existing traditional router to vRouter ?
let’s say there’s existing PE router with couple GE & 10 GE connection towards uplink/downlink,then we want to move to PE vRouter while in the PE site there’s no data center available ?
Thanks Leon. The initial use case may be a CPE at customer site which you can easily virtualize using VR. The other would be to start using it in new areas where customers density is less and you dont need a powerful edge router. A virtual router can seamlessly live with physical router so you can easily consider it when planning for your next purchase.
It is a very nice article, after reading this, my doubt has cleared on this topic and also helps enhance my knowledge.
Great work!
Thanks for your blog that talks about this hot topic.
In seems rational to use vRouter in CPE cases. Now, I’m curious to know, is it possible to run vRouter in case of CSG cases? Although, I think there is two subjects to be considered, throughput and timing requirements.
Yes definitely, if not as server, I have seen CSGs available on broadcom commodity switch with timing features….
I think you’r talking about Volta Network or DriveNets companies. Do you think, is it possible to see CSGs based on x86 data-plane by using DPDK in near future?