How to build a huge virtualized SpyNet

The show was originally named "Interop" precisely to portray the "interop"erability demanded for the latest generation of technology--non-trivial in the beginning since equipment from different vendors often didn't play well with each other. Over the years, companies big and small have succeeded or failed for the most part based on their performance at Interop. Enduring technology emerged and large sums of money was gained or lost. In retrospect, Interop uniquely and single-handedly contributed to the successful realization of multi-vendor enterprise networks that we take for granted today.

Visitors traveling to Interop expect to see newly released products working under fire. The vehicle at Interop for the competing vendors to show off performance and reliability is the mission-critical InteropNet (or what was once called ShowNet, Event Network, eNet) touted as the world's largest temporary enterprise network. For years, hundreds of handpicked volunteers (the NOC team) come together to take on the ultimate networking challenge of their career, to build InteropNet from scratch in four days. As one veteran of the process said, "So cutting edge, we are still bleeding years later."

InteropNet and SpyNet
InteropNet is actually three networks in one. The primary network is the production network that supports hundreds of vendor booths and classrooms, starting with two redundant links to the Internet, two routers, two firewalls and two 10G core switches. Connectivity to the rest of the show is accomplished through a fault tolerant 10G fiber ring, stitching together eight different wiring racks distributed across the show floor. Each rack contains additional switching gears to provide connectivity to the individual booths.

The other two networks are less visible but no less important. They are "out-of-band" in the sense that they provide alternative data paths unobtrusive to the production (in-band) traffic. They are also completely sealed off from the outside world for absolute security.

The first out-of-band network is the out-of-band "Management" network, otherwise known as Access Ether, which allows network engineers to quickly communicate with various pieces of networking gears even if the production network is severely under attack and performance is highly compromised. The second out-of-band network, which is the focus of this "How-to" article, is the out-of-band "Monitoring" network, otherwise known as the Spy Network or SpyNet.

This secondary network allows the NOC team to backhaul "replica" traffic through a completely separate overlay infrastructure, giving them great flexibility and fidelity in non-intrusive network monitoring. SpyNet enables changes and customization of monitoring traffic on-the-fly, without the need to alter the configuration of the production network during the three-day show and eliminates any possibility of overloading the already well-utilized production network.

In the early years of InteropNet, SpyNet was just another physical network, a parallel cable plant such that engineers can have a separate media-level link to any part of the network. SpyNet was deemed necessary because time-to-resolution must be near immediate and also because InteropNet is physically spread out, thereby allowing the team to see MAC layer errors and traffic from anywhere without leaving the NOC.

SpyNet was a revolutionary concept when it was introduced and today it is accepted as an industry Best Practice. Although with modern switches, one can create VLANs to accomplish similar functions, SpyNet still provides far more flexibility in unobtrusive data access and moreover, since the monitoring traffic does not need to go through the same trunk links as the user traffic, it is considered highly fault tolerant (an important feature for mission-critical networks such as the InteropNet).

SpyNet and virtualization
Today's SpyNet is no longer just a physical network. In fact, SpyNet has evolved together with advances made in the in-band network. Technology deployed in SpyNet has become every bit as complex as the switches/routers being monitored. In short, SpyNet has been completely virtualized.

There are two reasons why virtualization is important for SpyNet. One is simply that the production network itself has been virtualized. In the beginning of the industry, enterprise network topology was described as large broadcast domains separated by a few routed connections. Such flat architecture made it very easy to diagnose with a traditional protocol analyzer. All one needed to do was to plug a sniffer into a hub. Today's network is entirely switched and it is a converged medium meaning that multiple applications share the same physical conduit, making it exceedingly difficult to monitor and troubleshoot.

SpyNet evolved so that today's SpyNet is no longer just an automated patch panel. While any-to-any cross connect is still an important function, SpyNet must now provide many packet-aware functionalities (such as many-to-any aggregation, any-to-many multicasting, packet filtering, flow-mapping and load-sharing) so that instead of simple physical connections, tools are now connected virtually to the SpyNet.

Instead of receiving packets from a single access point such as a SPAN port or a tap, tools now receive packets that belong to a multi-link trunk (i.e., to get a Big Pipe view), a VLAN or a "flow" (e.g., all packets that possess the same IP address pairs and application port number so that they belong to a single end-to-end transaction or VoIP conversation).

SpyNet must deliver any desired payloads by collecting traffic from different physical parts of a redundant, asymmetrically routed and load-balanced network. In summary, SpyNet is now a "virtualization" layer between the production network and the out-of-band monitoring tools such that not just any packets, but any logical slice of arbitrarily aggregated traffic can be delivered from anywhere to any tool, at any time.

The second reason for the need to virtualize SpyNet is the fact that there are now many monitoring tools, each requiring custom data access, each competing for scarce resources such as SPAN ports and taps, and each having a potential mismatch between available processing capability and bandwidth requirement.

This year at Interop Las Vegas 2006, five vendors provided fourteen pieces of monitoring equipment for the SpyNet and one vendor provided the data-access switch, which is the infrastructure building block for the virtualized SpyNet.

Figure 1 shows the actual equipment that was deployed as part of the SpyNet at Interop Las Vegas 2006 which include the data-access switch from Gigamon, troubleshooting tools from Fluke, security tools from Extreme and Juniper, application tools from Network Physics, forensics tools from Network General and optimization tool from Internap.

Page 2: next page
Page 1 | 2