Problems and solutions in designing low-cost, high-performance multi-service communications equipment platforms

A multi-service switch system (MSS) is a network device, usually deployed at the edge of a network, that connects access network elements with core network elements. The role of a MSS is to consolidate and aggregate traffic from access networks to the core network.

A MSS is designed to handle multiple services and protocols because of its role in the network architecture. A core network is typically based on a scalable and reliable protocol such as asynchronous transfer mode and Internet protocol. The core network trend is to migrate to multi-protocol label switching (MPLS) in the future. MPLS is a versatile technology that addresses current communications network issues, including speed, traffic engineering, and quality of service (QoS).

However, there are multiple access technologies such as integrated services digital network (ISDN), T1/E1 lease lines, xDSL (digital subscriber line), cable, and wireless. These access technologies use different transmission media and protocol at various data rates. As its name suggests, a MSS provides multi-service traffic aggregation, switching, and routing at the edge of the service provider network. These switching systems remain a hot commodity in the network today because of the existing and new services they provide. These services allow service providers access to new revenue and enable them to make use of the bandwidth installed in the core of the Internet today. Its role also demands carrier class characteristics such as high level reliability and service availability.

Currently, MSS enables the aggregation of both voice and data traffic. Such switches also must be able to deal with a multi-protocol environment where more mature technologies such as time division multiplexing (TDM), IP, ATM, and frame relay (FR) are merged with newer technologies such as Metro Ethernet and MPLS. In addition to aggregating many types of traffic, MSS must be able to provide service creation capabilities. It is these services that allow service providers to maintain current revenue and create new revenue. These services include virtual private networking (VPN), fast provisioning of new customers, subscriber management and billing, service level agreements (SLAs), and security services, including fending off denial of service (DoS) attacks and providing firewalls.

In addition, the newest MSS continue to address the issue of costly central office (CO) space by offering high density/cost effective systems. In fact, in some cases, many older, lower density systems can be replaced with a single, higher density router. It is this ability to create new services and yield new revenue, along with the new lower cost, high-density routers, which has created a dual business case for service providers. This, in turn, has created continuing demand for MSS in this time of capital expenditure cutbacks by communications service providers.

MSS System Requirements
A MSS has a set of common system requirements. System engineers design their platform and select solutions based on system requirement parameters.

A system original equipment manufacturer differentiates its product in size, capacity, scalability, feature set, service support, and price. A clear trend is the "common platform" approach-using a chipset that can be used to build common blocks, allowing different hardware configurations and software loads to meet different market needs, such as location variation, capacity variation, and feature variation. This approach minimizes the hardware and software development cost of system OEMs. More importantly, it protects service providers' equipment investments. When higher capacity is needed or a new service is required, a service provider does not need to "fork-lift" the equipment; instead, the new software can be loaded or new line cards can be installed without any service disruption. This common platform approach becomes reality thanks to the technology advancements such as system-on-a-chip implementation and programmable soft cores.

Multi-service switching systems come in various sizes and architectures. All such systems, however, must process information on two different planes, a control plane and a data plane. The data plane handles the forwarding and modification of the data as it moves through the MSS data path from the ingress port to the correct egress port. The control plane is responsible for understanding the network architecture and communicating with other routers or network elements to which it is attached. The main functions of the control plane include signaling and routing. Additionally, the control plane creates and maintains all tables used by the data plane in the forwarding operation.

A line card provides multiple user interfaces and most of the data plane functions. A line card can be implemented as a single printed circuit broad or split to a physical interface module and a protocol engine module. The latter implementation maximizes the hardware commonality. Various physical (PHY) modules can be designed for different interface types and port density. For low rate interfaces, a traffic multip-lexing function may be used between the PHY module and the protocol engine. Line cards are connected via the switch fabric.
A switch fabric provides connectivity from any line card to any line card. Its performances, such as throughput and QoS capability, influence overall equipment system performance. The switch fabric is the central point of all traffic. Its protection should prevent or minimize the data loss in the event of system failure. Additionally, the fabric should be protocol independent, natively supporting all traffic types.

The control system of a MSS consists of system control processors and module processors. The software architecture and the software function partition is system dependent. The system control module is typically protected by a stand-by control module, because many service providers require "carrier class" MSS that provide for 99.999% uptime.

Switching Sub-System Solutions
The switching sub-system is responsible for connecting line cards via a system backplane and switching fabric. This is typically the first design item a system engineer will start. Choosing a fabric configuration should take into account the following system parameters: the number of line card slots in a particular platform configuration and user bandwidth of line cards.

Line Card Solutions
A line card provides most data path functions, such as packet classification, policing, statistics, forwarding and protocol inter-working.

Network processor technology contains multiple programmable engines for the packet processing functions. This means software determines the types of services that can be supported. The following functions are relevant for the multi-service applications.

• Packet classification. Network processors are capable of classifying multiple packet/cell types at the line rate. The result of the classification determines the type of the processing treatment and forwarding destination.


• Packet assembly and segmentation. The standard AAL5 SARing (segmentation and reassembly) at the line rate is supported for the IP-over-ATM application. Any proprietary segmentation and assembly is also supported for the fabric interface to work with a cell-based fabric.


• Packet policing and statistic collection. The packet policing algorithm is user programm-able. Statistic collection is also user programmable.


• Protocol Inter-working. On the per traffic flow basis, ingress layer 2 protocol and egress layer 2 protocol can be different. Various encapsulations and tunneling can be supported by the programmable packet modification engine.


• Traffic Management. There are dedicated engines for buffer management and scheduling functions. A comprehensive set of traffic management functions such as DiffServ, RED/WRED and QoS/CoS can be applied to different traffic flows.


Network processors need to provide a high performance processor architecture that gives system designers a powerful building block for the development of MSS Solutions.

They can be used in stand-alone systems or as a component in large multi-rack solutions. Network processors should be capable of performing the following tasks at full line-speed.


• Packet classification.Identifying a packet based on known characteristics, such as address or protocol-including SARing for AAL5 based traffic.


• Packet modification. Modifying the packet to comply with IP, ATM, or other protocols (for example, updating the Source IP address for NAT.


• Queue/policy management. Reflects the design strategy for packet queuing, de-queuing, and scheduling of packets for specific applications.


• Packet forwarding. Trans-mission and receipt of data over the switch fabric and forwarding or routing the packet to the appropriate address.


• Policing, statistics collection, traffic management and control plane interaction are all functions of the Agere Systems PayloadPlus network processor family of solutions.


System Design Considerations
Designing a MSS platform is a complex task that involves many aspects of equipment design. One may argue this is true of all communications product design. What is special about designing a MSS platform is its particular feature requirements for QOS system reliability, and scalability. The reason these are important is the way in which a MSS is deployed in a network.

The following sections describe how some network processing manufacturers can address these issues.

QoS/CoS
The QoS and class of service (CoS) are important for the following reasons:


• Some networking technologies such as ATM and FR are designed for providing the QoS capability. A MSS platform must be able to support these "legacy" services with QoS.


• The QoS capability allows a service provider to offer premium services to business users who pay the premium price for the guaranteed service. This value added service capability is critical for the service provider to maximize their equipment investment.


• At the edge of a network, the bandwidth is still a valuable commodity because it aggregates access traffic in many cases over legacy transmission infrastructure.


IP QoS has been a hotly debated topic for a long time. Currently, few argue if it is needed, but rather how it should be implemented. One thing is clear: Implementing QoS requires the fundamental architectural support of data path silicon devices. It involves many complex logic functions such as queuing, buffer management, and scheduling. Implementing QoS functions such as managing buffers and queues has been under-estimated by many vendors.

A general-purpose-based network processor architecture may improve the processing power by using the most advanced silicon processing technology and increasing the operation clock speed. However, the performance of a traffic management function has more to do with the memory technology. Simply throwing in more reduced instruction set cores (RISC) and winding up the clock speed cannot resolve this funda-mental architectural challenge.

Contrary to the "sea of cores" approach, network processor designers can provide a number of purposely designed building blocks for specific functions. The specialized hardware assists the operation of programmable engines wherever it makes sense to provide guaranteed performance. Moreover, a high level language, called field programmable language, is designed for networking protocol processing and is directly mapped to the processor engines' implementation.

Due to technology limitations, system engineers of early generations of switch router designs could only implement some levels of QoS as line card functions and often had a simple fabric with little or no QoS capability. For example, a simple crossbar was the choice of the fabric. The lack of QoS in the fabric was partly compensated for by the capacity "over-engineering." This was acceptable when the system capacity was small, or the QoS was not a critical system requirement, or dedicated hardware was designed for a specific application such as ATM switching. However, this approach is no longer adequate. A MSS must be able to perform QoS for all types of traffic. Simple over-engineering does not make the product cost competitive. System engineers are looking for a solution that will provide consistent QoS features across the data path.

Reliability
When a MSS platform is deployed in a carrier network, it must meet the carrier's stringent reliability requirements. A minute of downtime is directly translated to a big revenue loss. Severe downtime means loss of business. Achieving carrier-class reliability is not just about avoiding the single point of hardware failure. It requires a consistent approach in both hardware design and software design. Agere's solution has some unique advantages of achieving a very high degree of system reliability.

To achieve higher levels of reliability, network processor designs can use FPL language, which is specifically designed for processing data packets with various network protocols. This results in very efficient data path codes. Compared with other high-level languages such as C, the FPL language has an average code density of 20-to-1 for the networking applications. Fewer lines of code mean much less chance of logic errors, meaning improvement in code reliability. The C language can never match this inherent advantage of the FPL language no matter how efficient the C compiler becomes.

Often, line card protection is needed in a MSS platform. For example, an uplink port may require 1+1 equipment protection. Whenever the port switch-over is triggered, it should be avoided to update all ingress line card forwarding tables. Finally, a system can only be as strong as its weakest link. If a data path silicon device is full of defects, the system can hardly be considered as highly reliable. To this end, a verification team and the validation team are an integral part of the development programs that are governed by vigorous program management procedures and review processes.

A field programmable gate array (FPGA) platform is frequently built for a network processor before the silicon. This FPGA platform runs the same design code and is capable of testing millions of packets in the data path. Design errors can be identified before the silicon mask order, which significantly increases the quality of the real silicon.

Scalability
The scalability requirement influences both the architectural design and physical design of a MSS. The goal of system scalability is to increase the system capacity without a fork-lift upgrade. One possible way of scaling up the system is to upgrade the switch fabric depending on the customers' capacity requirement.

In some applications, the system upgrade can be achieved by swapping a new fabric with a higher capacity fabric. This approach requires system engineers to design a common backplane. Sufficient backplane traces are laid out for the upgrade. The SERDES (serializer/deserializer) based serial backplane helps this upgrade option because all connections are point-to-point.

There are many system engineering issues that must be considered to implement a multi-shelf system.


• The integrated SERDES has the industry standard CML (current mode logic) interface signals. This allows the direct connection to Infiniband cables or commercial optical transceivers.


• The SERDES has integrated clock data recovery (CDR). The clock forwarding is in-band.


• The clock reference distribution must be considered for the multi-shelf configuration.


• The delay skew across a bundle of multiple SERDES links must also be considered. A stringent skew tolerance makes the design very difficult.


• The switch fabric is implemented in multiple shelves, the synchronization of multiple fabric shelves can also impose severe engineering difficulty.


• It's important that OEMs seek three key features in their next-generation multi-service systems:


• A common platform for a variety of system configurations to reduce the development cost;


• Seamless upgrade path for adding different services via new software download; and


• Scalability for future bandwidth expansion.


Common Platform Approach
A clear market trend is the move towards common platforms. This approach provides two significant benefits. First, the system OEMs can reduce their development cost. It is typical to take 12 to 18 months for a system OEM to roll-out a new platform. This requires many years of effort in the system, hardware and software designs. With the resource limitation common to many OEMs, different platform designs means significant resource burden and often longer time-to-market.

Second, service providers can protect their equipment investment and reduce the operational cost. There is no need to have multiple racks for different services in the central office. When new service is identified and designed, the service provider does not need to fork-lift their equipment. Instead, the new service can be introduced via the software download. It protects the equipment investment and allows a controlled introduction of the new service.

Service Upgrade Ability
Traditionally, many functions were implemented in hardware to achieve the required performance. The software approach was only possible for products with small capacity. The advent of the network processor technology breaks the performance barrier and makes the software implementation of the wire-speed data path functions a reality. With the software implementation, new services can be introduced via software download. The service upgrade can be either for a new SLA or a new protocol. There is no need to change the hardware and no service disruption.

Customer Support
It is a complex process to design a MSS platform. It involves many and often complicated engineering tasks. Technical support is an important issue for system OEMs. Having the best silicon parts are only half of the story.

Many of today's silicon products are true SoC products. It is common to find a part that integrates millions of logical gates and millions of bits of memory. Without adequate customer technical support from the vendor, a system OEM risks a long development time, or even failure. It is natural that system engineers are focusing on the technical issues. However, nowadays it has become common for system engineers to take into account the customer support and commercial issues when they select a solution.

System Requirement parameters

System Capacity
- Number of line cards
- Port density and type of line cards
- Typical and maximum number of line cards
- Scalability

Physical Design
- Backplane technology
- Power distribution
- Broad space and power dissipation
- EMC compliance

Protocol Types
- ATM
- FR
- IP routing
- Ethernet
- VPN
- Circuit Emulation

Reliability
- System protection scheme
- Hardware fall over criteria
- Software download

Service Capability
- Service creation
- QoS and Service Level Agreement enforcement
- Billing

OAMP (operations, administration, maintenance, and provisioning)
- Fault diagnostic and isolation
- Performance monitoring
- Network management
- In-service upgrade and maintenance