The Zen of VOIP

Operators deploying voice over Internet protocol (VoIP) must address a large number of new and old issues in their networks-many of which can fly under the radar screen and significantly impact successful rollout. It is well understood that solid hybrid fiber/coax (HFC) plant test and maintenance practices are critical to implement, but many of those considered necessary are insufficient. A new level of IP-related expertise is needed to address how the system handles critical issues, such as variable delay, congestion and packet loss. Operators deploying the service are seeing 10-20 percent more truck rolls when compared to high-speed data deployments, an alarming statistic considering the fast-paced rollouts operators are planning. Why is this number so high? Unlike high-speed data service, voice service is a real-time service. A real-time service does not retransmit lost/corrupted packets (user datagram protocol, or UDP, vs. transmission control protocol, or TCP). Plus, almost every problem will be immediately known to two parties (talker/receiver). The customer-quality metric is based on the "traditional" phone system, which is characterized by consistent quality of voice calls. Contrast this to high-speed data service, which is an order of magnitude faster than dialup, where lost packets are ‘hidden’ by retransmits, packet jitter is not an issue and delay is not a significant factor. With VoIP, the industry is challenged with a service where problems are very noticeable and competitive comparisons are unfavorable. It warrants consideration of a new test paradigm, since a "manageable" plant today may not be ready for VoIP service tomorrow. HFC problems

VoIP problems fall into three categories: core HFC, IP network and voice-specific. Packet loss and signal distortion can be caused by the physical layer of the HFC plant. At the home/premises, close attention is paid to the home wiring, drops, connectors, and unterminated or nonoptimal runs. From a network perspective, poorly balanced systems, corroded connectors and faulty components are the main culprits. The fundamental HFC tests (sweep, home wiring tests, return path monitoring, etc.) are more critical than ever. Yet, there is a need to rethink testing in this area based on how well existing equipment and techniques are implemented. For instance, how often is the plant swept, how tight are the limit sets, and how proactive is the return path monitoring? IP network problems

Different problems will surface on the IP network. As indicated in Figure 1 (on page 28), a major portion of the system is considered an IP or digital network; each element can and will contribute to packet loss, jitter and delay. Packet loss and delay will be observed on high-speed data as throughput problems, but customers are not likely to notice unless these become very high (for example, 5 percent, 500 ms). Customer dissatisfaction on a VoIP system occurs with much lower limits (0.5 percent, 120 ms). Jitter will cause few to no problems on high-speed data, but will be a major factor on voice. Focus areas should be on congestion and component configuration. Operators have assumed, mistakenly, that congestion is not an issue, since adding VoIP service on top of high-speed data does not add significant traffic (~17 kbps for G.711 Codec), and the Data Over Cable Service Interface Specification (DOCSIS) PacketCable specification also applies dynamic quality of service (DQoS) to ensure prioritization of VoIP over high-speed data. However, the prioritization is only for the embedded multimedia terminal adapter (EMTA) to and from the cable modem termination system (CMTS), not in the IP network, unless there is separate prioritization architecture in place such as a virtual local area network (VLAN) or multi-protocol label switching (MPLS). Therefore, there is an increase in traffic on a network that could have "hidden congestion" causing queues and buffers to overload, increasing jitter, delay and packet loss. In systems implementing random early detection (RED), the router purposely drops a packet when the queue is filling up to trigger a reduced transmission rate so that the router’s queue buffer does not continuously overflow. This drop results in loss that will lead to "pops" heard by the customer. Troubleshooting these problems is difficult, requiring a level of IP and network element knowledge and specialized IP analyzers that typically reside outside of the operator’s core expertise. Operators may first focus on addressing the HFC plant, but new focus must be placed on optimizing the IP network. Networks must be architected specifically for VoIP operation, with systems having sufficient capacity to avoid congestion (packet loss), reduce changes in routing (jitter), and minimize delay in every element. One commonly overlooked but important IP network design parameter is the "testability" of the system. Without establishing the correct network architecture, technicians can struggle to find access points to test. This can be critical when trying to segment and isolate performance issues in the network and/or verify service performance without taking the network down. Designing test point access (taps, mirrored ports) on the core optical and IP network elements so IP and MPEG analyzers can view data streams saves time when issues occur. Implementing optional real-time control protocol (RTCP) can be a tremendous aid in diagnosing IP and VoIP issues. RTCP contains statistical performance measures on the RTP data flows, such as end-to-end delay. Having this data helps segment delay issues in the plant. In the HFC plant, implementing special provisioning and configuration files for DOCSIS and PacketCable field test equipment can allow enhanced testing on VoIP services. Voice-specific problems

The third problem area is voice specific. PacketCable adds up to fifteen different components, CMTS blades/options, servers, MTAs, etc., on top of DOCSIS. Provisioning is more complicated than high-speed data, and there is signaling from the network out through the public switched telephone network (PSTN), adding additional complexity. In this area, most problems will be encountered at the start of the call-the absence or delay of dial tone, constant ring, etc. Getting a handle on this and managing it initially seems quite complicated, but it is relatively stable once all components are in place and the EMTAs are provisioned properly. A more dynamic entity is the analog portion of the network (see Figure 2) that introduces a number of problems new to today’s operators. An EMTA has elements to compensate for jitter and packet loss. Jitter buffers compensate for packets arriving with variable delay and are sized based on tradeoffs between packet loss and delay. If the jitter buffer is too small, there can be excessive packet loss ("pops," "gaps," garbled voice) or more delay (response, echo). The EMTA also can compensate for packet loss using a technique known as packet loss concealment (PLC). These algorithms "fill in" lost packets. PLC is very effective for masking packet loss problems and should be enabled. The customer (and operator) may not know there are packet loss issues until they become severe, resulting in voice sounds that are "robotic" or interrupted with gaps. It may be optimal to test/monitor the network without PLC to find/fix the packet loss problems before they become too severe. In the PSTN, the network has been optimized for analog service. A network loss plan (NLP) is established with optimal loss, echo and delay throughout the system; not necessarily optimal at the media gateway or soft switch or, for that matter, a PacketCable VoIP network. A low signal will result in low signal-to-noise ratio (S/N), resulting in garbled voice. Echo cancellers also may work improperly as voice activation detection algorithms may not pick up the speech and apply correction. If the gain is too high, clipping may occur, causing distorted signals. To this point, attention to "echo" has been minimal. However, make no mistake: It remains one of the most frequent and difficult problems to solve from an overall network perspective. Echo is caused by an impedance mismatch in the analog portion of the network and becomes noticeable if the delay is significant. During a call, the perception is that the voices are heard immediately. If this is delayed, we hear it as echo. Customers may also report a "tunnel" or "cave" effect in systems with smaller delay. Typically, the root cause is echo. In most systems, echo is caused by mismatches in the far end causing a reflection that is delayed through the system. This is commonly misdiagnosed as twisted-pair wiring issues in the near-end (where the person hearing it is located). Twisted-pair home wiring is not a typical issue, unlike in cable, which is programmed with coaxial RF home wiring. The operator must focus on minimizing delay wherever possible and optimizing echo cancellation around the delay in the system. PSTN echo cancellation systems are optimized around small delays and not for the delay seen on the PacketCable VoIP network (>100 ms). Echo cancellation in media gateways and other system elements must be set up for the longer delay. Low-quality phones compound each of the problems discussed. Cheap cordless phones inherently "pick up" environmental noise, RF interference (RFI), hum and delay in the system. These systems may seem to work acceptably for traditional phone service, but the added problems (on top of those discussed) will result in customer dissatisfaction. Balance, Grasshopper

To summarize, in today’s environment, preparing the HFC plant is only one step. A balanced end-to-end view is needed, including raising the bar on current HFC management practices, increasing competency in IP network analysis and managing overall VoIP service performance. A coordinated system of test tools can help in segmenting issues and managing services across HFC and IP domains. These practices enable network operators to roll out VoIP service quickly with the level of satisfaction that customers expect. References

"Planning for VoIP," White Paper, Figure 9. Codec Defaults, page 17, downloaded from http://www.netiq.com/products/va/witepapers.asp, October 2003

"Supporting Differentiated Service Classes: Active Queue Memory Management," page 5, downloaded from http://www.juniper.net/solutions/literature/white_papers/200021.pdf, January 2004 Doug Franchville is engineering director of Acterna’s CATV Division. Reach him at [email protected].