June 1, 2009
High-Speed Internet Woes
Troubleshooting the Entire Chain
By John Alcock, Charter Communications
The customer premises equipment (CPE) is only one of several bottlenecks where subscribers can see a slowdown in speed. Let's look at the different points of congestion from the subscriber to the Internet and see what we can do to solve some of them.
• CPE issues (covered previously)
• Local wiring from modem to tap
• Ingress on local node combining
• Congestion of subscribers on local downstream or upstream port
• Congestion of data throughput at egress point to the Internet
Depending on the organization, different people will work on different parts of the equation. The key to making this work is to make sure the issues closest to the subscriber are taken care of prior to escalating to the next level.
For example, it does no good to troubleshoot congestion at the egress point if the installation at the subscriber's house is not up to spec. In most organizations, technicians are under the gun to get a specific number of trouble calls done in a specific time. It is to easy to walk into a subscriber's house, tell the subscriber it is a problem with the plant, and head back out the door. This leaves an unsatisfied subscriber, a wasted truck roll, and probably a return truck roll. That is, if the subscriber has not decided to go elsewhere.
This was covered in CT's December 2008 issue, "Avoid High-Speed Data Meltdown," available online at http://www.cable360.net/ct/sections/features/32736.html.
Local wiring from modem to tap
After ruling out CPE issues, the next step is to rule out issues from the tap to the CPE. Each organization has basic standards for installation. If followed correctly, one could probably rule out local wiring. However, in cases of substandard installation practices or perhaps a case where the subscriber has done a self-install, troubleshooting needs to be done.
Rule out the obvious stuff first:
• Are all ports in the house connected to an active device, or have they been terminated?
• How about splitters?
• Are there any hidden splitters in the house or even obvious splitters that are not up to specification?
The biggest trap is the gold-plated splitter bought from the local big box or electronics retailer. Even if the splitter is of tolerable quality, the crimp-on connectors typically sold at retail can ruin the signal. Many technicians have seen the look of pain on a subscriber's face when they cut out the splitter and throw everything in the trash. Lastly, do not forget to check behind every wall plate that has an active device. Chances are there is a crimp-on connector in there, and this will cause a return trouble call.
Another pitfall is amplifiers. Big box retailers sell these amplifiers with what one could consider deceptive marketing. The only thing the subscriber sees is "Boost your speed ... Increase your digital picture quality" on the box. Many of the amplifiers do not even meet plant standards and can actually inject noise back into the plant. Subs who've spent $50 to $100 on a device to improve their speed may be upset when the technician has to take it out. As I stressed in my last article, communication with the subscriber is key. If you don't communicate with the sub, then chances are he'll put the amplifier back in after you leave.
DOCSIS specs for downstream signal level at the modem inputs is -15 dBmV to +15 dBmV. The sweet spot for most modems is around -5 dBmV to +5 dBmV. It can sometimes be very difficult to attain this depending on the devices in the house. Some subs may have a DOCSIS multimedia terminal adapter (MTA), a DOCSIS modem, a set-top box with return, perhaps a digital video recorder (DVR) and several TV sets in the house; with all that going on, it becomes very difficult to get the modem and MTA in the sweet spot.
This is where management of the installation becomes critical. The best option is to bring everything in the house to a common demark point. Manage the link loss when splitting the signal. Take the main splitter coming into the house and split to minimize loss on active DOCSIS devices. One leg off your splitter can go to a second for MTAs or other DOCSIS devices. Take the other leg of your splitter to your set-top box/TV sets. If the signal needs amplification, then amplify only the leg with set-tops and TV sets.
One last thing: Do not forget to look for the hidden splitters in the basement or the attic. A simple check with your meter can determine if there is a higher loss than expected. Perhaps that splitter can be eliminated. If that is not possible, make sure the splitter and the connectors are up to spec. Common sense is the rule. If your meter shows a 5 dB loss or greater from the house block to the modem, then there is probably a splitter, a crimp-on connector, or even worse in line. Climb into the attic or get into the basement and hunt down the culprit.
Upstream has not been discussed much. Assuming the plant was designed correctly and the downstream in the house is at the correct level, then the upstream should fall right in place. DOCSIS for the upstream is +8 dBmV to +55 dBmV for 16-QAM. The upstream needs to be somewhere in the +40 dbmV to +50 dBmV range. Much lower than this and the modem's upstream carrier-to-junk ratio may be compromised. Higher than this and the system could reject the modem, depending upon headend equipment configuration. As with all the ranges mentioned, this is an approximate range. Refer to your local system specifications and make sure the devices are in this range.
After bringing the subscriber's wiring up to specification, the next bottleneck is ingress in the node combining group. In most organizations, the local tech will hand this issue up to either a maintenance tech or even a headend tech. In some of the smaller organizations, this may be the same person. The key again is communication. The tech at the house needs to tell the subscriber that the issue is somewhere in the plant and that he will follow up. If the tech also needs to escalate the issue, he needs to communicate all that he has done. This way, the next level is not trying to troubleshoot anything below.
One more thing on communication: The technician who made the initial visit needs to be the sole contact with the subscriber. This tech needs to be an advocate for the subscriber. It is too easy to just leave the subscriber with a note that says problem is elsewhere and will be fixed later. The tech needs to keep a line of communication open with the subscriber and explain what is being done. This does not need to be very technical. It can be as little as: "We are checking possible plant issues. I will follow up with you when completed to see if the problem has cleared up." This gives the subscriber a good feeling that someone is taking care of the issue.
Ingress is a big issue and can be caused by a multitude of problems. Some of the more common ones are squirrels (yes, they love to chew cable), poor installation, loose setscrews in nodes, broken taps, and subscriber-installed components. The list goes on and on.
The best way to fight ingress is making sure an aggressive leakage prevention program is in effect and that signal leakage is being taken care of. Wherever there is a leak, there is ingress. Assuming that the plant is fulfilling signal leakage requirements defined by the FCC, most technicians have the tools to find and fix leaks. Find the big leak, and many speed issues go away.
From this point forward, the field technician can do very little. It is up to engineering and management staff to take over and properly plan and budget for upgrades. Though an over-abundance of customers is a good problem to have, data capacity is a limited resource, and as more subscribers are added, more capacity is used. In addition to more subscribers, many peer-to-peer (P2P) applications use throughput even when the subs using them are not actively on their computers.
There are several ways to fix congestion issues on downstream and upstream ports. The issue is that almost all of them require capital expense and need to be budgeted and planned well ahead of needing the additional capacity.
The only "freebie" upgrade (and it's not actually free) that can be done to add more throughput to the plant is to change modulation formats and/or channel RF bandwidth. If the plant is fairly clean, this can be relatively painless as well. Most large cable operators have already taken advantage of this; however; some of the smaller operators may still be able to change modulation. By changing to 256-QAM (quadrature amplitude modulation) on the downstream and 16-QAM on the upstream, one can get more headroom on throughput.
The typical small cable operator has downstream modulation set at 64-QAM, which yields a raw data rate of 30.34 Mbps. Switching to 256-QAM changes the raw data rate to 42.88 Mbps.
Smaller organizations probably launched service with quadrature phase shift keying (QPSK) on the upstream. For them, upstream gains are even greater and are listed in Table 1.
Table 1: Bandwidth and throughput comparisons
Ron Hranac wrote a great article on 16-QAM back in CT's January 2003 issue that does more justice on cable modulation. It's available online at http://www.cable360.net/cablefaxmag/technology/strategy/14829.html.
The next step in taking care of congestion of downstream and upstream ports is good old-fashioned node recombining. The key here is proper monitoring of data throughput on a per-port basis and planning ahead. Recombining needs to keep several factors in mind, such as total subscribers and type of traffic on each port. For example, a port with 50 subscribers in a college town probably uses much more data capacity than a port with 200 people in a rural area. By being smart with recombining, different ports can be combined to take advantage of bandwidth. Load balancing may help, too; consult with your CMTS vendor about configuring this.
The last step in port congestion is taking advantage of DOCSIS 3.0 features. DOCSIS 3.0 allows channel bonding on both the downstream and upstream ports. This would allow greater than 100 Mbps on both downstream and upstream direction.
Congestion at the egress point to the Internet
Cable operators need to keep recurring costs for data backhauls under control. In the "dark ages" of the Internet 10 years ago, it was common for a cable operator to purchase multiple DS-1s from the local telco provider to provide needed data rates. In the majority of cases, the operators added subscribers faster than additional capacity could be ordered. This year is not much different in that respect. The only thing now is that cable operators order OC-48s and larger. Still, a lead time of three to six months is not unheard of when ordering circuits, and if additional capacity is not ordered in time, subscriber speeds will suffer.
Data usage habits of subscribers have changed over the past decade as well. Music and huge video files are constantly being transferred. YouTube has a lifetime of video to watch, and user-generated content is everywhere. P2P applications have changed this as well. Upload or return throughput is stressed more with P2P applications such as BitTorent.
Several vendors have products that can shape or rate-limit data and give certain traffic such as P2P applications lower priority. But cable operators can generate bad press when they aggressively curb P2P traffic on their networks and fall short in public relations (PR). Without getting into a discussion of Net Neutrality or PR strategy, one can safely say that all operators have a responsibility to optimize the end-user experience. Allowing better response to interactive traffic such as Web browsing than to non-interactive traffic such as P2P is a good business decision.
Cable is no longer the only high-speed game in town. Most subscribers have a choice. The key to keeping them happy is to make every experience they have with your company a great experience. The call center must do proper troubleshooting to eliminate and fix problems before rolling a truck. The field technician must use common-sense judgment to fix issues at the house and communicate with the subscriber. The maintenance and headend staff must maintain plant integrity and signal quality. Engineering must monitor plant conditions and employ common-sense rules to data capacity usage. Lastly, management must communicate with all parties concerned and properly budget the upgrades and tools required.
John Alcock is a network engineer for Charter Communications.