Peer to Peer
Cable’s data network engineers have done well. Especially those who transitioned off Excite@Home, and since then have added diverse routing, fail-overs, reduced points of presence, internal efficiencies, national backbones and other enhancements.
"Literally nothing but upside," Jay Rolls, vice president of data engineering for Cox Communications, says about the transition. His counterpart at Comcast takes similar pride. "Our call contact rates now are a fraction of what they were even when @Home was running at its best performance," Rick Gaslowli, vice president for online and Comcast IP services (CIPS), says.
Amidst these accomplishments has arisen the nonetheless vexing phenomenon of peer-to-peer applications. Originally a way to promote resource sharing in small networks by sidestepping "control nodes," this family of software has struck discordant notes in larger area networks, ever since Napster’s arrival in May 1999.
Napster died, but from its ashes rose such robust phoenixes as Kazaa, which claimed in May 2003 to have produced the most downloaded software ever. (For more on peer-to-peer’s evolution, see sidebar) Over time, this perplexing category has proven nearly as much a bother to manage as a stretch to describe.
"A rather obese and aggressive fellow on an episode of Hill Street Blues years ago," suggests Michael Harris, president of Kinetic Strategies, by way of analogy. "The restaurant owner had to call the cops in to get him away from the all-you-can-eat salad bar."
The rules changed
The image is amusing but apt. Because from an engineering standpoint, what matters is not the contents or quantity of data piled onto a peer-to-peer application’s "plate"—as obnoxious or illegal as they may be—but rather the app’s sheer persistence. It keeps coming back for more, not simply at mealtime, but around the clock.
And that shift in subscriber behavior has effectively changed the rules of the game, experts say. "The problem is, you built a network with statistical models that didn’t count on that happening," Gerry White, Motorola’s senior director of advanced technology, says.
Tom Cloonan, CTO of Arris Broadband, agrees that the increase in the percentage of time that a user is online has serious consequences. "The business models are not holding up any more," he says.
"Peer-to-peer ranks fairly highly," says Scott Painter, director of engineering at Susquehanna Communications. "This is one that, depending on the solution implemented, can dramatically cut costs or increase revenue."
Quantifying P2P
Estimates vary on how much bandwidth peer-to-peer applications consume, but none is trivial. "In some cases, they can be up to 50 percent of the traffic on a particular CMTS," Comcast’s Gaslowli says. And that’s low-end, in this random survey.
"I’ve never seen either broadband or cable where it’s less than 50 percent," Marc Morin, CTO of Sandvine, a service control provider, says. Yuval Shahar, president and CEO of P-Cube, another data traffic cop, moves from minimum to average numbers: "We’re seeing consistently 70 percent of IP traffic being peer-to-peer."
Joe Jensen, CTO of Toledo, Ohio-based Buckeye Cablesystem slices the numbers from a monthly subscriber usage report. The distribution is extraordinarily skewed: The top 5 percent used 24 billion bytes; the next 15 percent used less than 5 billion; the next 50 percent used about 1 billion, and usage by the final 30 percent was not even measurable.
Thus, in this variation on the 20/80 Pareto principle, only 5 percent of Buckeye’s subs accounted for 80 percent of the monthly byte total. "There are people who really pound the network," Jensen says.
The return of Jaba the Hut
Up until a year ago, it had been relatively easy to identify the peer-to-peer apps because of the distinctive field port in their packet headers. Reducing, if not eliminating, traffic was simply a matter of configuring edge routers or aggregation devices to throttle down a known port.
"You would put a rate limit on well-known TCP (transmission control protocol) or UDP (user datagram protocol) ports," Varun Nagaraj, executive vice president of marketing and product management for Ellacoya Networks, says. "They were basically writing ACLs (access control lists) on their big Cisco routers."
But then an updated version of Kazaa Media Desktop appeared.
"Starting in September, they went to dynamic ports," Sandvine’s Morin says. "For one and a half to two months, the perception was that the problem was going away. (But) it was going into a bucket called ‘other.’ It actually got worse."
In other words, Jaba the Hut continued frequenting the salad bar, even more often, only now he looked like a normal patron. Operators wanting to distinguish peer-to-peer from honest-to-goodness port 80 Web traffic—and do something about it—had to call in the deep-packet sniffers.
Sniff deeply
Tracking this disguised traffic means "look(ing) for markings that are deep in the packet, what you’d call application signatures," Ellacoya’s Nagaraj says.
Enter the external devices with dedicated processors and Layer 4 through Layer 7 functionality. "The problem is that deep-packet inspection isn’t something that Layer 2 and 3 devices do, because if they started looking that deep into the packet, they wouldn’t be able to route or switch," Nagaraj says.
P-Cube’s Shahar uses similar terms. "At Layer 3 and 4, you can just look at packet headers," he says. "In our world, that’s where we start."
Operators are already testing and using these technologies. "Last year we put the Ellacoya system in," Buckeye’s Jensen says. "That has given us a window into what’s going on. That’s been very helpful for us in … managing our network."
Allot CEO PG Narayana says he’s sold "around 20-something" boxes to cable operators, with several international and second-tier MSO trials ongoing. P-Cube also has some international wins. Sandvine counts Rogers and Bell Canada as customers, and has a dozen trials elsewhere.
Another deep-packet sniffer, Packeteer, is helping school districts comply with the Childhood Internet Protection Act (CIPA), which is aimed at keeping pornography and other illicit materials away from young students. Doing so entails identifying applications by means other than ports.
"If you don’t, (students) can and do bypass Internet filters using peer-to-peer, file-sharing programs like Kazaa, AudioGalaxy and Morpheus," said Roger Courtney, an administrator for the Birdville, Texas, Independent School District, in a statement lauding Packeteer.
The cost of P2P
Whether these service control providers succeed further with mainstream North American cable operators depends, in part, upon cost factors. Just how much of a financial burden does peer-to-peer impose?
Some costs are indirect. With only 20,000 data subs, Buckeye nonetheless has dedicated a full-time employee to handling copyright infringement correspondence. Bandwidth management by postal delivery can be effective, especially when it astounds parents who were otherwise unaware that their teenagers had turned the family computer into a copyright-infringing server and subjected them to privacy or security risks. But do the math, and you see a big administrative tax on the cable industry at large.
Kinetic Strategies’ Harris points to other pressure points. First is the potentially negative impact on customer satisfaction, which could translate into downgrades to dial-up. As for expenses, delivering more bandwidth means more links to the Internet and additional access equipment.
Internet links appear to be the less pressing of those two expenses.
Links and modem groups
At Cox, Rolls says it has handled all of its high-speed data growth by increasing peering links. Whereas last summer, 100 percent of its traffic was passing to the Internet through transit links, by May 2003, that had been reduced to 54 percent.
In other words, Cox substituted nearly half of its paid transit links for peering links, which are bartered or free arrangements. "Always try to do more peering," Rolls, says. "More is better."
Does Rolls see a need for deep-packet inspection at Cox? "It’s a nice thing to do every once in a while," he says. "At the very least, it’s a good thing to know how your network is being used."
One service control software actually tries to make those paid and unpaid links work even harder. "We’re certainly unique in addressing the dramatic inefficiencies of the peer-to-peer network," Sandvine’s Morin says. (See Figure 2)
Because peer-to-peer is oblivious to the physical topology and cost of an operator’s network, a policy redirection that incorporates such information can "tame" this traffic, and provide operators with the "biggest bang-for-the buck," Morin claims.
At the macro level, Comcast also appears to be deflecting peer-to-peer’s blows. "I guess a good side of peer-to-peer, if there is one, is that it’s not a bursty, hits-you-all-at-once kind of traffic," Gaslowli says. "It’s more of a constant stream of traffic, which has allowed us to better plan for additional capacity on our network."
"To date—and I don’t want to sound cocky—but peer-to-peer really hasn’t affected us from a customer responsiveness standpoint," he says. "But we are very aware of trying to make sure that we are deploying equipment in the most economical manner, as well."
Like it or not, peer-to-peer is already impacting the configuration of some systems. "It has actually accelerated our timetable for reducing the size of our cable modem groups," says Susquehanna’s Painter. While deploying new DOCSIS 1.1 CMTS equipment, Susquehanna is moving from 2,000 to 1,000 homes passed per port.
"Without peer-to-peer traffic, we have calculated that we would not have needed to do this size reduction for another 12 to 18 months," he says.
Access network
Painter says his analysis confirms CableLabs’ findings that peer-to-peer runs as high as 50 percent on the upstream. "The average customer using peer-to-peer services consumes four to five times the upstream bandwidth that a normal customer does," he adds.
That may be the crux of the matter. "The real problem is on the upstream, not the backbone," Steve Lee, manager of broadband solutions for Cisco Systems, says.
So what do Cisco and other CMTS vendors propose? Cisco’s thinking is colored by what Lee says was an "exorbitant amount" of time and energy spent with operators in Europe and Asia who were chasing peer-to-peer traffic.
"They tried to throw technology upon technology on it," he says.
Lee says Cisco’s network based application recognition (NBAR) tool already does deep-packet inspection, but that getting to the "end game" may entail a simpler approach, one that is less susceptible to the "arms race" dynamic of competition with peer-to-peer applications.
Instead of trying to digest packets (which may end up encrypted, anyway) Cisco proposes another way with what it calls subscriber traffic management (STM): find the heavy users, allow them to run full rate, but put a meter on them.
"The tool is a set of rules that let a sub go over the speed limit," Lee says. "But when I rate him back down, if I notice that he shuts (the app) off, then after an hour, I’ll restore him back." Meanwhile, the MSO can refer the sub to its "acceptable uses page," with information on "how you can get out of the penalty box."
Byte caps
Arris also appears to be stepping back from direct engagement with peer-to-peer applications, and is patenting features on its CMTS that would help operators enforce volume-based limits.
Discretion may be the better part of valor. "It’s becoming very much like the virus, anti-virus software battles," Cloonan says. There’s also the question of distinguishing between subs sending legitimate business files, obnoxious spam mail or feature-length movies.
"If you think about it from an operator’s point of view, peer-to-peer is irrelevant," says Motorola’s White. "The problem is a small percentage of users are using more bandwidth than they’re paying for in the statistical model."
Cloonan believes a consensus is emerging on the use of byte caps. Cox initiated a policy last November. A Comcast spokesperson notes that "it’s something that we’re watching." And now Time Warner Cable reportedly has begun testing Road Runner Xtreme, a faster, higher priced cable modem service with speeds up to 3 Mbps downstream and 512 kbps upstream, with monthly download caps raised from 15 gigabytes to 40 gigabytes.
Yet if the reaction that greeted Cox—which levied no new charges—is any indication, PR battles lie ahead for operators going down that road.
"The specter of broadband download caps rears its ugly head again, this time in the form of a dunning letter from Cox Communications," begins a heated article last December in Broadband Business Report (published by Communications Technology parent, Access Intelligence.)
Operators would like to see this liability turn into an asset, their worst data subs become their best ones. But the chagrin of power users who believe that MSOs are changing the rules mid-game could turn this issue into a PR minefield. Solutions that offer the quietest efficiency wins are likely to prevail.
Jonathan Tombes is executive editor of Communications Technology. Email him at [email protected].
Did this article help you? Send comments to [email protected].
Three Generations of Peer-to-Peer
In a paper presented at the the SCTE’s Cable-Tec Expo, Sandvine CTO Marc Morin describes three evolving styles of peer-to-peer. (See Figure 1) In Napster, a central server indexed the files (not the contents) that participants could obtain from each node of the network. This "first-generation" approach was relatively efficient—and easy to shut off, as it happens.
Gnutella is an example of "second-generation" peer-to-peer, in that it uses no central server and democratizes all nodes. This network is nonetheless limited by "time-to-live" (TTL) constraints and results in relatively long search times.
"Third-generation" applications, including FastTrack, Kazaa, Grokster, Groove and updated Gnutella clients, use both central-server and decentralized frameworks, with certain nodes elected "super-nodes," "ultra-peers," or "super-peers." The effect is greater scaling and speed.
Battling Peer-to-Peer Traffic
Operators can gain visibility into and control over bandwidth-intensive, peer-to-peer traffic through packages or devices that have capabilities at Layers 4 through Layer 7 of the OSI stack. Emerging traffic management software on CMTS platforms may help operators to police heavy users’ compliance with byte caps. Setting up such caps moves the discussion away from purely peer-to-peer applications, and lays PR minefields that operators may have difficulty avigating.
Archives
