System of the Year

By the end of May, Charter Communications had turned up about 75,000 set-tops to All Digital within the Madison, or Southern Wisconsin, system. With All Digital now expanding from its launch on March 15 into the other six key market areas (KMAs) of the Great Lakes Division, was it time to take a breather? Not yet. With some 425,000 homes passed and a service that goes head-to-head against satellite, there are miles to go before the All Digital train sleeps. And knocking off in early June would have left some 30 percent of homes within the Madison KMA yet without access to Charter’s telephone offering. "Since the day I hit the ground it’s been really moving," Chris Kennedy, director of operations for Southern Wisconsin, says. "It’s got to the point where if it slows down, I’m not going to know what to do." Formerly with Altrio Communications, the Los Angeles-based "overbuild" engineered by cable technology heavyweight Dave Large, Kennedy joined Charter Madison in March 2004. "Coming here was all about the new product launches, where Charter was headed: telephony, VOD, DVR, and All Digital," Kennedy says. "To me this is kind of like an entrepreneurial small competitive shop." Those launches, along with the technical excellence and teamwork they both presuppose and require, are also the reasons we have designated Charter Madison as Communications Technology 2005 System of the Year. The All-Digital story How have these kinds of intense technical efforts paid off? For consumers, there are more choices. "We have all (the products) that are available to the industry today," Dan Murphy, vice president of engineering for Great Lakes, says. And for the Charter team, there is a renewed confidence well suited to an increasingly competitive era. "We own the platform that will drive the future of this business," Steve Apodaca, interim senior vice president and vice president of sales and marketing for Great Lakes, says. The Madison story is in large part about the expansion of that platform. That story begins in January 2004, with Charter’s announcement that it had activated all-digital video in Long Beach, Calif. Charter wasn’t alone in exploring the simultaneous transmission of both analog and fully digitized channel lineups (`simultrans’) as a way to improve picture quality and, ultimately, reclaim analog spectrum. Indeed, much of the industry now has swung into action. But Long Beach may have helped set that trend. It certainly gave Charter a way to map the path forward. "When we went to Long Beach, we wanted to understand the process first, especially the DPI (digital program insertion) side," Pragash Pillai, Charter vice president, advanced engineering, for digital video, says. Once convinced that DPI could scale across multiple zones, the Charter team next had to pick another market. Why Madison? "The first thing…is the people there," Pillai explains. "They have a very solid operation." The second reason concerns the division’s infrastructure. "We own pretty much all the fiber in that state," Pillai continues. And that raises a key distinction between Long Beach and Madison. IP Multicast While the C-COR (then nCUBE) ad servers, Terayon splicing gear and Harmonic encoders remained constant, the discontinuities were significant. Long Beach involved one headend, one ad zone and distance-limited asynchronous serial interface (ASI) transport; the ongoing Madison/Great Lakes project entails a master headend, six remote master headends and (significant amounts of) IP multicast across the Cisco-powered Wisconsin backbone. "We’re between 400 and 500 Mbps of IP multicast," Sean Hayes, director of technology for the Great Lakes Division, says. That’s a lot of traffic, which speaks to the value of owning fiber, as well as IP talent that understands what he calls the "backwards" nature of multicasting. Whereas traditional routing is source/destination, in multicast the remote client asks to join a session, a technical reversal that turns all-digital into a serious, complex brew. "It’s not for light-hearted IP transport people," says Hayes, whose pre-cable background includes nearly six years as a test engineer at supercomputer giant Cray Research. Audio engineering was another challenge that Madison faced prior to launching its first cluster of customers in March. The last piece—turning up the customers—entailed pointing a set-top box toward the "simultrans" channel map within the Motorola Digital Addressable Controller (DAC). Only that’s easier said that done, especially given Charter’s use of higher order modulation for HD, VOD and digital simulcast. "All the flaws of in-home wiring, when you start launching 256 QAM, show up," Murphy says. "All-digital will force you to address the in-home wiring issue." Cutting-edge telephony Telephony supposedly does the same. As it happens, Madison was raising the tide of network performance by launching VoIP last fall, along with an EQA (electronic quality assurance) Live initiative. It had begun driving a standard in-home wiring architecture in early 2004. But the relationship between Madison and telephony goes farther than that. Late last year, for instance, Charter used the state to determine whether it’s possible to launch multiple markets off a single server (in St. Louis) and then later, as a market matures, re-point a gateway (in Stephens Point, Wis.,) to another subsequently deployed server (in Madison). That test was successful, thus enabling capital expenditure to map more closely with revenue. Charter’s earlier and historic VoIP launch in Wasau, Wis., is another reason to associate this state with telephony. "The Wisconsin launch was the first, primary-line VoIP launch in North America," Mark Barber, Charter vice president, telephony, says. Barber singles out talent in Madison, especially Director of Telephony Engineering Mike Cornelius, for driving this project from its inception. "It was in September 2002. So they were really on the cutting edge," Barber says. "There wasn’t anybody to learn from at that point." Great Lakes VP Murphy also credits Cornelius, Barber and the associated vendors (ARRIS and Nortel) for their determination to sustain the project during subsequent uncertainty within Charter itself. Credit also belongs to the organization’s larger technical team, as Barber explains: "One of the reasons we chose Wisconsin for our early trials, and one of the conditions that continues to persist in Wisconsin and in a lot of our Great Lakes markets in general, is that they had some of the best plant performance, low service-call rates, in the country for Charter." Opening up VOD Madison has also been the site of another implementation of Charter’s migration to an open VOD architecture, where it mixes and matches technologies from vendors such as Concurrent, C-COR, Kasenna and N2 Broadband. Other sites include Allendale, Mich., and Rochester, Minn. The deployment parallels the "end-state call server" strategy for telephony. "Basically, we have one (deployment) per state, and each N2 backoffice will run the entire state," Great Lakes Director of Technology Hayes says. Charter also uses N2 on the client side in Great Lakes. "The beauty of N2 is the speed. Other than stability, which is a given, their speed on the…set-top box is just smoking," Hayes says. In addition to such best-of-breed outcomes, VOD interoperability has helped drive the per-stream costs drop to one third of what they were when Charter launched two years ago. As for where to place the servers, Hayes says a simple math equation puts their system, with hubs placed 40 to 60 miles apart, in the distributed camp. Product lines and networks The flurry of deployments in Madison has made life interesting for Charter’s broadband technicians (BBTs). Take telephony. "There was a fear element at first," VP Kennedy says. But once that’s overcome with solid understanding, "there’s a great feeling of empowerment." Today’s technologies also have slowed the pace but expanded the focus of a service technician. "Back in the 450 (Mhz) forward-only days, you could have twelve to fifteen service calls a day and whack them out pretty quickly," Kennedy says. "Today, you’re running eight calls and you’re really qualifying an entire house to deliver seven product lines across five different networks." Count them. That’s seven product lines (even more products, including commercial class services) across five networks. You might be tempted to call it a Full Service Network. But that phrase has already been taken. So we’ll just call the infrastructure and the skilled technical teams that are running it our System of the Year. Sidebar 2005 Top-Tier Systems Adelphia West Palm Beach: Courage in Motion Given the speed and intensity with which Adelphia Communications ran its technical operations over the past year, there was reason enough to nominate practically any of its systems for special recognition. Adelphia West Palm Beach was among those systems in busy catch-up mode. By August, the technical team had upgraded some 1400 miles (in only six months), deployed HD content and DVRs, made VOD available to 50 percent of its homes passed, and expanded and improved its high speed Internet service. "We were running with fervor," says Len Falter, Adelphia VP for area operations. "There was definitely a great deal of urgency." Then, over the Labor Day weekend, Hurricane Frances made landfall at Sewall’s Point, Martin County, hitting one of the four systems in the West Palm Beach cluster. The storm then slowed down to a crawl. "It sat and dumped rain," Falter says. The downpour resulted in the "unique experience of not only aerial plant coming down, but underground plant literally being ripped out of the ground," he says. Power outages in some systems lasted for nine days. "We did everything we could to creatively restore services," he says. Three weeks later, as Adelphia reached an 82 percent restored status, disaster hit again in the form of Hurricane Jeanne, which made landfall again in Martin County, within about a mile of where Frances hit. Clocking winds of 115 mph, Jeanne snapped poles and toppled a 400-foot tower at a headend facility. An initial assessment put Adelphia’s network availability at 16 percent. "It was another punch on the nose," Falter says. "But the team got busy, rolled up their sleeves." By Thanksgiving, West Palm Beach was 98 percent, business as usual. Lessons learned? "A good disaster recover plan is crucial for any operations," Falter says. "(But) you have to continually revise it." (For more on disaster recovery, see related article "Disaster and Your NOC.") While not the only system impacted by the severe weather last year, West Palm Beach received more than its fair share. It handled these challenges with the same grace and tenacity exhibited in its fast-paced technical deployments, for which we name it a 2005 Top Tier System. Time Warner Cable Oceanic: Innovation and Self-Reliance Time Warner Cable Oceanic is notoriously innovative. Less well known is the Hawaii-based system’s excellence in technical operations. Oceanic President Nate Smith believes the two are linked. "The key there is self-reliance," he says. "(We) are so far away from everything. I think that’s what allowed us to be so innovative. You can’t rely on anyone to arrive and fix things. Whatever you do you have to know you can solve it." Oceanic does seem to thrive when given the right kind of tools, such as C-COR’s network management system (NMS), which requires an operator to get a considerable database in order. A "field-hardened" technical operations team with an average tenure of 15-18 years also correlates with a high level of system intelligence, Smith says. Managing its own billing system likewise has enabled it to interface rapidly with innovative partners, such as Navic Systems, with whom it launched the legendary pizza-over-the-television application. "This year we’ve expanded that and have an entire food court," Smith says. How does it work? "Mice under the hood," Smith says. "We generate faxes." Innovation can be wrapped in old or new technology. The key is that it works, is flexible and creates a distinct and desirable offering, such as another Navic application that allows consumers to pay and view the cable bill on line. "Our whole bent around here is to figure out a proposition that the consumer will value," Smith says. A year ago, Oceanic also launched the Game Show Network’s first TV-only application, powered by software and technology from GoldPocket Interactive. Oceanic also has worked with NDS to create an application that gives viewers the ability to select different camera angles for sporting events. "If our ESPN doesn’t do anything different from our satellite competitors then it’s just about price," Smith says. On-demand and non-linear programming are differentiators. "That’s something that we can do and the others can’t." Cable’s tinkering ethos may wither under pressure from telephony-induced methods of procedure (MOPs) but Oceanic wins a Top-Tier designation for reminding us that innovation and self-reliance are virtues, too.