Comcast: Operator of the Year

Last November was the first-year anniversary of Comcast’s acquisition of AT&T Broadband. In San Ramon, Calif., Comcast celebrated by highlighting products and services, such as high-definition TV (HDTV) now available over its upgraded plant. Comcast also announced that it had accelerated the previous operator’s California upgrade schedule from five years to two. In Texas, New England and elsewhere, the stories were similar: Comcast was making the AT&T desert bloom—and quickly. In February, Comcast corporate announced that in 2003 it had completed the upgrade of more than 53,000 miles of cable plant. It originally had described these upgrades as a two-year, 60,000-mile project. As an example of the kind of achievement we look for in our operator of the year award, this project was tough to surpass. And while we noted feats of engineering excellence and audacious turnarounds at other MSOs, Comcast made at least two further claims upon our attention: Breakthrough advances in its digital video and high-speed data platforms, and effective leadership regarding the future of technology for itself and the industry at large.
Comcast wins extra praise for those advanced services and leadership contributions, but the big points come from its successes in San Ramon and Irving, Texas, and the countless other sites where it effectively upgraded and integrated the AT&T properties. It was a stunning display of project management and technical fundamentals that proved difficult to discount. More experience than luck Not that we didn’t try. There is the matter of timing, for instance. Being the only major MSO engaged in upgrades on a massive scale no doubt helped Comcast stay ahead of schedule and under budget. Three years earlier, any such effort would have involved bidding wars on construction crews and global hunts for fiber. Even with plenty of labor and material on hand, however, there are many ways for upgrades to go wrong. The cable industry’s reputation for on-the-spot fixes, for instance, led some to wonder how much quality went into this project. But the more one learns of the systematic process this effort entailed, the clearer it becomes that Comcast knew what it was doing, and got it right. One reality check comes from the full-year report, which shows that compared to a pro-forma loss of 415,000 basic subs in 2002, Comcast added 140,000 basic subs in 2003. The overall upgrades, including the designation of some five-and-a-half to six million homes as “return-certified,” truly looks to be one for the record books. “The pride that’s going on around here in pulling it off is just unbelievable,” John Donahue, Comcast senior vice president of engineering, says. And a better explantion of Comcast’s success than luck is its long experience with this process, combined with a pragmatic model and effective leadership. Comcast actively had been upgrading systems since about 1993, with volumes picking up in 1998 as its pace of acquisitions quickened. “We’re getting a lot of visibility for 2003, but that wasn’t our first year of doing it,” Donahue says. “The infrastructure was already in place, and from day one, it was built for speed.” As it happened, Comcast completed the upgrade of most of its “classic” systems at about the same time (November 2002) that the AT&T deal closed. It was as though the upgrade teams took a breath, then started all over again. “Going in and dealing with the integration was not much more than just expanding what we were currently doing,” Donahue says. Comcast’s upgrade process is a function of its decentralized structure and lessons that it has learned along the way. Now organized into five divisions, Comcast began the AT&T upgrades with six divisions and a corresponding number of managers in the field responsible for plant miles in their respective systems. Divisions call the shots Donahue underscores the role of those six divisional managers: “There isn’t an army here calling all the shots.” Vice President of Operations Chris Coffey, who acted as the hub of all divisional upgrade activity on Donahue’s team, confirms that point. “Even though from our address and org chart it looks like we would be more on top, we really play more of a support role for the five divisions in all the projects,” he says. Coffey also singles out Senior Director of Construction Dave Lorenzi, who reports to him in Philadelphia but is “exclusively responsible for the upgrades and had probably 100 percent of his time over the past year-and-a-half dedicated to the former AT&T systems.” The Comcast system also helped the divisional upgrade leaders focus exclusively on their assigned 12,000 or 20,000 miles of upgrade. “Your typical construction and engineering guy…has a pretty full plate,” Coffey explains. What Comcast did was make the upgrade teams an adjunct to operations, freeing them from the regular repair and maintenance work and dispatch of installers and service technicians. In addition to insulating the upgrade teams, Comcast further streamlined its activity by pooling common back-office processes related to such things as detailed tracking and accounting. Purchasing additionally was simplified by reducing the number of approved vendors.
“It’s been our experience over the years that the fewer participants, whether labor or materials, the better.” Coffey says. “Less to control, less confusion, more focus, much more control.” The upshot was project managers were freed to concentrate on meeting Comcast’s production and quality standards. And although some analysts persist in imagining that cable operators use the proverbial baling wire and duct tape to build and maintain their networks, those involved in the process have another view. “We have a standard practice and a procedure not only for the physical elements of the upgrade, but the process to which we then verify that the network is complete,” Coffey says. And speaking in early 2003 at the start of this upgrade process, one contractor confirmed Comcast’s emphasis on internal controls, “Documentation is key,” he said, stressing the need for standard procedures at “every possible aspect of the project, from inventory to sweeping to certification.” A critical 10 percent These standard procedures are extensive, but only cover 90 percent of the process, Coffey says. What remains, and where the divisional upgrade leaders really earn their pay, is with the critical 10 percent of an upgrade that depends on local circumstances. The necessarily disruptive nature of upgrades, for instance, puts a premium on leaders with good political instincts. “You alienate a local permitting authority for just some oversight—maybe you don’t have a personal relationship with them—and they have the authority to slow the work down, or stop it,” Coffey explains. California is noteworthy in this regard. It seems foolish to have neglected the cable plant in Silicon Valley, of all places, but there evidently were reasons why AT&T had failed to upgrade it. “Some of them were operational, some were local issues, franchising agreements,” Coffey says. An underlying regulatory factor is that much of the cable runs underground in California, which makes an upgrade all the more costly and disruptive to a local community. Comcast nonetheless succeeded where AT&T had failed. What made an impact in California was getting the right person—Ray Clarke—into the position of regional director of upgrade, Coffey says. (See sidebar on page 34 for more on the Comcast upgrade team.) The “just-do-it” model Apart from effective procedures and sharp leaders in the field, another reason for Comcast’s success is a corporate culture more pragmatic and less academic than that of its predecessor.
“At AT&T, we studied fiber to the home, passive coax, (and) ring structures,” David Fellows, Comcast CTO says, noting that other architectures were under serious consideration, as well. But he contends that studious attachment to pet theories makes minds less open to clear answers of how to move forward. Say a 500- or 600-home pocket is faster to deploy, costs less than and provides every service that a fiber to a 75-home node could carry, Fellows suggests. The obvious answer is to forget fiber-deep and go with the higher number of homes: “Just do it. No more debating, no more thinking, no more arguing. No more senior management coming in with their twist to it. Just do it,” he says. The point is not that Comcast is building to a 500-homes per node model, but rather that its business-like culture encourages meeting real demand. Its model is more of a dynamic, even messy, equation than an elegant architecture. “We had our end-of-the-line design guidelines. And we built to that, and whatever homes that passed, that became the node size,” Donahue says. “That’s the most economical way of building a system, so you don’t waste signal. It’s very difficult to portray the size of the node ahead of time, because the buy rates are going to dictate what the size of the node actually needs to be at the end of the day.” While Comcast has an 1,800 homes-per-node upper limit, in practice its densities rarely allow it to approach that point. Instead, densities have tended toward an average of around 860, Donahue says. But, again, there’s no single target: “With the interactive services, some nodes are going to need to be at 50 homes. And some nodes can operate just fine at 1,000 homes.” Fiber and spectrum are more constant figures. While saying the fiber count “depends on the node,” Donahue adds that as the project progressed most nodes were at 12 fibers, as Comcast benefited from the economies of a standard count. On the question of spectrum, or bandwidth, Coffey is emphatic: “Almost exclusively, they’re 860 MHz. There’s hardly any upgrade that was not.” For whatever reason—perhaps it’s the Adelphia snakebite—some financial analysts wonder if that’s really the case, and what percentage of plant still lags behind. There is a place for skepticism. But when taking down 450 MHz gear, it makes little sense for an operator benefiting from huge economies of scale to install anything less than 860 MHz. One might ask instead why they did not push the envelope further to 1 GHz, as Time Warner has begun to do, if the answer were not obvious. Obvious, because a just-do-it philosophy entails going with the best possible, and in early 2003, that meant 860 MHz. The winning vision By themselves, pragmatic models and effective procedures were necessary but insufficient means of winning the upgrade battle. It took something else again to pull this off. “Far and away what made it work is the people that were involved,” Donahue says. “Not only were they talented, but—and this is Comcast culture, and we took advantage of it—there was just an overwhelming drive to succeed.” One way of harnessing that culture to bring aboard the thousands of contractual employees in the field was simply to offer everyone the big picture. “From Brian (Roberts) on down, from day one,” Donahue says, “(we) explained why we were doing these upgrades. So that it was not just to go out and upgrade, but so that we had the platform that is capable of delivering all the products and services that people want.” The upgrade teams were only a step removed from those products and services. A rolling spreadsheet bumped nodes from the upgrade to deploy columns. This added a kind of positive loop-back to the upgrades, with the new video and data products generating additional revenue. “The fact that we got so far ahead of schedule last year did have a positive impact on our business metrics, because for every home you’re ahead of schedule, it’s a home that you can provide advanced services to,” Coffey says. Advanced services: Digital video The data indeed show that across all systems, advanced services were growing propositions. The full-year report from February shows that in 2003, Comcast added more than 1 million digital video subscribers and nearly 1.7 million high-speed Internet subscribers. Moreover, in both categories, Comcast has engineered breakthrough products that have not only won over customers and but also have become points of reference for the industry at large. The expansion of Comcast’s video-on-demand platform is a case in point. For much of 2001 and 2002, Comcast—along with other MSOs—had spent considerable effort on integrating its on-demand technology. The ambitious goal was for its servers, guides and billing systems to work as a unified platform. Meeting that objective was no mean achievement, but far from an occasion for standing still. Comcast quickly refined and opened up the transport and management components of its on-demand platform, which facilitated the early 2003 launch and subsequent expansion of its flagship “Phillyvision” service, whose trademark has been an exponential growth in content.
“We’ve moved, from a transport standpoint, to an all GigE system,” Mark Hess, Comcast vice president of digital services, says. “We’ve spent a lot of time with people like N2Broadband and IMAKE on asset management and content propagation. We’re moving a whole lot of content to the network now. We’re even now utilizing the Comcast Media Center, Gary Travers’ whole team, in terms of being able to deliver content directly from our own uplink sites.” The “Phillyvision” platform originally entailed 1,100 hours of on-demand programming, most of it free content. “We are well beyond 1,000 hours at this point, Hess says. “I think every market has or will have at least 2,500 moving to 4,000.” True to Comcast’s “federal” structure, individual systems will use the Philadelphia experience as a model, creating new templates such as “Bostonvision” in the process. The same applies to the basic on-demand platform itself, which now exists in more than half of all Comcast systems. “We don’t go and do this for them and leave. They do it themselves,” Hess explains. Changing behavior Going forward, Hess says that on-demand will be available in 80 percent of Comcast systems by year-end, and that content will continue to be a major challenge at headquarters. “Where to store it, how to move it, where it comes from, how it’s managed, how it’s propagated, especially when you’re…trying to be as real-time as possible,” Hess says. Related to those questions is how to improve the customer experience. In a development overshadowed by the simultaneous February 11 bid for Disney, Comcast announced a joint venture with Gemstar TV Guide that makes the MSO the managing partner of the guide and user interface (UI) development. Gemstar, in turn, sells the guide and markets the data to the wider industry. While more UI development is in store, Comcast currently is on the verge of deploying the so-called I-Guide, whose easier click-throughs, sub-menus and visual art is aimed at helping the MSO harvest what Hess calls the “low-hanging fruit.” Whatever questions are raised regarding the on-demand business model, it is clear that Phillyvision is changing customer behavior. In January, Comcast logged 7.9 million orders for individual streams in Philadelphia, up 21 percent from the prior month. The most recent data shows average usage up to 16.2 times a month. “The best part about this is we are in the midst of changing the way people watch TV,” Hess says, “and they are enjoying it.” And the industry is well aware of what is happening in Philadelphia. “(National Hockey League) Flyers on demand, local content on demand, children’s shows. And on top of that broadcast on demand,” Yvette Gordon, vice president of strategic planning at Seachange International, says. “They’ve moved far beyond S-(subscription) VOD,” she says. High-speed Internet, voice As with VOD, Comcast has both unleashed and refined its cable modem service. Like other MSOs, Comcast accelerated its high-speed data service; specifically, doubling it in all markets from 1.5 Mbps to 3.0 Mbps, without increasing prices. In addition to its higher data rates, Comcast further differentiated its service through a user-friendly and award-winning portal launched in November. In the 39th annual Society of Publication Designers awards competition, Comcast’s broadband multimedia player, toolbar and daily interactive guide helped it win the gold medal award in the new media category. Attention to both speed and content appears to be working: As of the first quarter, 2004, Comcast was serving an unprecedented 5.7 million data subscribers. Comcast’s launch of home networking and its exclusive purchases of DOCSIS 2.0 modems ever since that specification became a standard is further evidence of Comcast’s intention to layer voice and additional multimedia services across its broadband platform. As for voice over IP (VoIP), Comcast focused its efforts over the past year with trials in Coatesville, Penn., and Philadelphia. The circuit-switched telephony business it inherited from AT&T continues to lose subscribers, although insiders say that operating margins have improved, in part through back-office efficiencies and the loss of delinquent accounts. Regarding what appears to be cable’s slow start from the VoIP box, industry analyst and gadfly Michael Harris asked in the April issue of his Cable Datacom News whether operators had abandoned their shoot-from-the-hip cable cowboy heritage and were losing the voice opportunity to more nimble players. In response, Comcast’s Fellows notes taking down more than 5 million high-speed data subscribers would be high on the list of items for which he could get fired. “We don’t shoot and ask questions later when it comes to operating the core business,” he says. “On the other hand, we do a lot of shooting in the lab and on paper and modeling,” he continues. And the existence of multiple labs—in Philadelphia, New Jersey, Denver, and elsewhere—leads us to a final point. Technical leadership Some companies lead by example. And Comcast’s commitment to upgrading some of the industry’s laggard systems, as well as to deploying cutting-edge video and data products speaks for itself. There are also the untold hours that Comcast devotes to committee work that keeps the industry’s technical community vibrant. Whether Comcast contributes more time than its MSO brethren is an open question, but from Brian Roberts (in his second term as chairman of the board of CableLabs) to Comcast Vice President of Engineering and SCTE Chairman of the Board Wayne Hall to individual SCTE chapter leaders, Comcast’s record of service is impressive. Another and more elusive area of leadership concerns Comcast’s first-among-equals role in asking critical questions about cable’s future and placing prudent bets on enabling it. Much of this forward thinking originates from Comcast Senior Vice President of Strategic Planning Mark Coblitz, whose role in the first phase of the plug-and-play negotiations we highlighted in our January 2004 issue. It also emanates from the smart folks in Advanced Engineering that Fellows has working for him. Here it is worth mentioning Comcast’s role in the hush-hush, request for information (RFI) regarding next-generation network architecture (NGNA). Insiders say Comcast’s work on this complex series of questions preceded that of the related Comcast, Time Warner Cable and Cox Communications collaborative effort by some six months. But it’s not necessary to break nondisclosure agreements to recognize Comcast’s leadership or its modified cowboy spirit. “In the old days, there were just a couple of suppliers to the industry, and your next big rollout was going to be whatever the next product was on their roadmap,” Fellows says. “Much more now, we are inventing new requirements and procuring the product that meets them.” For instance, with an astonishing 100 million IP addresses under management, Comcast cannot simply copy existing back office systems, but has to invent them itself, he says. Another example is the DOCSIS set-top gateway (DSG) specification. Had Comcast not pushed and prodded, this spec would not be as far along as it is. “We didn’t go around CableLabs’ back. We didn’t go behind the backs of the MSOs. But we felt a little more strongly than our colleagues that we needed to do this right now,” Fellows says. Picture of a thoroughbred Taken together—the turnaround upgrades, the breakthrough services and the confident leadership—Comcast’s achievements over the past year give it the feel of an athletic team closing out a championship season, or a thoroughbred just hitting its winning stride. Nothing in Comcast’s unprecedented size endows its executives with infallibility. The industry’s ongoing journey toward an all-digital, anything-to-anywhere future will be bumpy. There will be disagreements from within (as there is now over optical metro transport) and competition from without. Comcast could stumble. But the evidence is encouraging. It shows that the biggest MSO in the industry has a lot of common sense, knows how to execute complex projects, keeps its eye on the customer and understands leadership. This horse could yet stumble, but the odds of its winning remain high. Jonathan Tombes is executive editor of Communications Technology. Email him at [email protected].