System of the Year: Comcast Oregon/SW Washington

It’s hard to discount the significance of Project Cavalry, Comcast’s two-year, $1 billion analog reclamation project that began in November 2008. The massive undertaking is a bona fide game changer. In a May 5, 2009 note, Bernstein Research Senior Analyst Craig Moffett called it "the most important capacity expansion project in Cable’s history."

It is likewise difficult to dismiss the role that Comcast Oregon and Southwest Washington (aka Portland) played as the first region to complete Cavalry. It was a job that the region’s leaders had eagerly sought. "We really raised our hand and wanted to go first," Curt Henninger, regional SVP said.

Hundreds of thousands of digital terminal adapters (DTAs) and 300 MHz of re-purposed spectrum later, in June Portland became the first Comcast region to complete the project. But it wasn’t their only accomplishment over the past year. The region also upgraded to DOCSIS 3.0, further enhanced a network surveillance tool, and launched Comcast to Go, to name a few.

Extending along Interstate 5 from Long View, WA to Eugene, OR and encompassing most of the metropolitan Portland and Vancouver footprint and more than 20 municipalities, this region has caught our attention in previous years.

They have a reputation for being the first in with other technologies; their operational metrics repeatedly have won internal praise; they have leveraged an early jump in today’s business services market; and their tech bench is admirably deep. Their role as the lead ride in Project Cavalry both sets them apart this year and confirms those underlying strengths.

Teamwork 101

In fairness, Comcast had reclaimed spectrum pre-Cavalry in Chicago and Detroit. Other systems under consideration for the lead position that went to Portland no doubt could have handled that role. Moreover, all Comcast systems will be expected to complete their own analog reclamations. This is a corporate-wide ride.

But Comcast corporate gave the team headquartered in Beaverton, OR, the chance to prove out the operational template. In learning more about Cavalry and moreover the Portland team, one sees why this was a good choice.

Making this spectrum-harvesting exercise work clearly requires the right kind of technology, especially the low-cost DTA that enables the delivery of video services to legacy analog television sets. But it also calls for a lot of teamwork, both internally and with customers.

"It was really one of those projects that everyone in our entire region had to become involved in," Henninger said.

The payoff to the customer — "tons of bandwidth, the ability to get over 100 HD channels, which we have now, a more robust Internet offering — is what allowed Henninger and his team to rally the organization around the project, including the additional challenges of being first.

The leadership team itself has been together for nearly ten years. It includes Henninger as well as Mike Mason (tech ops), Lars Lofas (marketing), Brad Kaplan (finance) and Kelly Johns (call center). They had assumed these roles when AT&T Broadband (having purchased TCI and merged with MediaOne) decided to address operational issues in Oregon and Washington by putting separate management into Portland, after splitting it from Seattle.

"To say this place was a mess was kind of an understatement," Henninger said. "Our telephone answering stats were very low… We had three installation teams (for voice, video and data). Everything was siloed."

The adversity of those early days was formative. "From the very moments we put our team together," Henninger said, "we collectively agreed to try to operate our region as if we were all general managers first, as though we were all there to support each other."

"We never allowed ourselves to have walls built up," Henninger said.

Vice President Technical Operations Mike Mason, known as a roving "fix-it guy" in the TCI days, led 750 MHz and 860 MHz rebuilds across the region’s "Heinz 57"-style assortment of technologies. That work was largely completed by the time Comcast acquired AT&T Broadband in 2003.

Henninger credits Mason with creating a customer-focused culture and a single one cohesive winning team. Known for his focus on process, Mason himself said his management style was light.

"You give (your employees) a little bit of direction and the latitude to make the decisions they need to without trying to second guess them, and they’ll generally come up with the right choices every time," he said.

The key word is "direction." Mason’s team knows not only where they’re headed, but also where they stand, especially in terms of their internal reporting areas. In the case of Portland, the resut was to raise the bar for others. "We’ve been able to move our metrics in the company in most areas to be among the best in the company," he said.

Portland won Comcast’s own System of the Year Award in 2005 and 2007. (Seattle, where Mason had been "fixing things" before arriving in Portland, won in 2006 and 2008.) No wonder that they got to lead Cavalry’s charge.

Executing Cavalry

To execute Cavalry, management turned to a tech ops manager at one of their FFOs, Todd Gorder, making him a full-time project manager. In turn, he built a cross-functional team of 35 people that began meeting weekly a few months before the project started.

Gorder faced a challenging mix. "There was a new product, the DTA, a lot of different ways to build M&Ps (methods and procedures), folks in corporate engineering, in the division, in the market," Mason said. "Trying to get the alignment of all the opinions and directions, Tom was really the focal point for all of that."

These meetings, which continued after the project launched in November in Salem and then rolled across the region’s seven ad zones, provided the forum for adjusting M&Ps and capturing lessons.

One of those tweaks came the first week of the launch. Henninger said that the plan for assisted living communities had been to handle both installs and paperwork on site, but a course correction moving the backoffice to a team at the FFO proved more efficient.

Mason also pointed to the one-offs. One involved configuring services for a quadriplegic customer, a case that Mason figured could be replicated as Cavalry rolled out nationwide, possibly encountering wounded veterans.

Fitness centers and hotels also posed challenges. Mason said Paul Klein, senior director of engineering, was "instrumental" in solving these corner cases.

Effective outbound messaging, critical in self-install scenarios, was also part of the Portland experience. "The pay off for me was I didn’t feel like our customers were left out of the mix," Mason said. "They were in the loop."

Customers with QAM-tuner, flat-screen TV sets posed perhaps the largest challenge, according to Henninger. But once they saw the small DTA’s size and infrared remote, "the vast majority of them got over it in a hurry."

"Our communication plan really touched every customer six times," Theressa Davis, regional VP for public relations said, but that doesn’t count the number of screen crawls launched as Cavalry rode sequentially across ad zones.

Scout, HSI, WiMAX

As thrilled as they are to have completed Cavalry, Henninger and Mason seemed equally pumped about another project.

In 2006, the Comcast national team developed an IP-based network surveillance tool called Scout. Leveraging talent on Paul Klein’s team, the region soon began looking at ways to partner with Philadelphia to link Scout with the region’s own robust set of network maps.

Dubbed Scout Extreme internally and being re-cast as "GeoCom" by the national team, the proactive monitoring tool has its fans. "It’s not only been proved, it’s being emulated," Mason said.

"We also have an application called Stealth Scout, which is kind of invisible to the user," Senior Director of Telephony Operations and Network Dispatch Joe Mulder explained. The upshot: "It doesn’t have be a customer-reported ticket that prompts us to do a manual map query."

As for high-speed Internet, in December 2008 Portland completed its DOCSIS 3.0 upgrade, a six-month project of its own. Mason said they have dual 64 QAM  channels and a separate 16 QAM channel on the upstream.

The system officially has launched 50 Mbps downstream and has the capacity to launch 100 Mbps, if it hasn’t already for some commercial customers.

The big "ah-ha" in this project was realizing the danger in focusing exclusively on the CPE and modems on a particular blade or port. "In the beginning we were looking at it more mathematically than we were geographically," Mason said.

"Electronically everything looks just fine," he said. "But to get around a river, you’re adding 20 miles of drive time, when it looks like it’s just a mile away."

The region was also the first market to launch Comcast to Go. Business partner Clearwire itself decided that Portland would be the launch site for this WiMAX product. "But we felt good about doing it," Henninger said. (Several years ago, Portland had volunteered to go first with Pivot, the former joint venture with Sprint.)

As is the case elsewhere in the industry, the wireless background of employees reaches from front line call center employees up to the top leaders, in this case including Henninger and Kelly Johns.

Although Comcast doesn’t touch the WiMAX infrastructure, Portland characteristically figured out a way to add value, equipping a team of technicians with 4G devices, all the better to identify dead zones and create more granular maps than their competitors.

"We’re really trying to pay attention to the coverage issues," Henninger said.

More business and apps

In the business services arena, the region is leveraging early work that the region had done in institutional networks (INETS).

"We’re a very highly regulated market that has had us involved early on in a lot of commercial and municipal customers that were part of our rebuilds from the beginning," Mason explained. As has occurred elsewhere, the franchise basis of many such relationships has shifted to a business footing, with bandwidth demands and capacity also accelerating.

"We’re just trying to move the bigger ones, who want higher bandwidth, more point to point, onto (our) Metro Ethernet platform," Klein said.

Accelerating business is partly a matter of "choreographing your assets with the need," Mason said. That returns to answering geographic questions, as well as interfacing with the business and enterprises appropriately.

"Our current staff is dedicated and has to work with the IT staff of all these places and is succeeding fairly well," Klein said.

That would apply to the small to medium business (SMB) market. And when the region officially pulls the trigger on 100 Mbps DOCSIS, the message is likely to be aimed there, as it was in Minneapolis.

While largely a corporate-driven initiative requiring the addition of some routers and MacG code, Caller ID on the TV is popular. "Customers love it," Mason said. Portland rolled that service out earlier this summer, and followed it up in August with Universal Caller ID application that also displays on the iPhone and iPod touch.

Deep bench

Another indicator of Portland’s operational vitality includes the strength of the local SCTE Cascade Range Chapter with which it is associated. Cascade won the Chapter of the Year, and Chapter Secretary and Comcast employee Amanda Walton won Chapter Member of the Year Award both in 2007.

Then there is the frequency with which its teams have succeeded in Comcast’s annual Cable Jeopardy tournament. "We’ve been back three times to the national finals," Davis said.

After one such performance, a particularly sharp and enterprising member of their team, Scott Johnson, forwarded an article to us on how to teach digital modulation to front-line employees.

The bottom line: Johnson was exceptional, though possibly not singular. "We’ve got a whole herd of folks just like him," Mason said.

-Jonathan Tombes, Editor