Disaster and your NOC

Joe Tumolo can rattle off this season’s hurricane forecast like a jazzman riffing a snare drum. "There are 12 tropical storms; 10 of those would be named, and it looks like there would be five that would be hurricanes with the potential to reach land," Tumolo said. Whither the weather? Tumolo is neither a hurricane watcher nor a weather forecaster; he’s the director of business continuity planning for Verizon Communications. He’s accumulated the hurricane information, including a database that tracks hurricanes over the last 100 years or so, because Verizon’s footprint on the East Coast is prime hurricane strike territory, and Tumolo is responsible for keeping Verizon’s networks running. Tumolo also tracks daily news from multiple sources and keeps in touch with government agencies about terrorism threats because Verizon is a major service provider in New York , Philadelphia and Washington D.C. , among other key potential target areas. He’s even conversant about the amount of silicate being tossed out of Mount St. Helen’s when that volcano erupts. "When Mount St. Helen’s started to spew ash last year, we immediately activated the emergency operations center in the local market area and went through a detailed and exhaustive planning process for what we would do if we had a full-blown eruption—how we were going to treat the ash and silicate," he said. Verizon bought Hepa vacuums and arranged with outside contractors for clean-up services. "We have preparations for how to monitor entrance and egress from our central offices; put up plastic shields; had body suits, hair nets, foot protectors. We pre-purchased carpeting that when you walk on it is like Post-It notes. The particles that are on the booties stick to the carpet," he said. Planning That’s the type of thorough planning for natural disasters—hurricanes, tornadoes, earthquakes, floods—that cable operators are learning on the go as they move from video entertainment to best-effort high-speed data to lifeline telephone services. Unnatural disaster preparations—beefed up across the board after the 9-11 terrorist attacks and reinforced by the crippling Northeast blackout in 2003—traverse the spectrum, from keeping a close watch on network operations and keeping in close touch with elected officials and emergency agencies and power utilities to constantly being ready to dive into a self-sufficient bunker and hunker down with frozen TV dinners. Disaster preparation starts with the construction of the buildings that house electronics for lifeline services. "Probably the main difference between a site built for stability and a site built for long-term catastrophic failure is the ability to have bathrooms in the site, to have a small kitchen area where you can throw some frozen dinners in a microwave or something," said Steve Duchene, senior division engineer for Bright House Networks. Bright House, he said, has a facility where a technician can take refuge from a howling hurricane and maintain network operations—or at least keep things ready to go if the outside elements are blown away. "You’re not going to want to come here and have dinner with your wife," Duchene said. "On the flip side, it will sustain you." Most events, he said, last between 24 and 48 hours. "You’re not talking about having to do this for a month or two at a time. The site is not designed for nuclear war; it’s designed for a weather event and a long-term power outage," he said. Duchene’s main switch facility is a windowless reinforced concrete slab—a "built-in cave," he called it—with redundant generators, an uninterruptible power supply (UPS) system and redundant air conditioning because "if your air conditioning fails and you’re pouring 300,000 watts into a building, it’s going to get hot fast." The purpose of the building is to protect and preserve the equipment needed to maintain a network during and after an emergency. "If you’re doing your job right, the equipment will never know anything is going on," Duchene said. "If the equipment can feel any change because of the weather outside, you haven’t done something right." Redundancy The buildings—and every service provider has something similar—had one common feature. Everything was redundant, even down to dual steel doors to permit entry and reinforced walls. Scott Smith, director of Cox Communications’ network operations center (NOC), would like to take that redundancy even further and operate mirror facilities. The MSO has backup primary NOCs in Atlanta and Phoenix with secondary locations within 10 to 20 miles of the primary; if a primary fails, the secondary kicks in and begins operating. Smith would like to see every NOC on full-time status. "With the mirror NOC, we have the geographical dispersal we are looking for as well as the tools (in-building equipment) that are being used every day," he said. "It becomes, `System A, you have the control now because we have to evacuate system B.’" Cox, he said, is developing a situation room or "war room" within its Atlanta NOC with full network connectivity and "the ability for us to bring all parties together on disasters and sit collectively and have access to the network." The telco model Emergency preparedness and disaster preparation and recovery are areas where cable operators can learn from the telcos. The phone companies have been at it for more than 100 years and have learned a few tricks as their little concrete offices are hammered by hurricanes and tornadoes and shaken by volcanoes and earthquakes. "When you pick your phone up, you know you’re going to have dial tone," said Bradley Coleman, BellSouth’s network operations support and maintenance director, who experienced a lifetime’s worth of disaster work during last summer’s active hurricane season. Today’s telecommunications infrastructure also is threatened by unnatural disasters like terrorist attacks and power outages. "We’ve learned a lot when you deal with hurricanes, and that gave us a good starting point when talking about unnatural disasters," Coleman said. The crippling power outage in the Northeast in 2003 taught Duchene a lesson even though he was in Florida , at least 1,000 miles from the epicenter of the problem. "A few years back, we would only have standby generator fuel capacity for maybe 20, 24 hours," he recalled. While driving to work during the blackout, Duchene heard a radio interviewer say the signal would soon be lost because the station’s generators had run out of fuel. It was less than 24 hours since the start of the event, and the back-up power was drained. "When I heard that … I thought we need larger fuel tanks, and we started putting in larger fuel tanks," he said. "This all goes back to power," said Duchene. "Network stability, if you get right down to it, everything goes back to power. The limiting factor that determines how long your site can stay unattended is the amount of fuel you carry for your standby generators." Data and voice While voice is a lifeline service, data is growing in importance, especially as the two begin to blend as lifeline voice over Internet protocol (VoIP). "We hear more and more that it’s not just voice, it’s data that they’re using as a critical tool from day to day," said John Quigley, director of network operations and a member of the corporate emergency management group at Sprint. Lifeline VoIP means lifeline data, a lesson that Cox is learning as it moves from circuit-switched to IP telephony. "Our customers are demanding that our network be as robust as it can possibly be so that we have to make sure we have the diversity and backup systems in place so that our customers don’t know whether we’re working on primary routes, primary servers or on some kind of `Band-Aided’ backup server," Smith said. Sprint’s VoIP customers include Time Warner Cable and Cablevision Systems. The telco provider maintains VoIP and hands off to the cable companies. "Obviously, they would need to have that last mile, out the coax to the customer," said Quigley. "All of that interface is managed through our operations centers, so we monitor and manage that traffic like any other customer out of Sprint’s network. The benefit to the Time Warners and other customers that use us for their VoIP is that they have the very robust Sprint network behind them to make sure that the service is very survivable." Time Warner, Cablevision and Comcast were all asked to discuss their disaster preparedness for this story; they either declined or did not respond. Cover your assets There is also an increasing amount of critical business data running across the networks that could be endangered by an unnatural event such as a terrorist attack. No service provider was closer to ground zero than Verizon on Sept. 11, 2001 , having a primary facility next to the World Trade Center . It’s enough to make a provider think of moving to safer ground, but Verizon can’t leave Manhattan because its customers are there. "When you’re a facilities-based provider, it’s very difficult to move," Tumolo said. "You can’t go to New Jersey and service that area." What you can do, and what every provider is doing, is take steps to protect your assets. "We look at every facility within a 10-mile radius of Lower Manhattan , northern New Jersey and Washington D.C. , and then we just put additional security procedures in place that already existed," he said. It is, said Coleman, a fact of life for today’s service provider to include unnatural disasters with the natural. "We all made some major adjustments. I think we always realized the assets that we had and the value of keeping them secure," he said. Redefining `lifeline’ In an emergency, even cable TV service becomes more than entertainment. Larry Day, director of business resilience for Cox Communications, is developing a preliminary plan to submit to senior management that would, in a pinch, help restore cable to disaster-stricken areas. " What we’re working on … is a mobile solution that would involve trailers that could be pulled into an area and set up quickly to get all the services back online in a reasonable length of time," said Day, emphasizing that the idea is now only in its very earliest planning stages and has not been submitted to Cox management. "We do not have this," he said. "It is not funded. We’re looking at the viability of such a design." On the surface, it seems like a good idea. Television becomes an information lifeline—as powerful as voice and data, perhaps more so, in an emergency because of its ability to broadcast to a wide swath of audience. "Our challenge is to try to get some television in to the folks who are locally trying to keep up with what’s happening," Day said. "We want to get some television back, some functionality." Jim Barthold is a contributor to Communications Technology. Reach him at [email protected]. A Haven in Any Storm Here are a few things that a service provider should have to maintain a network in a natural or unnatural disaster. A solid building . Windows are an option; double steel doors are not. Bathrooms and dining area—nothing fancy—are nice to have, especially if you’re planning to put techs in the building and lock them in for the duration of an event. Don’t forget food, water and first aid supplies. Fuel tanks . Most hurricanes last a day or so. Other events, such as blackouts, are harder to peg. Power is essential to drive electronics, including air conditioning, in any sort of hardened facility, so there should be enough fuel to run generators for at least 24 hours, preferably longer. Sand bags . A seemingly minor detail, these can save low-lying facilities from the ravages of flooding. Earthquake-resistant construction . Less so on the East Coast, but definitely a necessity on the West Coast. Cleaning agents . Super vacuum cleaners and or other agents to remove potential pollution from fallout like volcanic ash or mud from hurricane and other storm-related flooding. Communications . While locked in the facility, the techs should have communications with the outside world to learn what’s going on and how long the event is likely to last. And besides, if it’s nuclear war, they have bigger things to worry about than keeping the system going. Tools, repair materials, spare parts and documentation . If the event damages the facility, tools and materials can make the difference between getting back up quickly or not at all. Electronics may be damaged, cables down, windows blown out, and debris strewn everywhere. At minimum, tools and materials should include cable, electronic parts, up-to-date equipment documentation, plywood and plastic sheeting, fasteners, electronic test equipment, connectorizing/splicing gear and supplies, shovels, saws, and such basic carpentry items as a hammer and nails. Don’t forget flashlights and fresh batteries. Transportation . Have suitable means to safely transport personnel and equipment to and from critical sites. Training . All personnel must be adequately trained in preparation for disasters. Conduct periodic drills beforehand and have a detailed plan in place—don’t wait until an actual event happens and assume that everyone will know what to do.