Building fire – Lessons learned

By Geoff Turner, President of Tech Valley Continuity LLC

When fire struck our building we did not expect a six week recovery and several other crises to occur along the way. Fortunately, our well rehearsed continuity plan saved the day.

“A well-rehearsed plan for both I.T. and operations is critical to a company's survival”
-Geoff Turner

One hot, muggy evening in July, I sat down to dinner and my cell phone rang. "This can't be good," I said to my wife. "Geoff, there's been a fire at the office," said our facilities manager. "You have to come up right away. We need to activate our business continuity plan." That was the beginning of a six-week adventure.

Driving up I imagined flames pouring out of the two floors we occupied in the 10-story building. But when I arrived, I saw no flames and no smoke - only 12 fire trucks with the fire fighters standing around - waiting. I found my boss, the GM, standing on a grassy island in the parking lot. His administrative assistant was making calls from an out dated phone list and having some very interesting conversations with ex-spouses.

As it turned out, the fire was on the 10th floor in a cement-walled electrical room. The main power transformer had shorted out. People were struggling to turn off our generator because the key to the cage was in a desk drawer on the 6th floor, and we were denied access to the building. Finally they managed to hit the emergency shut-off switch with a stick. We then waited another hour before a city engineer confirmed that the generator would not kick in and electrocute the fire fighters. By 11:30pm the fire was out, and we decided to meet at our recovery center across town at 7:00am.

The next day we assessed the damage. Although we could access our floors and run our generator, the building could not be occupied for normal business.

During the next six weeks, it was anything but "business as usual." Two of our computers were damaged by the initial power fluctuations. Then the HVAC unit broke. After a week, our generator failed. A week later, the building generator failed. A tenant on the 8th floor broke a toilet handle, which flooded our network cabinet, and our backup center had a power failure. It got to the point where every time my cell phone rang, I cringed.

But the important thing is - the plan worked. As data providers, our computers were critical. We had redundant computers, redundant workstations and a place to work from. We had rehearsed our plan on several occasions and because of good luck, good planning and the dedication of many good people, our clients saw little or no disruption in service.

But the real story behind this success occurred 9 years earlier, when one of our largest prospects demanded in an RFP that the winner would have to secure a working disaster recovery plan. They reasoned that since their business depended upon the winner being able to survive a fire in the middle of the night, the winning bid would include a continuity plan. Because we committed to the D/R plan, we won the contract and both businesses survived that fateful July fire.

So what did we learn from this experience?  First, a well-rehearsed plan for both I.T. and operations is critical to a company's survival when something unexpected occurs. Second, when working during a crisis, there is a good chance something else will occur, either as a direct result of the incident or from the plethora of threats businesses face every day. Finally, having a good continuity plan can win you business, and when needed, make you and your clients glad you made the commitment.

But what makes a "good continuity plan"?  Probably the most important factor is the support from senior management. These are the folks that assign corporate priorities and allocate funding. Since continuity plans need to grow and change along with the company, it is critical that planning for the unexpected be part of the corporate mindset. It needs to be considered with every business decision - whether you are adding a new product line, making changes to your computer system, or moving into new facilities.

Bad things happen. It's a fact of life - especially business life. In today's tough economy, even a week of lost revenue can cripple cash flow. Remember too, that not only does a company lose immediate sales, but now loyal customers are forced to buy from the competition which could result in the loss of future sales and a long term decrease in market share. Business continuity planning is not just a good idea, it is a strategic move that can save a company's bottom line.

About the author

Geoff Turner has been a Certified Business Continuity Professional (CBCP) since 1997. He founded Tech Valley Continuity LLC to help businesses develop and test comprehensive disaster recovery / business continuity plans so they too can survive the unexpected fire. Find out more by visiting TechValleyContinuity.com.