Irene Lesson #3: Critical Data Belongs in the Cloud, Not Under It
"As flood waters from Tropical Storm Irene swamped the Waterbury state office complex, seven employees from the Vermont Agency of Human Services rushed inside to rescue computer servers that are critical for processing welfare checks and keeping track of paroled prisoners living around the state," according to a story by Shay Totten on the 7days blog Blurt. Two of the employees - network administrator Andrew Matt and deputy chief information officer Darin Prail - lost their cars in the parking lot as the river rose but kept on working to assure that our servers were not lost. "We didn't know how much time we had," Matt said, "and our job was to save the servers."
The story continues: "The employees' quick thinking is being credited with saving the state's largest agency from disaster. AHS oversees not only the Department of Corrections but runs programs that serve thousands of Vermont children, families, senior citizens and individuals with disabilities. Within days, AHS was up and running again — its servers installed at an alternate site."
- Vermont is lucky to have such dedicated employees who took action regardless of personal hardship and even though experts said that particular area wouldn't flood.
- State servers and the data on them don't belong in state. The more critical the application, the more important that it be running in a cloud with several replications and that it NOT be vulnerable to any catastrophe that might hit the state. The Internet makes the actual physical location of servers irrelevant to users. It's more important that they be well-connected in cyberspace than they be located physically close to their users.
I know from Mary's experience in the State Emergency Operations Center, which also had to evacuate the Waterbury complex, that their computer systems became unavailable just when they needed them most. Sure, you can argue that all of the servers should've been on higher ground; but you never really know what kind of catastrophe is going to strike. In an emergency communication and power outages may be widespread as well as physical damage to buildings almost anywhere. When a disaster strikes, you want your critical data and servers (as well as your less critical ones) to be as far from the impacted area as possible; that means out-of-state. It is routine in cloud computing for Amazon or Microsoft or Google or whoever hosts the cloud applications to replicate them in several different locations so you don't have to worry about out-of-state catastrophes either.
Another reason for outsourcing state computing to the cloud is to make it possible to handle spikes in demand without having to have huge amounts of expensive standby capacity during normal times. When the recession first hit and unemployment claims skyrocketed, the newly unemployed had to endure the extra pain of unresponsive or unavailable servers because of the sudden surge in claims. What is an unmanageable surge of volume to Vermont or even California is a blip that a hosting service like Amazon won't notice. They won't run out of server capacity; they won't run out of Internet access capacity; and they won't be affected by instate earthquakes, tornadoes, hurricanes, or floods just when you need them most.
Some people argue that critical state data – welfare records, for example – are only safe if hosted in state. That sounds right but it's wrong. The greatest threat to data is disgruntled or crooked employees. State employees know where the state's critical data is and may have a particular grudge against an individual or a department. Google employees are much less likely to have a grudge against anyone in Vermont or know anything about the specific nature of the data hosted on their servers. Moreover, the big hosting companies spend a fortune on both physical security and hacker-proofing. No state is going to be able to match that. Of course web-accessible applications need to be developed to be hacker-resistant; but that's true whether the hosting is in or out of state.
Employees of a cloud service are unlikely to go to the extremes our state employees did to save us from data disaster. But, since resources in a cloud can be replicated in several locations at little cost to the customer, such heroics won't be needed.
A final reason for moving state computing into an out-of-state cloud is cost. The big hosting companies have huge economies of scale which they share with their customers. Forward–looking companies have moved to cloud computing so that they can focus on what's best for their customers and products and leave the physical care and feeding of servers to those who do nothing else. Many investors I know won't look at a new company which plans to spend scarce capital on servers; inhouse servers are usually a waste of capital, an unnecessary risk, and too inflexible in dealing with unpredictable demand.
BTW, what goes for the state goes double for its towns. We don't know yet how many town records – computerized or otherwise – we're lost in Irene. A town size disaster is much more likely than a statewide one. And towns have even less resources to devote to servers and their proper backup and possible disaster recovery.
I understand that Vermont had to get up and running quickly and probably had to buy some new computers to do so. Nevertheless, one of the ways that we can learn from Irene and be stronger than we were before is to get our servers into the cloud and out of harm's way.