AboutUVMMC Ransom Attack Postmortem
Preparing for next time.
First, kudos to the University of Vermont Medical Center (UVMMC) for not giving any serious consideration to paying ransom, as reported in VTDigger. Even if they had trusted the hackers to unlock the files and remove all malware, each ransom paid guarantees more attacks on someone else. The hackers are in it for the money.
Second, more kudos to the staff of the hospital system who soldiered on without access to key information as round two of the virus pandemic reared its ugly head. They worked very hard to protect their patients from both dangers.
Third, though, based on public information, the hospital should have planned better for recovery from an attack like this one. They had to wipe 5000 computers clean and put them back in service before they could use their applications again. Even a month later and with the help of the National Guard and a private security firm, the hospital had not restored full functionality and estimated the cost for each day the systems were down at a million and a half dollars NOT counting the toll on the staff and the dangers to patients.
Planning for a disaster means having a plan which works even if the original computers have been hacked, burned, or flooded out of existence! Apparently UVMMC did not have such a plan.
Hospital leadership says attacks like this are inevitable; they’re right. They cite an arms race between hackers and defenders in which the good guys sometimes lose. True also. But, if you know there is a significant chance that you are going to lose access to all your servers and laptops, then you must make sure that you can restore service without those laptops and servers. The plan must be made and rehearsed in advance of the disaster. Even the “unsinkable” Titanic had lifeboats.
According to the hospital, 1300 of the infected computers were servers – more on them in a minute – leaving 3700 infected laptop and desktop machines. Even assuming these cost an average of $3000/each (a lot) and assuming that all of them had to be replaced for service to resume, buying all new laptop and desktop machines would have cost only about $10 million – less than seven days of outage. Buying new computers quickly – starting with cheap ones to get back up and running – as well as a rehearsed protocol for loading all needed software onto them from somewhere other than the infected servers must be part of a disaster recovery plan. Replacing the desktop and laptop machines is actually the easy part of the recovery.
The hard part is doing without the servers which have been infected. Two parts to this:
- Getting access to the data. Presumably UVMMC transmits a copy of its data to a location which is both physically offsite and is not part of the hospital network. I would be very surprised if they weren’t doing this. Even if the hackers locked up the onsite data, they shouldn’t have had any access to offsite data.
- Putting the data back on servers which are not infected. As UVMMC saw, you cannot assume that your old servers will be available. Unlike the desktops and laptops, it’s not practical to buy all new servers on a moments notice. However, the advent of cloud computing means that you can rent the capacity of thousands of servers from providers like Amazon, Google, Microsoft, or IBM with just minutes of notice and without a standby fee. You pay for and use these only until your old servers are back. Rent stops as soon as you can turn them off.
However, turning up a thousand servers in a cloud, loading them with your applications, restoring backup data to them, and putting them in use in place of your own compromised servers only works if the process has been carefully planned and practiced. Even for installations larger than UVMMC, recovery should take hours, not days or weeks – if it’s been practiced. Fatalities were high in the Titanic disaster because the crew and passengers had not had proper lifeboat drill.
I’m not writing this to be critical of UVMMC; I owe the hospital my life for their medical skill. I’m writing in hope of encouraging those who are responsible for critical IT systems in an age when attacks are inevitable to make sure that, even if there is no fool proof way to prevent all attacks, there is always a quick recovery path which does not require regaining use of the compromised computers.
See also:
Comments