A Cyber-Attack on an American City

[Eugen Leitl]

http://perens.com/works/articles/MorganHill/

A Cyber-Attack on an American City

Bruce Perens

Just after midnight on Thursday, April 9, unidentified attackers climbed down
four manholes serving the Northern California city of Morgan Hill and cut
eight fiber cables in what appears to have been an organized attack on the
electronic infrastructure of an American city. Its implications, though
startling, have gone almost un-reported.

That attack demonstrated a severe fault in American infrastructure: its
centralization. The city of Morgan Hill and parts of three counties lost 911
service, cellular mobile telephone communications, land-line telephone, DSL
internet and private networks, central station fire and burglar alarms, ATMs,
credit card terminals, and monitoring of critical utilities. In addition,
resources that should not have failed, like the local hospital's internal
computer network, proved to be dependent on external resources, leaving the
hospital with a "paper system" for the day.

Commerce was disrupted in a 100-mile swath around the community, from San
Jose to Gilroy and Monterey. Cash was king for the day as ATMs and credit
card systems were down, and many found they didn't have sufficient cash on
hand. Services employees dependent on communication were sent home. The many
businesses providing just-in-time operations to agriculture could not
communicate.

In technical terms, the area was partitioned from the surrounding internet.
What was the attackers goal? Nothing has been revealed. Robbery? With wires
cut, silent alarms were useless. Manipulation of the stock market? Companies,
brokerages, and investors in the very wealthy community were cut off. Mayhem,
murder, terrorism? But nothing like that seems to have happened. Some
theorize unhappy communications workers, given the apparent knowledge of the
community's infrastructure necessary for this attack. Or did the attackers
simply want to teach us a lesson?

Although they are silent on the topic, I hope those responsible for emergency
services, be they in business or government, are learning the lessons of
Morgan Hill. The first lesson is what stayed up: stand-alone radio systems
and not much else. Cell phones failed. Cellular towers can not, in general,
connect phone calls on their own, even if both phones are near the same
tower. They communicate with a central switching computer to operate, and
when that system doesn't respond, they're useless. But police and fire
authorities still had internal communications via two-way radio.

Realizing that they'd need more two-way radio, authorities dispatched police
to wake up the emergency coordinator of the regional ham radio club, and
escort him to the community hospital with his equipment. Area hams dispatched
ambulances and doctors, arranged for essential supplies, and relayed
emergency communications out of the area to those with working telephones.

That the hospital's local network failed is evidence of over-dependence on
centralized services. The development of the internet's communications
protocols was sponsored by the U.S. Department of Defense, and they were
designed to survive large failures. But it still takes local engineering
skill to implement robust networking services. Most companies stop when
something works, not considering whether or how it will work in an emergency.

Institutional networks, even those of emergency services providers, are
rarely tested for operation while disconnected from the outside world. Many
such networks depend on outside services to match host names to network
addresses, and thus stop operating the moment they are disconnected from the
internet. Even when the internal network stays up, email is often hosted on
some outside service, and thus becomes unavailable. Programs that depend on
an internet connection for license verification will fail, and this feature
is often found in server software. Commercial VoIP telephone systems will
stay up for internal use if properly engineered to be independent of outside
resources, but consumer VoIP equipment will fail.

This should lead managers of critical services to reconsider their dependence
on software-as-a-service rather than local servers. Having your email live at
Google means you don't have to manage it, but you can count on it being
unavailable if your facility loses its internet connection. The same is true
for any web service. And that's not acceptable if you work at a hospital or
other emergency services provider, and really shouldn't be accepted at any
company that expects to provide services during an infrastructure failure.
Email from others in your office should continue to operate.

What to do? Local infrastructure is the key. The services that you depend on,
all critical web applications and email, should be based at your site. They
need to be able to operate without access to databases elsewhere, and to
resynchronize with the rest of your operation when the network comes back up.
This takes professional IT engineering to implement, and will cost more to
manage, but won't leave you sitting on your hands in an emergency.

Communications will be a problem during any emergency. Two-way radios have,
to a great extent, been replaced by cellular "walkie-talkie" services that
can not be relied upon to work during an infrastructure failure. Real two-way
radios, stand-alone pager systems, and radio repeaters that enable regional
communications are still available to the governments and businesses that
endure the expense of planning, acquiring, maintaining, and testing them.
Corporate disaster planners should look into such facilities. Municipalities,
regardless of their size, should not consider abandoning such resources in
favor of the less-robust cellular services.

Satellite telephones can be expected to keep operating, although they too
depend on a land infrastructure. They are expensive, and they frequently fail
in emergency situations simply because their users, administrative officials
rather than technical staff, fail to keep them charged and have no back-up
power resource once they are discharged.

A big plus for Morgan Hill was that emergency services had an well-practiced
partnership with the local hams. Since you can never budget for all of the
communications technicians you'll need in an emergency, using these
volunteers is a must for any civil authority. They come with their own
equipment, they run their own emergency drills and thus are ready to serve,
and they are tinkerers able to improvise the communications system needed to
meet a particular emergency.

Which brings us to the issue of testing. No disaster system can be expected
to work without regular testing, not only of the physical infrastructure
provided for an emergency but of the people who are expected to use it, in
its disaster mode. But such testing takes much time and work, and tends to
trigger any lurking infrastructure problems, creating outages of its own.
It's much better to work such things out as a result of testing than to meet
them during a real disaster.

We should also consider whether it might be necessary to harden some of the
local infrastructure of our communities. The old Bell System used to arrange
cables in a ring around a city, so that a cut in any one location could be
routed around. It's not clear how much modern telephone companies have
continued that practice. It might not have helped in Morgan Hill, as the
attackers apparently even disabled an unused cable that could have been used
to recover from the broken connections.

Surprisingly, manholes don't usually have locks. They rely on the weight of
the cover and general revulsion to keep people out. They are more likely to
provide alarms for flooding than intrusion. Utility poles are similarly
accessible. Much of our infrastructure isn't protected by anything so tough
as a manhole cover. Underground cables are easily accessible in surface posts
and "tombstones", boxes often located in residential neighborhoods. These can
be wrecked with a screwdriver.

Most buried cable cuts are caused by operating a back-hoe without first using
one of the "call before digging" services to mark out the location of all of
the buried utilities. What' you start planning now.