NSA to store yottabytes of surveillance data in Utah megarepository (update: not so much)

[Eugen Leitl]

(the estimates are likely bogus, but petabytes are really cheap
now, given 2 TByte nearline SATA drives, zfs and Thumper-like
hardware it's just two racks, so exabyte-range storage is
perfectly feasible, and it would be a data warehouse, not
just dumb Backblazes)

http://www.crunchgear.com/2009/11/01/nsa-to-store-yottabytes-of-surveillance-data-in-utah-megarepository/

NSA to store yottabytes of surveillance data in Utah megarepository (update:
not so much)

by Devin Coldewey on November 1, 2009

Therebs an interesting article in the current New York Review of books
(predictably, a book review) detailing the history of the National Security
Agency, that shadowy power-behind-the-power to which we surrender much of our
privacy. That in itself is interesting, but I found the introduction a bit
shocking: the NSA is constructing a datacenter in the Utah desert that they
project will be storing yottabytes of surveillance data. And what is a
yottabyte? Ibm glad you asked.

There are a thousand gigabytes in a terabyte, a thousand terabytes in a
petabyte, a thousand petabytes in an exabyte, a thousand exabytes in a
zettabyte, and a thousand zettabytes in a yottabyte. In other words, a
yottabyte is 1,000,000,000,000,000GB. Are you paranoid yet?

The more salient question is, of course, what are they storing that, by some
estimates, is going take up thousands of times more space than all the
worldbs known computers combined? Donbt think theybre going to say; they
didnbt grow to their current level of shadowy omniscience by disclosing
things like that to the public. However, speculation isnbt too hard on this
topic. Now more than ever, surveillance is a data game. What with millions of
phones being tapped and all data duplicated, constant recording of all radio
traffic, 24-hour high definition video surveillance by satellite, therebs
terabytes at least of data coming in every day. And who knows when youbll
have to sift through August 2007bs overhead footage of Baghdad for heat
signatures in order to confirm some other intelligence?

As for the medium on which the data might be stored on, thatbs anybodybs
guess. Whoeverbs making the estimates is probably playing a bit fast and
loose with exponential curves, but if any of the alternative storage
technologies we cover here on CG are any indication, yottabytes wonbt seem so
big a few years from now. We can be sure, however, that despite their better
dollars-per-gigabyte cost, spinning hard disks wonbt be in use as a main
medium. The electricity required, mean time before failure, and other
maintenance issues are probably unacceptable for an economy-minded government
agency b interestingly, it seems that lack of electricity is one of the NSAbs
primary concerns.

The article mentions that the NSAbs equivalent in the UK, the Government
Communications Headquarters, asked that all telecoms providers store and hand
over a huge amount of customer data for an entire year. They refused, citing
bgrave misgivingsb and noting that at any rate the level of data collection
expected was bimpossible in principle.b Tut tut! Those Brits lacked the
American can-do spirit. Thus it was that AT&T and other telecoms instantly
complied with US mandates following September 11. The extent of the
governmentbs meddling with switches, routers, antennas, and so on may never
be fully known, but I wouldnbt be surprised if everyone reading this article
isnbt on the record somewhere. Storage capacity of this magnitude implies a
truly unprecedented amount of subjects for monitoring.

There is talk of the NSA shutting down altogether or being rolled into
another agency, but I suspect that the btoo big to failb idea, as well as the
bour safety is worth any priceb dogma, will prevent that eventuality. Itbs
more reasonable to ask when or if its expansion will cease being sustainable.
These datacenters, and the yottabytes they will hold, are extremely expensive
as well as practically having bulls-eyes painted on them to the enemy
(whoever he is) b though at under $10bn the NSAbs budget is a footnote
compared to other programs and agencies. So is the increasingly (to use a
semi-word that is only rarely usable) tentacular NSA a necessary evil of the
digital age, or a cancerous money sink born from the colossal intelligence
competition of the Cold War?

The answer will only be visible in retrospect years from now, perhaps when a
sequel to the book being reviewed (The Secret Sentry: The Untold History of
the National Security Agency, by Matthew M. Aid) is released covering the
heavily-redacted records of the early 2000s. In the meantime, itbs probably
best to assume that the walls have ears.

(Updated with a note on storage medium)

Update 2: A commenter points out that in the study cited, yottabytes are only
one possible estimate for total storage requirements. The more realistic
estimates are in the hundreds of petabytes, which is much easier for a
datacenter to accommodate. That said, Ibm leaving the post as it is because
the speculation still stands with bonlyb hundreds of petabytes being stored
in these datacenters. However, adjust your tinfoil hats accordingly.

Update 3: A tipster informed us that this facility is not being set up in the
desert, but actually in suburban Salt Lake City, near the municipal airport.
Interestingly enough, theybve built massive earth berms to conceal the
massive amounts of military hardware being relocated to this location. There
are also constant flights of black, unmarked helicopters flying in attack
formation over the valley, making the residents quite nervous. So whatever it
is they are building, itbs very well protected. Apparently you can see all
this for yourself from the Jordan Commons shopping mall.

Oh, and sorry Robert. The NSA already knows it was you who told us about
this.