On May 3, 2004, at 9:14 AM, Contempt for Meatheads forwarded:

We desperately need adult supervision and high quality minds in the intelligence business! I am growing more convince that the security clearance process, the government hiring/promotion process, and information silos are overwhelming our ability to get even a marginally adequate level of intelligence needed to fight terrorism. Wow, this is depressing.

My confident belief (100%): we will continue to lose the war on terrorism until we fix our intelligence system.

I think this analysis is correct, but also a bit too shallow to be really insightful. While there are some significant institutional problems and byzantine self-defeating regulations, these things are masking a much bigger technical problem that desperately needs to be tackled from their perspective. The volume of data they collect has reached the point where good analysis is no longer tractable in a theoretical algorithmic sense with the best tools they currently have at their disposal, particularly when you have a data space as broad and diffuse as "terrorism" to sift. Institutional procedures and problems aggravate this, but the underlying issues are deeper. One of the ways I keep track of what the US DoD is up to is by analysis of the open research programs, contracts, and grants that they publish. By threading the many, many programs together over time, you can see how fast different technologies are progressing and you can chain inferences to make an intelligent estimate as to when specific capabilities (which may require the intersection of multiple research tracks) could theoretically be available to the DoD. Furthermore, the program managers have a habit of mildly editorializing their program descriptions in response to some of the proposals they have received and the success of the proposals they have actually funded, which also gives some added insight. One thing that I have noticed for several years is that the advanced data mining and automated intelligence analysis research programs have been essentially stalled for many years now despite aggressive marketing and a large number of agencies willing to liberally fund proposals. And the editorializing of the program managers on this research track makes it clear that they are quite frustrated both with the lack of progress in this area and with the fact that research proposals keep trying to beat the same dead horse over and over. Furthermore, while most programs have a shelf-life after which they are either closed (both on good progress or no progress), these particular programs keep getting extended and re-funded over and over, sometimes under a different name but always with roughly the same parameters. As long as this program track is stuck in neutral, the intelligence agencies will have serious problems that will be all but insurmountable. The US intelligence service is a victim of its own ability to acquire data. This isn't a problem that they can simply throw money at in the sense that it requires pretty substantial algorithm breakthroughs to even be tractable for high-quality analysis. To date, private research organizations have clearly been unable to solve this problem in any meaningful way, and there is substantial evidence of this fact. In the mean time, they are left using narrow brittle algorithms to sift and analyze the data, with holes you could drive a truck (bomb) through. Someone who fully understood the theoretical limitations and likely implementation parameters of the current state-of-the-art could likely defeat the automated analysis. Fortunately for the intelligence agencies, few people have those skills and they get by on a pretty broken system hampered further by institutional problems. j. andrew rogers _______________________________________________ FoRK mailing list http://xent.com/mailman/listinfo/fork ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net [demime 1.01d removed an attachment of type application/pgp-signature]