Bear wrote: (Bear, read the entire before you reply...) I said:
That is an over-simplification, but yes. Intelligence is not headlines. To a large extent, "what's happening" is not analyzed correctly, because the intelligence community lacks sufficient expert analysis to cope with the dataload. This capability is in the private sector. These information flows, between the government sector and the private sector, are unmapped.
I want to paddle back to the kiddie pool, but I'll try to address you Bear, you are way over my little head....and so is this subject matter.
This is not true any more. The automated analysis of trawled data has advanced considerably beyond keyword searching at this point; there are programs out there now specifically looking for much more subtle and complicated things, which were formerly the domain of intelligence analysts, and they are actually pretty damn good. The simple keyword searchers and keyphrase searchers you hear about with echelon are only the front line; they pass their data back to much more sophisticated AI programs that analyze content, and synthesize information gleaned from massive numbers of such missives.
Yes, but I'm still speaking of information that is not online, not siphonable and locked in the overt experts in the private sector. It is not "current events" or "happenings" or "what's going on." It's analysis and intricacies that are critical for decision making. Not raw data or intelligence headlines. It's Mr. X and his theories on Y, that nobody knows about - Mr. X is hidden away in the private intelligence sector or some university closet. He's a specialist on ...uhm....South African Zulu Warrior Chieftains dress and culture. He can tell you that when Zulus get in war dress and bring knives and spears to your VIP meeting - it's a sign of respect, and not a violence indicator. (I just ripped off the basics of this hypo from this guy: http://www.icon.co.za/~agrudko/ representative of the private intelligence sector) Without knowing this information, your diplomatic protection force is going to rat-tat-tat them to pieces and lead to a "diplomatic snafu" of major proportions. They need this information NOW, because the helicopter with your diplomat just landed in a remote area for this roundtable in a big grass hut, and is facing 1,000 Zulu Warriors jumping up and down and chanting in full war dress, and the protection force of 5 is counting rounds in the back of their heads. Their protocol officer fainted and is receiving medical assistance in the 'copter. They place a call - decision time is 8-10 minutes. Somebody has got to finger and find Mr. X's knowledge. This information is NOT online, not siphonable, outside of regular intelligence channels - it's in Mr. X. Mr. X is one of five western people in the world that know about these things. Right then, Chief Zulu walks up and points his knife at your diplomat. Was that a threat? Your diplomat pees in his pants in front of 1,000 Zulu Warriors. Ramifications? BTW, your diplomat is also president of a transcontinental resource-extractive company with operations in ZA and is a top-level kidnapping and hostage risk - his capture or death would have diplomatic ramifications and would affect upcoming treaty negotiations related to the world diamond market. National events often turn on intimate knowledge of the strangest facts - these facts are known by people like Mr. X. You have 8 minutes to tell these guys what to do. You can mine you data, use your CIA-google, ask your AI, make some phone calls - and you are still whistling Dixie. So, this is what you do: You CIA analyst, fire up your SIGINT/ELINT fed AI and analysis programs, you call around.... What have those Zulu Warrior's been talking about lately? You find, to your dismay, little information....Zulus don't even use phones. So what do you do? You find pictures of "Zulu War Dress" and some basic protocol. Your internal experts agree. Your call: "Zulu War Dress = War = Aggression = take immediate evasive action." You go look at online and offline sources on this diplomat's diamond company. Sadly, you do not have an expert's competitive intelligence analysis which would have told you this man is about to become pivotal in the world diamond market, due to a secretly planned merger and acquisition with a gem company. Because of this one man, the entire gem and diamond markets are about to be revolutionized. *bloody gunfire exchange* Confused Zulu Warriors. The chief was just giving a sign of respect. Dead diplomat. Zulus go on the offensive. World diamond market: kaput. Mr. X happens to consult with PPS (private protection services) in ZA in Zulu territory. Private intelligence. Yet, for some reason, Mr. X doesn't appear on your screen. Why? Because you haven't developed information flows between yourself and the private intelligence sector. By and large, you don't talk to them. If you did, Mr. X's information would be in your system. "Mr. X - expert in Zulu chieftain diplomacy." Additionally, you didn't know about the diplomat's importance to the world diamond market, because you don't have access to ZA's leading private competitive intelligence agency profiles or their data bases (i.e. Grudko has a "WOLF" database, maybe he's willing to sell some of that info, this info is not in YOUR data banks.) I know that's "out there" and there are a thousand holes in this hypo, but it gets my basic point across. Somebody smarter than me would have to give a better hypo.
Every time a situation like the Aum Shenrikyo (spelled?) subway attack happens, if the automated analysis suite didn't point it out first, human analysts come in and check out the dataflows that ran before it and around it, and create a new auto-analysis program. And then later, when another group that has anything like the same rhetoric and seems to be going through the same logistical steps pops up, the auto-analysis finds it without human help.
Your data flows are amazing quiet as to this little grass hut situation...strangely, even though the future of the world diamond market hinges upon it....
I do not speak of specific known programs here; but my primary background is in AI and expert systems, and I can state unequivocally that intelligence analysis funded most of the research in the field for a very long time, and that programs such as I described above are well within the current state of the art. It is unusual for them to be deployed very widely in private industry because in private industry there is a real problem of retaining personnel with the proper expertise to work on them. They tend to be delicate in their operation -- you go to make a minor change in the data or the rules or the schemas and the performance of all other parts of the system degrades unless you are extremely careful, well-trained, and, let's face it, consistently just plain smarter than normal people. But when they are in tune, and their vocabulary tables are up-to-date, they are highly accurate.
Okay, so you add in Zulu chief knife-pointing, diplomatic pissing...= ANSWER?
The problem of keeping these systems in tune is what drives most practical AI research today; the systems are effective, but brittle and unable to cope with subtle changes and variations very well. "Fuzzy" approaches like ANN's and Genetic Algorithms are attempts to get past this problem by making self-adjusting systems, but the volumes of data required to get self-adjustment working using such approaches are a problem; you'd have to have data from hundreds of Aum Shenrikyo type attacks before your GA or ANN really had a good chance of picking out what parts of the dataflow were relevant.
So, this isn't what I'm talking about.
So here's my speculation: human analysts are probably called in only after something takes the automatic tools by surprise, or when there is an administrative need for specific analysis that the automatic tools do not provide.
Sometimes, a human, overt, expert analyst is the one thing you need, and can't find. Our intelligence agencies are long on tricked out data whatzits,... short on analysis and overt experts, because we haven't tapped and mapped private sector capabilities, data and information flows. Again, this information isn't headlines, it isn't online, it's in some Mr. X somewhere. Sometimes OBSCURE, definitely not secret, and it's not a topic of conversation ANYWHERE. This information is outside of your intelligence channels and in hidden in the private intelligence sector. But, what you say is still *very* relevant, because in my hypothetical "Intellagora" - offering private sector analytical info for sale, exchange, or free - you would need THIS capability. Being the genius that you clearly are... I encourage you to read some Steele thought, etc.: <http://www.oss.net/infoMerchantBank2.html> mull it over, and see what you can come up with...to give it "smarts." At this point, I need kiddies floats, but the subject is "big enough" to interest you. I think my hypo makes this sound more like a "phone directory of experts," or "data mining" and that's not what I'm after..... Again, I apologize for my flailing on the subject matter. Steele is better able to explain this purported opportunity for an OSINT community or "Intel agora," and the strategic benefits to intelligence agencies and the private intelligence community. I appreciate your taking the time to give me some insight, you raised some very interesting questions, amazingly on point, that I otherwise would have never thought of. ~Aimee