Is there any open source - or otherwise - knowledge or speculation about which words/phrases the Terra-cycle cpu's are text-searching *for*? If it were your responsibility to eavesdrop on Iranian terrorists - or French Commercial Attache reports to Paris - or to have UK nationals, off in their private room of your building, write down the name of every in America who expresses a libertarian dissatisfaction with the Republicrat regime - would you know for sure which words/phrases to key on? It doesn't sound like a tractable problem to me.
To me it does sound completely feasible (you don't need very good accuracy). I've personally run packet filters (for statistical purposes only) on busy 10-mbit ethernets using BPF, FreeBSD, and 486 or pentium machines. They easily keep up with little packet loss. I understand T3 is 34 mbits, so only three times faster. No problem to optimize that much by specially written software, especially if you can do some of the low level stuff in hardware. As for the keyword search problem, it would easily be possible to scan much of the data (say, tcp ports smtp, nntp, login, exec, ident) in real time against a million-phrase dictionary (containing keywords, e-mail addresses, names, abbreviations, etc.). If there are performance problems, you can first limit by source/destination/protocol/port. Only intercepts (e.g., entire tcp connections) that pass this initial screening are passed on to other machines for more complicated analysis. Note also that many parts of the filtering problem parallelize quite nicely. For example, you can split the traffic to a number of machines based on the value of the numerically smaller of the source/destination addresses. I don't see any technical problems in doing large-scale internet monitoring. The equipment needed is even cheap enought to be done by motivated amateurs/individuals, assuming they can get a copy of the raw data from the T3. This is one of the reasons why strongly encrypting internet data is so important. Tatu See http://www.cs.hut.fi/ssh for information on SSH, the secure remote login program. See http://www.cs.hut.fi/crypto for information cryptography available to anyone worldwide.