December 2003 - cypherpunks-legacy

Announcement: Alert Mailing List
by Christopher Klaus 17 Dec '03

17 Dec '03

Announcing a new security mailing list - The Alert. The Alert will be covering the following topics: - Security Product Announcements - Updates to Security Products - New Vulnerabilities found - New Security Frequently Asked Question files. - New Intruder Techniques and Awareness To join, send e-mail to request-alert(a)iss.net and, in the text of your message (not the subject line), write: subscribe alert To remove, send e-mail to request-alert(a)iss.net and, in the text of your message (not the subject line), write: unsubscribe alert This is a moderated list in the effort to keep the noise to a minimal and provide quality security information. If your site is interested in network security, we put out several FAQes (Frequently Asked Question) that cover the following main areas of topic: Vendor Contacts - Who is the security contacts at IBM, HP, Dec, Motorola, etc. - Web page at: http://iss.net/iss/vendor.html Patches - List of all security related patches catergorized by OS type. - Web page at: http://iss.net/iss/patch.html Compromise - Check list of things to do if your machines are compromised. - Web page at: http://iss.net/iss/compromise.html Anonymous FTP Security - How to correctly set up FTP and check for vulnerabilities. - Web page at: http://iss.net/iss/anonftp.html Sniffers - What they are. How they work. How to detect them. And solutions. - Web page: http://iss.net/iss/sniff.html Security Mailing Lists - A comprehensive list of security mailing lists. - Web page: http://iss.net/iss/maillist.html If possible, it might be a good idea for you to add links to the above web pages on your own Web server and point people who need to know some of the network security issues to the web page. It is possible to point to all of the FAQ pages at: http://iss.net/iss/faq.html -- Christopher William Klaus Voice: (770)441-2531. Fax: (770)441-2431 Internet Security Systems, Inc. "Internet Scanner lets you find 2000 Miller Court West, Norcross, GA 30071 your network security holes Web: http://iss.net/ Email: cklaus(a)iss.net before the hackers do."

2 1

GISA budget: millions for cryptanalytic machine
by Ulf.Moeller＠hamburg.netsurf.de 17 Dec '03

17 Dec '03

According to `die tageszeitung' of 7 Oct 1995, last year the German Information Security Agency was granted several million marks for buying a computer to decrypt telecommunications traffic for the secret service BND.

1 0

Re: Crypto's Role in Evil?
by tcmay＠got.net 17 Dec '03

17 Dec '03

I agree with the points Jon Lasser makes, whether or not we have the same political point of view. At 8:37 PM 10/9/95, Jon Lasser wrote: >I'm not a libertarian, or an anarcho-capitalist. I do, however, support >rapid deployment, without restrictions, of strong crypto. Here's why: > >(1) It is impossible to stop these technologies; someone on the list >recently reminded us that a bright fourteen year old could reproduce the >basic functionality of PGP in a brief period of time. The mathematics >of, say, RSA, are fairly basic. I think it's unreasonable to outlaw >multiplication of prime numbers, don't you? I agree with this, but we have to be careful. Consider this parallel: "It is impossible to stop people from having high-power sniper rifles. Anyone can shoot people from afar with relative safety. I think it's unreasonable to outlaw use of high power rifles for assassinations, don't you." (I'm not making an argument for gun control, just citing a parallel--one of many I can think of--where just because something is easy to circumvent does not mean that it shouldn't be outlawed.) >(2) While these technologies allow people to violate the law, I have >enough faith in humanity to believe that civilization won't collapse as a >result. There have always been criminals, and there have always been >revolutionaries. Only recently has "The State" been able to supress >these forces without maintaining a physical presence. Crypto merely >returns us to that default. Agreed. Although it does allow certain new markets to be efficiently developed that could not develop otherwise. >(3) These technologies also allow honest, law-abiding citizens to protect >themselves from criminals, both within and outside the government. >Industrial espionage can be defeated through strong crypto, for example. >J. Edgar Hoover would have been powerless to harass Martin Luther King, >jr. had MLK had access to strong crypto. This is a good point to stress. --Tim May Views here are not the views of my Internet Service Provider or Government. ---------:---------:---------:---------:---------:---------:---------:---- Timothy C. May | Crypto Anarchy: encryption, digital money, tcmay(a)got.net 408-728-0152 | anonymous networks, digital pseudonyms, zero Corralitos, CA | knowledge, reputations, information markets, Higher Power: 2^756839 | black markets, collapse of governments. "National borders are just speed bumps on the information superhighway."

1 0

Basic Flaws in Internet Security and Commerce
by gauthier＠espresso.CS.Berkeley.EDU 17 Dec '03

17 Dec '03

Basic Flaws in Internet Security and Commerce We believe that the current focus on secure session-layer protocols and sufficient randomness have obscured more fundamental flaws in end-to-end security. In particular, secure end-to-end transactions require two parts: a secure protocol to communicate over untrusted channels, and trusted code at both endpoints. The latter problem has received less attention, but destroys security regardless of the quality of the protocols or of the random numbers. We have implemented a series of related attacks utilizing IP spoofing: * We can spoof NFS to patch binaries on the fly if we are on any subnet between the NFS client and NFS server. We used this to turn legitimate Netscape browsers into versions that used a fixed key (known only to us), thus invisibly eliminating security. * The same trick allows us to defeat Kerberos security by attacking kinit. * We can also spoof NFS file-handle lookups, so that we can replace any file (such as .login) with another file that runs with root access privileges (even if the requesting user cannot). These work because the trusted path to executables is really not trustworthy in most environments. Although we use on-the-wire patching to compromise executables, the client binaries can also be compromised during download, by on-the-wire patching of FTP or HTTP transfers. Trojan horses and viruses could also patch the client software after it's on the local disk, especially on systems like Windows 95 that do not provide access control for files. Given that these are realistic threats, we believe that these issues must be resolved before internet security and commerce are realistic. ------------------------------------------------------------------------------- We began to consider in more detail some fundamental weaknesses of common network security practices that would lead to trivial further attacks on Netscape as well as many other security tools like Kerberos. It was our goal to demonstrate that it is trivially possible to patch executables on-the-wire to completely compromise their security. In doing so, we hope to reinforce the point that security is an end-to-end problem that is far harder than getting the protocols correct. Strong, correct protocols only make more subtle endpoint attacks more likely, especially in light of the potential for financial gain as the amount of commerce on the Internet increases. Most of the attacks we discuss are suitable for the systematic exploitation of large groups of users: an entire organization, or even a large fraction of the user base of a particular piece of software. In many computing environments a pool of common executables, like the Netscape binary, are provided to clients by a fileserver. In such systems, including NFS, AFS and Windows NT, there is no authentication of the file contents sent between clients and servers. In these systems there are provisions for sophisticated access checks to determine file permissions, at open or handle lookup time. But the file contents that are read from the server are not authenticated in any secure way. The client has no way to determine if the bytes are indeed being sent by the server. Our first attack model is one in which the attacker has (promiscuous) network access to any machine on any ethernet subnet between the fileserver and the clients under attack. In under a day we produced software that can exploit the lack of authentication in NFS to patch the object code of any executable on-the-wire as it travels between the NFS server and the client machine. The technical details of the attack are rather simple. To retrieve data from the NFS server a client sends a short request message detailing which block from the file it is interested in (where a block is a range of bytes). The attack software is located on an ethernet segment between the client and the NFS server, so is able to snoop this traffic. The attack software snoops, waiting for any request for a particular block of a particular executable; for example, the block containing the session-key generation code in the Netscape executable. It is then able to forge a reply from the NFS server and transmit it to the client. If the forged packet reaches the client before the legitimate reply, it is accepted and the legitimate reply is discarded as a duplicate. There is obviously a race condition between the injection of the forged response and the true response. Since the attacking software is focused solely on this task, while the fileserver is certainly servicing requests from many clients, it stands a very good chance of winning the race. We have observed that the attacking software wins the race a large fraction of the time. Given this ability it becomes possible to compromise the security features of any executable loaded from the network. We have examined the Netscape v1.1N executable and located the code that selects the session key. By patching only 4 bytes we were able to cause the selection of a predictable session key every time the browser engages in the SSL protocol. It is then trivial to snoop and decrypt all traffic from the browser to secure servers, obtaining credit card numbers or other private information. Since this is really an attack on the client, it is not limited to the Netscape browser. On the contrary, it is extremely widely applicable. An appropriate patch to the Kerberos kinit executable makes possible the compromise of any passwords entered by users, and therefore all of the authentication facilities provided by Kerberos. In many environments, including our own here at UC Berkeley, all the Kerberos application binaries are served from an NFS server. This represents a major flaw in security as our attack demonstrates. Having authenticated file services (kerberized NFS or AFS) is useless if the integrity of the kinit executable cannot be ensured (most easily by obtaining it from local disk). However, making local copies of crucial binaries is not sufficient in the face of a more serious set of variants on the NFS spoofing attack. The spoofing software can be placed as before, in a position to snoop requests to the NFS server. As clients issue a lookup filehandle request the spoofing software can return the handle to a different executable and also forge its attributes. By tricking users into executing code that is setuid root, unlimited access to the client's workstation can be obtained easily. It is possible to mount NFS partitions so that setuid root executables will not be honored by the client. Still, the spoofing software can make arbitrary NFS filehandle lookup requests succeed, and substitute a trojan of some sort. The attacker could cause misspellings of commonly executed commands to appear to succeed, or could spoof other files that are trusted by the operating system. For example, the user's .login file is a natural and easy target from which to leverage further damage. This implies that it is unsafe to execute any program obtained via an insecure channel to an NFS server, no matter what the privilege level of the client user. Neither is it limited to NFS or file-serving protocols in particular. Protocols based on TCP, rather than UDP, are just as vulnerable. It is possible to hijack non-authenticated TCP connections, although it is somewhat more complicated. Attacks based on spoofing traffic coming from the distribution site of popular software packages is also possible. Berkeley, for example, is a mirror site for the Netscape browser. Any student with promiscuous network access on a machine between the ftp server and the main link to the larger Internet could have installed similar patching software to patch the huge number of copies of the binary that were retrieved from server.berkeley.edu. More mundane attacks based on trojan horses or viruses remain viable today. These attacks must exploit some other weakness in a system's security to infiltrate, but once in place they can perform patches to local binaries to fully compromise a system. Previously such attacks were mostly motived only by ego or malice; it is now more valuable to compromise a client invisibly, so that the user believes the system is secure. Thus, unlike traditional viruses, the new strains will aim to have no visible effect on the system, thus making them difficult to detect and easy to spread unintentionally. Our patch of Netscape has this flavor. We realize that it is impossible to eliminate all security holes; one can always question whether it is safe to trust the hardware, or whether outside channels used for communication of public keys or checksums are truly secure, etc. Fortunately, in practice it should suffice to handle far less than all of these risks. We hope to have demonstrated one gaping hole in practical security today, and to have highlighted the problem of the trusted endpoint. There is one simple step that we can suggest that would go a long way towards improving the security of endpoints. Increasing the practice of software providers widely publishing cryptographically secure checksums of their executables would be extremely helpful. A small amount of paranoia and care must be applied to securing the executables used in the verification process. A read-only floppy disk would be appropriate to hold the verification software, for example. We are concerned that security on users' workstations and PCs is currently insufficient. When real money is at stake, endpoint security must withstand greater scrutiny. In summary, protecting the communications channel doesn't help if the endpoints can be subverted. We implemented and discussed several related attacks that replace legitimate programs by compromised versions. Until we can trust every program that executes between the time we boot and the time we finish the secure protocol, we cannot reliably authenticate anything. Today there is no basis for this trust. Eric Brewer, brewer(a)cs.berkeley.edu Paul Gauthier, gauthier(a)cs.berkeley.edu Ian Goldberg, iang(a)cs.berkeley.edu David Wagner, daw(a)cs.berkeley.edu A copy of this post is available as http://http.cs.berkeley.edu/~gauthier/endpoint-security.html

1 0

Re: Certificate proposal
by Bill Stewart 17 Dec '03

17 Dec '03

At 09:46 AM 10/9/95 -0700, Hal wrote: >>This doesn't necessarily eliminate certificates - while you have a signed >>statement from Alice's key that she uses Bank Account X, and a signed statement >>from Alice's key authorizing transfer of $D from Bank Account X to Bank >>Account Y, >>the Bank, or a customer, may refuse to accept the request unless there's >>a signed statement from the Bank's key that Alice's key uses Account X. >>None of these need Alice's name, or for that matter the Bank's, as long as >>there's >>also a signed attribute statement from the Bank's key that it's a bank, etc. >>The meaning of the certificates changes a bit, but there's still a certificate >>from the bank binding Alice's Key to Alice's Bank Account. > >I can see using keys with attributes in this way, for credentials or as >other forms of authorization. But what about for communications privacy? >What is the attribute that tells you that using this key will prevent >eavesdropping? What I was trying to get at with this post was that the assertion that key- centered communications probably won't require certificates is incorrect. As far as privacy goes, this set of keys and certifications lets you create private communications (using signed DH, etc.) with the entity that owns the private key for Bank Account X. No, you don't know if that entity is really Alice or really MITM; in fact you don't know Alice's name, if it's done right. You just know that the Bank says it will honor requests for money from Bank Account X (assuming you know where to find the Bank, which is a separate but similar problem.) So assuming you're selling politically correct widgets and not pharmaceuticals or financial privacy consulting services, you probably don't care too much about who's on the other end - the person who's giving you the money is the person you want to be talking to. I'm not trying to define away the MITM problem - I think there _are_ times you want to know for sure who you're talking to - but I think there are also a lot of times that you really don't care, as long as you have continuity and access to reputations of long-persisting identities, where the key is often enough identification. In the case of the Bank, the reason you trust the Bank isn't that you know them physically (though it was interesting when I started dealing with a local bank where the tellers knew me by name after only two or three visits); knowing your local Savings and Loan by name doesn't guarantee you can get any money out of them if there's a bank run, nor does it really guarantee that they won't embezzle the funds and head for Argentina. The reason you trust them is that they (in this case the "they" identified by their key) are doing business dealings with a lot of people and it's more profitable not to abscond. And the reason you know it's really the Bank and not MITM is that they've always identified themselves by their key from the beginning. Just like the credit card who's owner we've been calling Alice has. And because you've successfully withdrawn money from the Bank before, and because you're clearing Alice's credit card transaction reasonably promptly. Checks and credit cards are especially good examples for this - the current systems need your name on them, because your name and signature are the closest they have to an authentication system. However, with digital signatures, the fact that you can sign a document verifiable by the public key is all the authentication that's needed; your name isn't. If the card has an account number for convenience, and Alice substitutes Carol's account number for hers on a statement, her signature won't match the public key the bank wants on the request, and it'll bounce. (In this case, the certificate from the bank would probably include the account number as well as the key, but it's not critical for on-line systems, just more efficient.) #--- # Thanks; Bill # Bill Stewart, Freelance Information Architect, stewarts(a)ix.netcom.com # Phone +1-510-247-0664 Pager/Voicemail 1-408-787-1281 #---

2 1

java security & the cypherpunk agenda
by Vladimir Z. Nuri 17 Dec '03

17 Dec '03

once again I will offer a contrary view, and once again to PM. PM predicts that serious (unfixable?) bugs will be found in Java on the level of what was recently identified in Netscape. I disagree. I am not intimately familiar with Java (nor, obviously, is PM), but I know Java was designed from the ground up to be secure on every platform it is running on. in contrast to most OSes, the whole theory of Java is based on creating a totally secure, unbreakable programming environment. most OSes start out with the idea, "what is an application? how does the OS handle it? how does the OS permit file I/O etc.?" Java starts out with the idea, "what operations can be allowed without compromising security but still allowing algorithms"? the software will succeed or fail in its aims on how well it accomplishes the criteria of unbreakable security.. I suspect it is eventually going to be considered more secure than many OS's out there that it runs on (even unix). (and Java may eventually evolve to become an integral part of future OS security apparatuses). one thing that people don't seem to realize: Java was clearly engineered to be a language that would allow "cycle sales" over cyberspace. in other words, how can companies lease out unused cycles to the world just like power companies sell off their extra watt-hours to adjacent cities? a secure language is the critical piece, and the Java designers were clearly aiming for this remarkable and commendable accomplishment. someday, even conservative companies such as IBM may agree that the security of Java is so great (and so widely well-proven as rock solid) that they will lease out their unused, nighttime CPU cycles to Java applications and interested buyers, because they can be sure that there is just no way to sabotage a system no matter how ingenious you are.. PM tends to suggest that any language or computing environment with a certain degree of complexity simply cannot be made secure. I generally agree with this. however I suspect that Java is beneath this complexity cutoff. I do agree with other cypherpunks however that focusing cpunk attentions on breaking Java is a highly worthwile activity. Java is an extremely promising development and I fully suspect it will become something of the first widespread "cyberspatial programming language". in another message I talked about how companies such as Netscape may not become so security conscious themselves but work with other companies who are security paranoid and try to successfully integrate their applications. Java is the prime candidate for this. and I am willing to bet the Java engineers have seen every security trick in the book (many I suspect perhaps even PM in his own paranoia and obsession is not aware of) and can demonstrate how Java, at least if implemented bug-free, defeats *all* of them and can virtually be proven secure. now, that little catchword "bug-free" I put in the last sentence is something that everyone here can jump on. "it might be possible to create a secure language *in*theory*" say the naysayers, "but in practice it will always have bugs that can break it." well, I think if anything will go beyond this weakness, Java has the best chance. and the above argument sounds suspiciously like, "no software can really accomplish exactly what it sets out to, because it will have bugs". sure, Word and Excel both have bugs, but they do not defeat the primary design. and this argument, "you cannot implement perfect software" is more of a complaint against human psychology than against a group who is trying to implement a theoretically secure programming environment. (perhaps they should use the qualification, "java is the first secure programming language, in theory; in practice YMMV <g>"). I think we should be careful to make this distinction. there is a lot of software that is not even theoretically secure, and there is some software that tries to be secure but is not due to implementation difficulties. the latter should not really be our concern so much. we are not here to try to prevent programming errors (i.e., not correctly translating a correct design into code) so much as to try to get companies to write software, that if written properly, is secure. (it seems to me this is the logical first step, and then then next step is trying to break the code that seeks to be secure and flame the eyebrows off its designers, as has been done here recently <g>, but unless one first has a good hammer, one does not care about faulty nails..) I was talking to a press reporter and he asked if the cpunks had gone into some sort of new phase in which they were becoming something like the policemen of cyberspace. I said that they had not really changed in their goals much at all, it's more that the software that cpunks have fantasized about for years is now starting to materialize, and there is now something concrete to focus energies on. Netscape is a good example, and Java is another example. I think if I had talked about Java a few years ago on this list, and about who invented it (Sun), what it was designed to do, etc., people here would be delirious with excitement. instead, there is a bit of cynicism here that is a bit surprising to me. if there ever was a "cpunk language" that seeks to achieve widespread, home-grown, secure algorithms in cyberspace, I would consider this this closest. I highly recommend cpunks pay close attention to Java and help in its development, because imho it really has been conceived largely with key cpunk agenda items as basic design criteria. I suspect it is going to become a very strong standard and eventually recognized as one of the most secure pieces of software ever designed, at least in regards to theoretical capabilities. --Vlad Nuri

1 0

Re: PC disk wipe software
by shamrock＠netcom.com 17 Dec '03

17 Dec '03

-----BEGIN PGP SIGNED MESSAGE----- In article <199510091318.JAA24166(a)jekyll.piermont.com>, perry(a)piermont.com ("Perry E. Metzger") wrote: > I thought I'd ask here, since its related to a lot of the stuff people > have written for the PC platform in these parts. I need to securely > wipe a bunch of data from an MS-DOS hard drive, so that it can't be > recovered. Are there any readily available utilities for this? Will > any of them selectively erase only data that isn't allocated to > existing files? Whatever you use, check its claims with a disk editor. None of the various wipe utilities for the Mac that I am aware of (Norton, Burn) work as advertized. They all invariably fail to wipe the slackspace at the end of allocated, but not fully used blocks. Consequently, several kB of data left behind in said slackspace by previously deleted files will be missed on your average drive. - --- [This message has been signed by an auto-signing service. A valid signature means only that it has been received at the address corresponding to the signature and forwarded.] -----BEGIN PGP SIGNATURE----- Version: 2.6.2 Comment: Gratis auto-signing service iQBFAwUBMHmgGioZzwIn1bdtAQFEsQF8CYfksEosFfBJoQckOM390b7CBeupCtrf T2Var/yEUwuT5lxH7xJ6y9p69wfaljtz =Hkkv -----END PGP SIGNATURE-----

2 1

Re: Certificate proposal
by tcmay＠got.net 17 Dec '03

17 Dec '03

I've never thought Hal just wasn't "getting it," as the semantic issues involving True Names, spoofing, and the ultimate reality of whom one is dealing with are not simple issues. At 11:16 PM 10/9/95, Hal wrote: >It occurs to me that perhaps I have been missing a point here when people >argue that having a "man in the middle" is not that different from >various forms of secure communication, such as where Bob has multiple >personalities or is a committee. I have been taking this to mean that we >should therefore not worry about MITM attacks, which seems crazy to me. > >Instead perhaps this was meant as a "reductio ad absurdum" argument for >why MITM attacks cannot be prevented in the scenario where people have no >out-of-band contact. Anything which could detect and prevent MITM >attacks could, by this analogy, detect whether Bob had multiple >personalities. Since the latter is obviously impossible, the former must >be as well. Hence the problem has no solution and we should not >waste much time on it. I think I made the "multiple personalities" and "Man in the middle stole my brain" points pretty clearly in a satirical post I did last week. Nobody commented on it, for whatever reasons. In case anyone didn't see it, I'll include it again here: Date: Fri, 6 Oct 1995 17:22:06 -0700 X-Sender: tcmay(a)mail.got.net Mime-Version: 1.0 To: cypherpunks(a)toad.com From: tcmay(a)got.net (Timothy C. May) Subject: MITM = Medusa in the Middle Sender: owner-cypherpunks(a)toad.com Precedence: bulk At 5:08 PM 10/6/95, the personality masquerading as Hal wrote: >Well, this is not necessarily the case. A MITM may be signing my >messages for me, and then putting them back the way they were before I >am allowed to see them. Granted, this would not be easy, and perhaps >the difficulty of this would be great enough that you will feel >comfortable using an unsigned key. But if it were accomplished, then >your messages to me would actually be insecure. No matter how >convinced you became of my sincerity and trustworthiness, actually our >conversations would be overheard by a third party despite both of our >efforts to the contrary. Our use of encryption would be rendered >futile. Doesn't this bother you? What the putative entity "Hal" is only hinting at, hypothetically, has actually forced this entity, sometimes known to many of you as "Tim," or as "tcmay," to reveal. This entity now feels the time has come to reveal it's True Nature, even though some have suspected it's True Nature (Lance, are you listening?). The "real" Timothy C. May has been locked in his room since 1983, fed through a slot in his door, and generally mentally tortured by Instrumentalities such as Ourself. We have interposed Ourself between the Real Timothy C. May and those who have communicated with him. All communications intended for Timothy C. May have actually been intercepted and processed by Us, and all communications attributed to Timothy C. May were actually generated by us. Very clever of us, don't you think? Very few have even expressed suspicions that this was the case. We have called our approach the "Medusa in the Middle," or MITM. Thank you for your attention. --Snake Views here are not the views of my Internet Service Provider or Government. ---------:---------:---------:---------:---------:---------:---------:---- Timothy C. May | Crypto Anarchy: encryption, digital money, tcmay(a)got.net 408-728-0152 | anonymous networks, digital pseudonyms, zero Corralitos, CA | knowledge, reputations, information markets, Higher Power: 2^756839 | black markets, collapse of governments. "National borders are just speed bumps on the information superhighway."

1 0

Re: Crypto's Role in Evil?
by mkj＠october.ducktown.org 17 Dec '03

17 Dec '03

-----BEGIN PGP SIGNED MESSAGE----- > > We mustn't forget that, until very recently, strong, widespread > > privacy has been humanity's "default" condition. > > Not much privacy in village or tribal life. Even neighborhoods of > earlier cities weren't very private. Neighbors always knew your business. The point is, strong privacy is readily AVAILABLE even under those circumstances. If tribal elders tried to prohibit people from wandering out of earshot, then you'd have an analogy. --- mkj -----BEGIN PGP SIGNATURE----- Version: 2.6.2 iQCVAwUBMHp+DV11Wd4tm8clAQHaCQP+Mi9Fno90Jj4wwaQm9M1TBUsjQENT94s1 ghHSmSSseF/UI0ReD0NoP7vBIqsieZsIS/nCgAasU4BBzAhqdn0PPu3FJayr3IBx ZUsOkzh2VCcmy0Av/zvDpCxp5xPrcJnIdhSqUxVz68J/L8eESuSrKf3tE5xsrUNK VW9Z5WytXWU= =g5YE -----END PGP SIGNATURE-----

1 0

spam detector algorithm?
by Greg Broiles 17 Dec '03

17 Dec '03

-----BEGIN PGP SIGNED MESSAGE----- I've been mulling over algorithmic/computational ways to spot spams for some time now. I think I might've come up with a way to represent messages (and compare representations) that would be useful to remailer operators who don't want to let spams (where "spam" == many messages with identical or very similar content) through their remailers. Any such technique will really only be useful at the last remailer in a chain, at least until people start sending encrypted spams (and there doesn't seem to be so much incentive for sending those). My proposed method is this: break the body of a message down into a list of words (with their frequencies). Eliminate words in that list which aren't in the "standard dictionary" (which ideally will contain many of the words used in the messages but doesn't need to have all of them). Alphabetize the list of words which remain. Plot a point in 3d space for each word in that list where its X coordinate is its position in the alphabetized list, its Y coordinate is its position in the dictionary, and its Z coordinate is its frequency (of appearance in the original text). This should produce a curve which "describes" the original text; messages which use many of the same words as the original (and don't use any new words) and have similar usage counts should produce similar curves. My assumption (which needs some testing) is that even moderately intelligent auto-spams (e.g., which assemble canned sentences into paragraphs or canned paragraphs into messages) are going to be similar enough that they'll eventually generate similar curves as other messages - the order in which the words appear doesn't matter (and isn't preserved). I'm also assuming that adding enough words to change the curve's shape would make the resulting messages nonsensical or wierd enough that they're unlikely to be useful for people who want their spams to get read. Evildoers solely interested in generating volume without coherence can just quote libertarian/objectivist texts (ha, ha, just a joke for all of you people who keep slamming "commies") or pick words/characters at random. I'm assuming - and this may be an erroneous assumption - that it's feasible to algorithmically describe and compare curves/lines in 3d space. My math is weak and spotty, but I think that's college-level (high-school, even?) math. It seems like one might compare equations which describe the curves for similarity (e.g., one curve might be x=2y+1 (in 2d space) and another might be x=2y+1.2, where "y=10" initially for each), and also compare the areas demarcated by the lines for similarity. My reason for including word frequency as a third dimension is to dampen the effect of an intelligent spammer throwing in a few early "A" words (e.g., "aardvark abcess absolute") or "Z" words to skew the curve. Any thoughts about this? Interesting? Stupid? Like I said, my math is weak. My intention is to try to cobble up a 2d version of this to see how it runs but I thought I'd see if anyone can point out why it can't work, or if it's useful enough that someone with a better math background than I've got wants to take this idea somewhere better. One side effect to the deployment of spam detectors may be that the remailer pinging services will need to move to using encrypted packets. It'd be possible for the remailer operators to identify and specially handle reliability measuring packets but that seems broken. Ideally, they should be indistinguishable from ordinary remailer messages. At least until money is involved, nobody's likely to give them special treatment - but even relatively small charges for remailing would make it more attractive for a remailer operator to try to skew the results of the pinging services so as to direct more traffic to themselves (my remailer recently hit Raph's Top Three again and that always brings a big traffic hit - it'll probably drop out again pretty soon and things'll be slow again. If I was getting $.10 for every message, though, I might care more about keeping it in the top 3.) My initial plan would be to include code in a spam detector which simply MD5's messages which don't seem to have identifiable words, and watches for a repeat of those hashes in, say, the last 100 messages seen; this would force someone who wants to send an encrypted spam (or uses a spam-detecting remailer to reach a non-detecting remailer) from encrypting once and sending 1000 times; they'd have to encrypt 1000 times to send 1000 times, which may be enough of a performance drain on them to make spamming less attractive. My impression is (speak up if I'm wrong) that requiring encryption for the ping packets wouldn't be an enormous burden on the pinging services because the new generation of software sends fewer pinging packets such that the CPU time required isn't an issue. -----BEGIN PGP SIGNATURE----- Version: 2.6.2 iQCVAwUBMHnYyn3YhjZY3fMNAQFnAgP/fBEaa7SObeu9wyqMCO6OW8rEraOtmxRG ynWeZVVvrtHgwuaS0NlhU4IMHVj/Laks4n6bbEbNRktfl/F5+HBvova52JQhoUkb 7EjEsRh57OwXHuVxJl/zODIH+qNd9lZP6+Tv7Vk2/SXVj3oRFD1jIZBUx6rBBZvf ZOaimDcSemw= =LjFV -----END PGP SIGNATURE-----

1 0