Greg Broiles writes:

I've been mulling over algorithmic/computational ways to spot spams for some time now. I think I might've come up with a way to represent messages (and compare representations) that would be useful to remailer operators who don't want to let spams (where "spam" == many messages with identical or very similar content) through their remailers. [many details elided...] Any thoughts about this? Interesting? Stupid? Like I said, my math is weak. My intention is to try to cobble up a 2d version of this to see how it runs but I thought I'd see if anyone can point out why it can't work, or if it's useful enough that someone with a better math background than I've got wants to take this idea somewhere better.

It sounds like you are liable to start reinventing parts of the field of information retrieval. The automatic construction and comparison of vectors of document parameters, as you suggested in the part I omitted, is one approach that has met with some success. (The common problem is, given a set of query attributes or a model document, to find relevant documents matching the query or similar to the model document. A variety of relevance measures has been considered.) I can't give you any specific pointers, but I advise you to check out existing implementations of these and other techniques for information retrieval before you spend too much time writing new code. FWIW, I _do_ think that such tactics would be very effective in combatting much of the spam served up these days.

One side effect to the deployment of spam detectors may be that the remailer pinging services will need to move to using encrypted packets. [...] My impression is (speak up if I'm wrong) that requiring encryption for the ping packets wouldn't be an enormous burden on the pinging services because the new generation of software sends fewer pinging packets such that the CPU time required isn't an issue.

Last time I looked, Raph's software already encrypts ping messages to remailers that have PGP keys. I assume you intend to perform the spam check after removing the optional outer layer of encryption on each incoming message. Perhaps the ping messages would survive unscathed if you only applied the spam scan to messages larger than some minimum size. I haven't seen too many 1 or 2 line spams. -Futplex <futplex@pseudonym.com>