Filtering CDR open
I've set up a new CDR that filters posts to it's subscribers- only posts from subscribers to other CDRs or known remailers (or subscriber's alternate addresses) will go to lne.com CDR subscribers. Here's the list welcome message, which explains how it works in more detail. We're not connected to all the CDRs yet but as far as I can tell we're not missing any posts. ------------------------------------------------------------------------- Welcome to the lne.com Cypherpunks Distributed Remailer This is welcome message v1.0, dated 2/22/01 Cypherpunks is a list for discussing cryptography and its impact on society. It is not a moderated list. The list operators are not responsible for the list contents. See the following pages for cypherpunks info: Background/general explanation: http://world.std.com/~franl/crypto/cypherpunks.html (read the two Manifestos-- Eric Hughes' and Tim May's-- for good background material). http://einstein.ssz.com/cdr Physical meetings: http://www.cryptorights.org/cypherpunks/meetingpunks.html Cypherpunks list archves: http://www.inet-one.com/cypherpunks http://cypherpunks.venona.com The Cypherpunks list is run in a distributed fashion. There are a number of Cypherpunks Distributed Remailers which handle distribution of the list. Different CDRs have different policies. http://einstein.ssz.com/cdr/index.html explains the CDR system. To unsubscribe from this CDR, send the words "unsubscribe cypherpunks" (without the quotes) in the body of email to majordomo@lne.com. If you are subscribed to another CDR, you will have to send your unsubscribe to it. The lne.com CDR is an attempt to reduce the spam that has invaded the cypherpunks list without blocking or delaying non-spam posts. The way we do that is to 1) drop posts which are not from a subscriber to any of the CDRs, and which are not from a known anonymous remailer 2) make an attempt to recognize spam from the contents of the mail message and mark spam as such. These filtering methods affect only cypherpunks mail going to subscribers of cypherpunks@lne.com- for example, if someone sends mail to the CDR cypherpunks@cyberpass.com, the mail gets passed to all the CDR nodes inluding lne for distribution to subscribers. If the sender isn't a subscriber to any of the CDRs, the lne.com CDR node will not send the message to its subscribers. It will forward the message to the other CDRs however. This also means that as a subscriber to the lne CDR, you may see replies from other CDR suibscribers to posts that you didn't see, because the were from non-subscribers. Posts sent directly to the lne CDR need to be from a subscriber to any of the CDRs (or a known remailer) in order to be accepted and passed on to the other CDRs. I'd prefer to have it send everything along, but that'd require a whole lot of majordomo hacking and it's just not worth it. Some cypherpunks posters post from addresses that are different from the one that they're subscribed with, causing their post to be dropped by lne.com. When this happens, I'll forward their post to cypherpunks@lne.com (headers intact) and add their alternate address to the list of posters. I will reserve the right to also delete an address I've added to the list if that address starts sending spam- this is to defeat an attack where the spammer sends a non-spam message to the list to get his address added, and then starts spamming the list. I doubt that I'll have to do this, but I'm mentioning it now in case it does happen. The spam-recognition software in use at lne.com is described at http://www.lne.com/ericm/spammaster. In short, it looks for key phrases in the mail body (like "Dear Friend" and "This is not spam") and assigns points to the different phrases. The spam point total is put in the "X-spam" header added at the end of the mail headers. The mail is still sent to the list. The rating is there for lne.com CDR subscribers to use or not as they please. The spammaster config file that the lne CDR uses is available at http://www.lne.com/cpunk/spamfilter and there is more explanation of the software at http://www.lne.com/ericm/spammaster. Other anti-spam measures: lne.com has a policy of blocking mail from sites that it deems are spam havens. It may also use the MAPS database(s) to block mail from spammers. These measures will affect the posting of mail directly to the lne CDR but will not affect the relaying of mail from the other CDRs. We do not get a feed from the CDR ssz, which is in the MAPS RSS database because it's an open relay. While we don't use the RSS currently, our alternate MX host does. If lne can't accept mail, then it goes to the MX host, which rejects mail from SSZ. Since this creates an ugly mail loop and neither our MX or SSZ are likely to change their policies, we don't get a feed from SSZ. This shouldn't affect CDR distribution though, as we'll get posts that originate from SSZ through the other CDRs. Lne.com runs the input to its CDR list through demime (http://scifi.squawk.com/demime.html) which deletes MIME attachments from mail. Demime leaves a note in the attachments place, so that recipients know that there was some cruft there. Lne.com does not archive cypherpunks messages. The Lne.com CDR strips out the "CDR:" tags in the Subject of messages that some other CDRs put in. If you have any questions that aren't answered by this message, send me email. Eric Murray ericm@lne.com Appendix: technical details. Here's the deep technical stuff, for those who're interested. Mail from other CDRs is sent to an address at lne.com, where it's processed by procmail. The procmail script checks for bounced mail and for dups, then sends the mail to the other CDRs, excepting the one that sent it. It also sends the mail to cypherpunks@lne.com, which is the list, run by majordomo. The list alias processes the mail first through a small script which cleans up the headers (i.e. removing CDR: and Re: Re: from Subject lines) and then passes it to demime. Demime strips out any MIME attachments, leaving a single line in their place to note what was there. Then the message is passed to majordomo itself. The majordomo config is pretty much standard. The allowed-posters list is seperate from the subscribers list, so we can use a list of all CDR subscribers, remailers, etc. Mail sent directly to the lne CDR goes to cypherpunks first, but the local address that runs through procmail is subscribed, so the mail also gets forwarded to the other CDRs. We generate the list of CDR subscribers by requesting a 'who cypherpunks' from each one, once a day. Some CDRs don't allow that command, so for those we ask that they send the list via a cron job. The list is processed along with a locall-maintained list of remailers and alternate addresses to make the list that's used for determing if a post is from a subscriber. The only modifications to Majordomo was a small one to allow wildcards in addresses, so we can allow posts from *@remailer.somewhere without having to add each different remailer user's address (for remailers that allow different user ids to show in the From: line). -- Eric Murray Consulting Security Architect SecureDesign LLC http://www.securedesignllc.com PGP keyid:E03F65E5
FYI I'm subscribed to cypherpunks from one address; I post to it from a second; my From: line is a third. I sometimes use a fourth address to post. I'm not sure how you're going to handle that, but you might want to consider semi-automated ways of extracting addresses from "known good" posts, to the extent that any of mine fit that category. -Declan On Thu, Feb 22, 2001 at 10:13:34AM -0800, Eric Murray wrote:
I've set up a new CDR that filters posts to it's subscribers- only posts from subscribers to other CDRs or known remailers (or subscriber's alternate addresses) will go to lne.com CDR subscribers.
Here's the list welcome message, which explains how it works in more detail. We're not connected to all the CDRs yet but as far as I can tell we're not missing any posts.
-------------------------------------------------------------------------
Welcome to the lne.com Cypherpunks Distributed Remailer This is welcome message v1.0, dated 2/22/01
Cypherpunks is a list for discussing cryptography and its impact on society. It is not a moderated list. The list operators are not responsible for the list contents. See the following pages for cypherpunks info:
Background/general explanation:
http://world.std.com/~franl/crypto/cypherpunks.html (read the two Manifestos-- Eric Hughes' and Tim May's-- for good background material).
Physical meetings:
http://www.cryptorights.org/cypherpunks/meetingpunks.html
Cypherpunks list archves:
http://www.inet-one.com/cypherpunks http://cypherpunks.venona.com
The Cypherpunks list is run in a distributed fashion. There are a number of Cypherpunks Distributed Remailers which handle distribution of the list. Different CDRs have different policies. http://einstein.ssz.com/cdr/index.html explains the CDR system.
To unsubscribe from this CDR, send the words "unsubscribe cypherpunks" (without the quotes) in the body of email to majordomo@lne.com. If you are subscribed to another CDR, you will have to send your unsubscribe to it.
The lne.com CDR is an attempt to reduce the spam that has invaded the cypherpunks list without blocking or delaying non-spam posts. The way we do that is to 1) drop posts which are not from a subscriber to any of the CDRs, and which are not from a known anonymous remailer 2) make an attempt to recognize spam from the contents of the mail message and mark spam as such.
These filtering methods affect only cypherpunks mail going to subscribers of cypherpunks@lne.com- for example, if someone sends mail to the CDR cypherpunks@cyberpass.com, the mail gets passed to all the CDR nodes inluding lne for distribution to subscribers. If the sender isn't a subscriber to any of the CDRs, the lne.com CDR node will not send the message to its subscribers. It will forward the message to the other CDRs however.
This also means that as a subscriber to the lne CDR, you may see replies from other CDR suibscribers to posts that you didn't see, because the were from non-subscribers.
Posts sent directly to the lne CDR need to be from a subscriber to any of the CDRs (or a known remailer) in order to be accepted and passed on to the other CDRs. I'd prefer to have it send everything along, but that'd require a whole lot of majordomo hacking and it's just not worth it.
Some cypherpunks posters post from addresses that are different from the one that they're subscribed with, causing their post to be dropped by lne.com. When this happens, I'll forward their post to cypherpunks@lne.com (headers intact) and add their alternate address to the list of posters. I will reserve the right to also delete an address I've added to the list if that address starts sending spam- this is to defeat an attack where the spammer sends a non-spam message to the list to get his address added, and then starts spamming the list. I doubt that I'll have to do this, but I'm mentioning it now in case it does happen.
The spam-recognition software in use at lne.com is described at http://www.lne.com/ericm/spammaster. In short, it looks for key phrases in the mail body (like "Dear Friend" and "This is not spam") and assigns points to the different phrases. The spam point total is put in the "X-spam" header added at the end of the mail headers. The mail is still sent to the list. The rating is there for lne.com CDR subscribers to use or not as they please. The spammaster config file that the lne CDR uses is available at http://www.lne.com/cpunk/spamfilter and there is more explanation of the software at http://www.lne.com/ericm/spammaster.
Other anti-spam measures: lne.com has a policy of blocking mail from sites that it deems are spam havens. It may also use the MAPS database(s) to block mail from spammers. These measures will affect the posting of mail directly to the lne CDR but will not affect the relaying of mail from the other CDRs. We do not get a feed from the CDR ssz, which is in the MAPS RSS database because it's an open relay. While we don't use the RSS currently, our alternate MX host does. If lne can't accept mail, then it goes to the MX host, which rejects mail from SSZ. Since this creates an ugly mail loop and neither our MX or SSZ are likely to change their policies, we don't get a feed from SSZ. This shouldn't affect CDR distribution though, as we'll get posts that originate from SSZ through the other CDRs.
Lne.com runs the input to its CDR list through demime (http://scifi.squawk.com/demime.html) which deletes MIME attachments from mail. Demime leaves a note in the attachments place, so that recipients know that there was some cruft there.
Lne.com does not archive cypherpunks messages.
The Lne.com CDR strips out the "CDR:" tags in the Subject of messages that some other CDRs put in.
If you have any questions that aren't answered by this message, send me email.
Eric Murray
ericm@lne.com
Appendix: technical details.
Here's the deep technical stuff, for those who're interested. Mail from other CDRs is sent to an address at lne.com, where it's processed by procmail. The procmail script checks for bounced mail and for dups, then sends the mail to the other CDRs, excepting the one that sent it. It also sends the mail to cypherpunks@lne.com, which is the list, run by majordomo. The list alias processes the mail first through a small script which cleans up the headers (i.e. removing CDR: and Re: Re: from Subject lines) and then passes it to demime. Demime strips out any MIME attachments, leaving a single line in their place to note what was there. Then the message is passed to majordomo itself.
The majordomo config is pretty much standard. The allowed-posters list is seperate from the subscribers list, so we can use a list of all CDR subscribers, remailers, etc. Mail sent directly to the lne CDR goes to cypherpunks first, but the local address that runs through procmail is subscribed, so the mail also gets forwarded to the other CDRs.
We generate the list of CDR subscribers by requesting a 'who cypherpunks' from each one, once a day. Some CDRs don't allow that command, so for those we ask that they send the list via a cron job. The list is processed along with a locall-maintained list of remailers and alternate addresses to make the list that's used for determing if a post is from a subscriber.
The only modifications to Majordomo was a small one to allow wildcards in addresses, so we can allow posts from *@remailer.somewhere without having to add each different remailer user's address (for remailers that allow different user ids to show in the From: line).
-- Eric Murray Consulting Security Architect SecureDesign LLC http://www.securedesignllc.com PGP keyid:E03F65E5
On Sat, Mar 03, 2001 at 01:15:25AM -0500, Declan McCullagh wrote:
FYI I'm subscribed to cypherpunks from one address; I post to it from a second; my From: line is a third. I sometimes use a fourth address to post.
You're not the only one who does that.
I'm not sure how you're going to handle that, but you might want to consider semi-automated ways of extracting addresses from "known good" posts, to the extent that any of mine fit that category.
Since the goal is to block spam, not to prevent all non-subscribed posters from posting, I put that in place before I announced the list. Bounces (from not being on the posters list) go to me. If it looks like it's not spam, I have a script which strips off the outer envelope and forwards the mail to the list, adding the contents of the From: line to a list of addresses that's sorted into the major posters list. So far I've added 39 addresses that way, 5 of which appear to be remailers. Of the rest, roughly half are people who're cross-posting from other lists like coderpunks or cryptography (or both), and the rest are people who're posting from alternate addresses. There's 2-3 spam emails for every list email. Previously I'd been filtering most of the spam. I had no idea it'd gotten this bad. -- Eric Murray Consulting Security Architect SecureDesign LLC http://www.securedesignllc.com PGP keyid:E03F65E5
At 8:20 AM -0800 3/3/01, Eric Murray wrote:
FYI I'm subscribed to cypherpunks from one address; I post to it from a second; my From: line is a third. I sometimes use a fourth address to post. .... So far I've added 39 addresses that way, 5 of which appear to be remailers. Of the rest, roughly half are people who're cross-posting from other lists
On Sat, Mar 03, 2001 at 01:15:25AM -0500, Declan McCullagh wrote: like coderpunks or cryptography (or both), and the rest are people who're posting from alternate addresses.
There's 2-3 spam emails for every list email. Previously I'd been filtering most of the spam. I had no idea it'd gotten this bad.
By the way, I've been subscribed to cypherpunks@lne.com for the past week or so. It's been a joy not to see the spam and junk and "can u help me mak bombz?" noise. (I was filtering a lot of this junk out by simply filtering all cypherpunks@toad.com posts into a special folder and then occasionally looking for posts by people I recognized, as sometimes even _good_ posters were posting to the toad.com address (usually in a reply-to-all mode, one assumes). But a lot of junk was still getting through.) Latency from when I post to lne.com to when I get my own message back is usually a couple of minutes. I have no idea what it is to other nodes. Probably not much longer. However, I see the address in my Reply-to-all is "cypherpunks@slack.lne.com". This is needlessly confusing, even if slack.lne.com resolves to lne.com. --Tim May -- Timothy C. May tcmay@got.net Corralitos, California Political: Co-founder Cypherpunks/crypto anarchy/Cyphernomicon Technical: physics/soft errors/Smalltalk/Squeak/agents/games/Go Personal: b.1951/UCSB/Intel '74-'86/retired/investor/motorcycles/guns
participants (3)
-
Declan McCullagh -
Eric Murray -
Tim May