"J.A. Terranson" wrote:

Great: if we go back to distribution, we can bring all of us back into the fold.

It was around 2005 that the last few nodes dropped off the old CDR, so it's been quite a while since we've been distributed. I went back through the list archives late last year in search of the history of the CDR. While I'm sure others can add more to the skeleton, what I found went into the Cypherpunk Wikipedia entry (with references to archived emails where possible). To be honest, I'm not sure distribution is a response to any real threat: to the extent that the Cpunks list is of interest, isn't it more sensible to keep it alive and monitored than to shut it down? Of course, as J.A. points out, there are plenty of other reasons, viz., different list policies at different nodes (reply-to:, subject modification, sender whitelisting, etc.)---and these create some problems that we'll have to address in designing CDRv2. So let's talk a bit about the CDR architecture. One obvious issue with the old one is that it was a bit rickety: at its heart it was a bandaid fractal built on top of procmail and Majordomo, and getting it set up took enough work that most people didn't want to do it. To this end, I suggest we build upon mailman this time around: the codebase is reasonably well maintained, plenty of people feel comfortable hacking in Python, and we automatically get more user friendliness in the subscriber and administrative interfaces. (In principle, an even better way to do this would be to cleanly separate the CDR functionality from the list management functionality so that individual nodes can decide on their own what list software to run. This probably ends up expanding the time cost of the project well in excess of the utility it generates, so my vote would be against making this a requirement.) One major question we need to address is the topology of the CDR network itself. From an implementation point of view it would be most straightforward if every node knew about every other node, but one could argue that this is too fragile. If we take that position, we have to solve a broadcast repeater problem: when a node receives a message, it needs to have some way of deciding what other nodes it forwards messages to. Obviously we'd like to do this in a way that doesn't result in a lot of useless echoes. (Recognizing that perfect is the enemy of good may be the better part of valor here; after all, we don't expect to have more than a handful of CDR nodes.) As before, we want to be able to have independent policies at each node at least with respect to: - subject line modification - header mangling (reply-to: etc) - attachments - sender whitelisting - other things I'm forgetting As far as I can tell, the ones that will require the most work are subject line modification and sender whitelisting. Mangling the subject line becomes a problem because eventually we end up with subjects like "[CDR] Re: [Cpunks] Re: [Cypherpunks] foo", which confuses a lot of mail clients. This isn't a problem for monolithic mailman setups because mailman knows not to add another prefix to outgoing messages that already have one, but the problem becomes more difficult when each node potentially has its own (possibly empty) prefix. This problem would have a trivial solution in the case that all nodes knew the tagging policy at all other nodes, but (as I point out above) we may prefer not to require nodes to know all other nodes in the CDR. (The fallback solution is the old one: manually construct an appropriate filter to demangle subject lines. Yech.) In the old CDR (possibly after the LNE.com modifications circa 2001), nodes had to expose their sender lists (at least to other nodes) to allow for member whitelisting. Obviously this is ripe for abuse, and I'd prefer to achieve this in a different way if possible. The goal here is to make it easy to verify that a given sender is subscribed to *some* node but difficult to enumerate all list subscribers. I can see how one might think of this as coddling the supposedly savvy and internet- hardened subscribers of the list, but I just don't see any reason to expose users to more abuse than necessary, especially when solutions to this problem have already been described and are nominally right in the cpunks wheelhouse! I have a skeleton architecture in mind that can be beaten into a shape capable of addressing the above issues, but as this email is getting long and I've doubtless forgotten several other important problems, let's discuss a bit more first. -=rsw

"J.A. Terranson" wrote:

Everyone sends to the node of their choice, the node sends to a broadcast repeater that knows the source, and sends to everyone else, after stipping any mailman specific things like tags, etc. The down side to this kind of dumb repeater is in the case of outages - the repeater will not know (or would it? I need to look at this in postfix) what to forward.

As far as I can tell this doesn't (yet) solve the problem of whitelisting subscribers to other nodes. However, we can add one more step and solve this: when a node receives an email from the repeater whose sender is a member of the node's local subscriber list, it bounces the message back to the repeater with an added header saying, in effect, "I vouch for this sender." Other nodes employing sender whitelisting would ignore the first email, since its sender isn't locally whitelisted and it lacks the aforementioned node-auth header, but would presumably forward the second email, assuming they chose to trust the node that is vouching for the sender. Nodes with no whitelisting policy could safely ignore the second email by filtering out duplicate msgids or something similar. I'm not totally in love with the master repeater scheme, though. Notwithstanding my previous comments regarding the supposed threat model behind the CDR's original conception, as long as we're paying the fixed cost of setting up a new system we may as well get *some* additional reliability out of it, right? -=rsw

On Mon, Jan 20, 2014 at 12:48:10PM -0600, J.A. Terranson wrote:

I think a two-way broadcast repeater is the answer:

Everyone sends to the node of their choice, the node sends to a broadcast repeater that knows the source, and sends to everyone else, after stipping any mailman specific things like tags, etc. The down side to this kind of dumb repeater is in the case of outages - the repeater will not know (or would it? I need to look at this in postfix) what to forward.

I think the key is getting the incoming emails to the repeater *prior* to mailman intervention.

maybe lamson/librelist with some changes in front of postfix is exactly what you're looking for -- pgp: https://www.ctrlc.hu/~stef/stef.gpg pgp fp: FD52 DABD 5224 7F9C 63C6 3C12 FC97 D29F CA05 57EF otr fp: https://www.ctrlc.hu/~stef/otr.txt

On Mon, 20 Jan 2014, grarpamp wrote:

Some notes...

- Some subset of nodes should commit to carrying all messages

Thats going to be strictly up the node owners. The node I anticipate here will likely carry all messages, but thats stricly a choice for the node operator.

- Consider instead deploying NNTP via Diablo with a list of current readers and a howto on each of the splash pages. Don't try to tell me cypherpunks cant grok this. Certainly you can DSPAM/spamassasin/moderate and all other things with this too. From an administrative point of view, nntp is a *major* PITA. I know, I used to run a full feed over DS3 (which could barely take the traffic back in '99!). NNTP is a really fucked up protocol, with tons of room for abuse.

- Or a hash based interchange backend? ie: message body synchronization between nodes. Mail is ugly, see nntp above. Thats a really good idea, but implementation across multiple nodes may prove difficult. I'll give it some thought.

- Umm, mailbox/maildir/nntp archives from day one, please, seriously. Seriously, if you want to support it, get a box and be a provider. I'll happily support mailman and its native archives, but it'll be a cold day in hell before I interface NNTP again, or worry about maildirs for every user: forget it. If you are hereby *seriously* offering to provide these, then put up a box on a permanent connection, and you can be the designated module writer for it :-)

//Alif -- Those who make peaceful change impossible, make violent revolution inevitable. An American Spring is coming: one way or another.

grarpamp wrote:

I mean ethically someone should do it. But unless a filtering node agrees to forward messages filtered from its own list through its backend and out to all the other nodes for them to make their own choices, there will be holes...

Similarly for carrying all backend received messages. carries posted: yes/no carries backend: yes/no despam mechanism [posted/backend]: describe here moderation [posted/backend]: policy here peers with: who

In CDRv1, all nodes would in principle forward all messages to other nodes, only filtering the feed going to that node's own users according to the local filtering policy. It makes sense to ask nodes to publish details of their local policies. I think personally I would hesitate to peer with any node that didn't forward everything and let me apply my own filter. I assume most other operators would as well, so practically speaking no one would run a node that didn't (claim to) forward everything. Next question: how paranoid are we, i.e., do we attempt to enforce this policy somehow? This goes beyond fault tolerance towards attempting to solve the problem of enforcing peering contracts with untrusted CDRv2 nodes, which is clearly a more... intersting one. I have been busy with real life, and haven't dedicated much more time to thinking about this. I'm hopeful that tomorrow I will have the opportunity to do so at least a little bit. -=rsw

coderman wrote:

keep it simple: NO FILTERING

if you can read this you are tall enough to filter/tag/label your own self.

I'm not sure if you mean on the backend (between nodes) or on the frontend (local delivery to node subscribers). In the former case, I agree: it makes sense to just have a blanket policy that peered nodes are expected to forward everything they receive at their ingress address (allowing other nodes to apply their own local filtering policies). If you mean the latter, I disagree that zero filtering is a good approach to running a node (but the whole point is that we can make both available and let the users choose). In practice, I'm quite certain that readership on the present cypherpunks envisagement would drop precipitously if I turned off sender whitelisting. There's a difference between "I can do this" and "I will do this because it is worth my time." The fact is, once the cost of being a subscriber exceeds its utility, a rational person will unsubscribe; a node with sender whitelisting (with explicit whitelisting for anonymous remailers) achieves a balance that, empirically, is worthwhile for most people: recall the rush to LNE.com when Eric introduced this. More to the point, the willingness of a person to sink time into wading through list detritus is no indication of his or her value as a contributor. Fundamentally, subscriber lists are a good metric by which to judge whether a particular message should or should not be delivered; it is therefore useful to build the notion of cross-node sender whitelisting into CDRv2 in a way that cannot be trivially abused. Of course, all sender whitelisting leaks *some* information about subscribers; the goal is just to do no worse than a monolithic list.

...

This goes beyond fault tolerance towards attempting to solve the problem of enforcing peering contracts with untrusted CDRv2 nodes, which is clearly a more... intersting one.

this is not simple, and not recommended.

No doubt about it; I'm certainly not volunteering to do any such thing! -=rsw

coderman wrote:

correct. sender whitelisting is useful!

what can others do to help?

I'm nailing down a few last details of a proposed CDRv2 architecture. I'll follow up with details tomorrow evening-ish and then we can beat up on it a bit. -=rsw

"Riad S. Wahby" wrote:

As proposed, a description of my proposed architecture. I actually have

Erm, "as promised." -=rsw