Adam Back <aba@dcs.ex.ac.uk> writes:
Recap on old idea of using dejanews and altavista
However is another important reason not to use altavista or dejanews: it's not decentralised. If something sufficiently hot gets published they'll both be taken off line long enough to figure out how to disable access to eternity. Even if they can't do it ultimately because of sufficiently advanced public key text steganography, it'll disrupt the service.
If we aren't going to actually shell out money for a worldwide network of Eternity file servers, we have to use existing publicly available services as our distributed data haven. The more places we can stash things, the merrier. Currently, we post documents to Usenet, and keep them alive by reposting prior to the server expiring them. This is robust, not easily censored, and little effort. I see nothing wrong with having the server also examine Dejanews for Eternity documents, lightly stegoed to avoid the deletion of encoded data. To the extent that Dejanews serves as a more permanent repository of data than Usenet, fine. The worst thing that can happen is that documents get removed, in which case, the guardian of the document in question returns to keeping it alive on Usenet. All this is transparent to people using Eternity to read documents. Dejanews likely removes encoded data because of storage requirements, not because of content. Unless the Eternity business becomes a significant fraction of Usenet bandwidth, stego wars with Dejanews are unlikely.
A distributed deja-news replacement
So what are the storage requirements? What does dejanews have in their machine room? (Huge raid server? How many Gigs?).
Dejanews started in 1995. By 1996, they had 80 million articles in 15,000 Usenet newsgroups. They currently have 109 million articles. The Dejanews database is presently 180 gigabytes. They index all the "most active" Usenet newsgroups, minus binaries and some other controversial content. A week of a full Usenet feed is presently just a tad over 20 gigabytes, and is growing by leaps and bounds.
I'm wondering if 1000 academic nodes, small and large ISP nodes, and indivduals each contributing 1Gb each could out perform deja-news.
That's more storage than Dejanews currently has. Performance is another issue.
That'd be 1Tb. How long would 1Tb last at the being consumed at a rate of a full USENET feed?
Less than one year, if you kept everything.
On top of distributed news archive, build an eternity service
So with the eternal news archive, you now have everything you need to easily build an eternity server.
How practical is this?
Not very. You are tossing out one of the most useful aspects of Usenet news as a Distributed File System, namely acting as a "Giant FIFO" of what is interesting during any particular window of time.
How much interest is there likely to be in creating a full USENET archive as a distributed net based effort.
None at all. Keeping everything that had ever been posted to Usenet forever would be a giant waste of resources. What I suspect you need here is the "Distributed File System" of the future, which could carry Usenet as well as support Web, NFS, FTP, and all other popular protocols to access its content. It would be a giant FIFO with hundreds of gigabytes of capacity, with reliable transport, authentication, encryption, and a micropayment system for accessing and prolonging the lifetime of data stored on it. It would of course be uncensorable, as Usenet is today, and the Eternity Service could run on top of it, as well as a lot of other interesting things.
It's clearly got vastly larger storage requirments than storing the full set of eternity articles posted to USENET. However it has wider uses, and has lots of innocuous reasons to exist. I guess it's quite an interesting project in it's own right. You could start with just some newsgroups, and build up as a way to boot strap it. But it may be harder to generate enough interest to deploy this than to deploy enough eternity servers to do build a distributed archive of just eternity articles.
It's certainly interesting to think of what the ultimate DFS for the Internet might look like, given the various white papers on NC, thin clients, fat servers, and big pipes. But this isn't something that is going to happen overnight. To the extent that we employ Usenet in interesting new ways as a DFS today, we create working models of many of these concepts and demonstrate their feasibility.
How soon could it happen?
Who knows. What do people reckon the interest would be in a distributed complete USENET archive for it's own sake. For purposes of the archive cancels would not be applied. Indeed archive the cancels too, and the control messages for everyone to see for perpetuity who did and said what.
Well, the disk manufacturers would probably love it, and want it replicated every ten feet across the planet. I don't know about other people.
Perhaps it will be more realistic to build a distributed archive for eternity documents only first. If it is designed as a separate service, then it could be the same software to do either, just restrict it to alt.anonymous.messages, or just for alt.anonymous.messages passing a filter program.
Comments?
First of all, instead of thinking of this in terms of being the singing and dancing Usenet archive of all time, we would do much better if we thought of it as a protocol-independent Network-wide Distributed File System with configurable characteristics suited to a wide variety of applications. If we could put together a prototype with a small number of nodes, and demonstrate it could carry both Usenet and Eternity as well as provide NFS3 access to commonly requested Network binaries, all supported by an integrated micropayment scheme, that would go a long way towards making it a standard. Would anyone care to donate some storage, cycles, and a fast pipe to the Net? -- Mike Duvos $ PGP 2.6 Public Key available $ enoch@zipcon.com $ via Finger $ {Free Cypherpunk Political Prisoner Jim Bell}