On Sun, Dec 29, 2013 at 12:17 AM, Jesse R. Taylor <jessetaylor84@riseup.net> wrote:
Recently there has been a lot of focus on the importance of developing more secure alternatives to email, instant messaging, browsing, etc. ... but I've seen very little focus on the need for development of alternatives to corporate search engines.
decentralized search (not just not-corporate search) persists as one of the great practical challenges in peer to peer networking. i have more to say later, but one effort from back in early 2000 is alpine: https://peertech.org/alpine inside the 2004 snapshot there is also docs and implementation of feedbackfs which is used to gather implicit feedback on recommendation / discovery of file based resources. alpine is explicitly highly connected, flatter than not network topology to improve robustness in the face of failure and active attacks, and to avoid limitations inherent in many connection oriented operating system facilities/sockets. i am not quite an impartial party ;) but other approaches which are not a feasible replacement include: - the old skewl (mostly)flooding broadcasts like gnutella - fragile, hard to defend constructs like DHTs as keyword indexes - aggressive caching with local search (110% useful, but not sufficient alone) - distributed (but better somehow) search engines on darknets, etc. these are more about search privacy or deep search more than decentralized search.
But I've seen very little information about practical/simple options that are available for anonymous and decentralized Internet search software. ... What are the major barriers to creating simple tools [...] [... for] anonymous, p2p web search (even if it's much slower than centralized search) and break away from using corporate search? Which current efforts to create decentralized search seem most promising to you from a privacy/security standpoint?
the longer discussion is how to make decentralized search useful. "Google style" search has a terrific performance advantage over decentralized designs by brute force. however, take advantage of massive endpoint / peer processing and resources combined with implicit observational metrics for reputation and recommendation, inside a well integrated framework for resource discovery in usable software, and you have something more robust and more effective than "Google style" could ever provide. this is quite the trick, however! despite an inter-operable component model interface, and dynamic runtime module support to extend discovery and wire protocol extensions, and other intentional efforts at encouraging adoption and integration, alpine failed to bootstrap. (i did many things wrong, but those things i did at least make conscious effort to do right. did i mention this is a hard problem? :) this project has been excavated from archives, and will receive maintenance upgrades[0] at minimum and significant improvement a possible option, depending. best regards, [0] maintenance work for testable alpine builds - fix/improve g++ usage. - add IPv6 support. (specifically ORCHID addrs for darknet search) - update feedbackfs to latest fusefs bindings - update inotify bindings in feedbackfs - multiple-socket support, multi-addr discovery