Replacing corporate search engines with anonymous/decentralized search

[coderman]

On Sun, Dec 29, 2013 at 12:17 AM, Jesse R. Taylor
<jessetaylor84 at riseup.net> wrote:
> Recently there has been a lot of focus on the importance of developing more
> secure alternatives to email, instant messaging, browsing, etc. ... but I've
> seen very little focus on the need for development of alternatives to
> corporate search engines.

decentralized search (not just not-corporate search) persists as one
of the great
practical challenges in peer to peer networking.

i have more to say later, but one effort from back in early 2000 is alpine:
  https://peertech.org/alpine
 inside the 2004 snapshot there is also docs and implementation of
feedbackfs which is used to gather implicit feedback on recommendation
/ discovery of file based resources.


alpine is explicitly highly connected, flatter than not network
topology to improve robustness in the face of failure and active
attacks, and to avoid limitations inherent in many connection oriented
operating system facilities/sockets.


i am not quite an impartial party ;)
 but other approaches which are not a feasible replacement include:
 - the old skewl (mostly)flooding broadcasts like gnutella
 - fragile, hard to defend constructs like DHTs as keyword indexes
 - aggressive caching with local search (110% useful, but not sufficient alone)
 - distributed (but better somehow) search engines on darknets, etc.
these are more about search privacy or deep search more than
decentralized search.



> But I've seen very little information about practical/simple options that
> are available for anonymous and decentralized Internet search software.
> ... What are the major barriers to creating simple tools [...]
> [... for] anonymous, p2p web search (even if it's much slower
> than centralized search) and break away from using corporate search? Which
> current efforts to create decentralized search seem most promising to you
> from a privacy/security standpoint?


the longer discussion is how to make decentralized search useful.
"Google style" search has a terrific performance advantage over
decentralized designs by brute force.

however, take advantage of massive endpoint / peer processing and
resources combined with implicit observational metrics for reputation
and recommendation, inside a well integrated framework for resource
discovery in usable software, and you have something more robust and
more effective than "Google style" could ever provide.

this is quite the trick, however!  despite an inter-operable component
model interface, and dynamic runtime module support to extend
discovery and wire protocol extensions, and other intentional efforts
at encouraging adoption and integration, alpine failed to bootstrap.
(i did many things wrong, but those things i did at least make
conscious effort to do right.  did i mention this is a hard problem?
:)


this project has been excavated from archives, and will receive
maintenance upgrades[0] at minimum and significant improvement a
possible option, depending.


best regards,


[0] maintenance work for testable alpine builds
 - fix/improve g++ usage.
 - add IPv6 support. (specifically ORCHID addrs for darknet search)
 - update feedbackfs to latest fusefs bindings
 - update inotify bindings in feedbackfs
 - multiple-socket support, multi-addr discovery