search engine improvement

17 Dec 2003

      -----BEGIN PGP SIGNED MESSAGE-----

Keywords: distributed ratings systems, search engines, spiders, 
spiderspace, idea futures, The Shockwave Rider, John Brunner

You know there is a trick that might greatly improve the 
effectiveness of a search engine at almost no cost to the end
user.  It is the well-known heuristic of "If Person A likes X
and Y, and Person B likes X, then Person B probably likes Y.",
combined with passive polling (which is getting information
about people's opinions just by watching their actions, instead
of by asking them).

A first simple implementation would keep a table of the pages
that people choose, keyed from the query that they originally
submitted.  Those pages that people choose most frequently from
the list of matching pages (and/or those pages that people
"stop" on-- that they do _not_ follow by further searching),
would get bumped up a little in the list.

This would be massively expensive in networking, storage, and
computation, giving those hi-tech Alpha clusters at AltaVista
something to do...  :^)

There are plenty of extras and refinements that could be added
(for example, put some keywords identifying your "affiliation"
in a separate field.  It will only consider the results from
other people who entered the same affiliation keywords when
weighting your search results.).  And there are some good topics
for further discussion, such as is it worthwhile to distinguish
between "relevancy" and "value"?

I don't have a comprehensive list of people who are already
working on this area (distributed ratings) (if I did, I might 
have Cc:'ed them), but I know that many people are.  I hope that
they and the search engine people get together and make cool
stuff soon.

There is the interesting issue of whether this will cause
self-reinforcing "degeneration", where people (or an
"affiliation"-keyed group of people) accidentally overlook a
worthy page early in the game, and then, using each other's
behavior to influence their own, reinforce that mistake.

As a final attribution note:  John Brunner thought of this idea
idea in his prophetic novel _The Shockwave Rider_ in the 75.  
There is a wonderful line which I can't find right now, about 
how it turned out to be a flywheel instead of an oracle, merely 
aggregating human mistakes and successes.

Regards,

Bryce

P.S.  ObCryptoRelevance:  Um...  you could get paid for your
ratings using Chaumian ecash, and even have your ratings popped
into the right "affiliation" using Chaumian credentials...

P.P.S.  CryptoRelevance isn't very Ob anymore, is it?  Just as
well, IMESHO.

search engine improvement

bryce＠digicash.com