Spiderspace

[John F. Fricker]

At 11:38 AM 1/16/96 -0800, you wrote:
>
>I've been thinking a lot about the problems and opportunities that are
>coming up as more and more "spiders" (Web searchers, crawlers) are indexing
>directories and files on systems they can find.

<a snip of agreement>


I haven't checked my logs for Alta Vista but many of the spiders do follow the robot exclusion standard. A simple text file that should be retrieved first by any spider that explains where one may not go.

The standard is fully explained at:

http://info.webcrawler.com/mak/projects/robots/norobots.html

But this does nothing for those items sitting in more traditional public space. Obscurity is no longer security.