Re: Spiderspace

At 11:38 AM 1/16/96 -0800, you wrote:

I've been thinking a lot about the problems and opportunities that are coming up as more and more "spiders" (Web searchers, crawlers) are indexing directories and files on systems they can find.

<a snip of agreement> I haven't checked my logs for Alta Vista but many of the spiders do follow the robot exclusion standard. A simple text file that should be retrieved first by any spider that explains where one may not go. The standard is fully explained at: http://info.webcrawler.com/mak/projects/robots/norobots.html But this does nothing for those items sitting in more traditional public space. Obscurity is no longer security.