Alta Vista searches WHAT?!?

Ed Carp @ TSSUN5 ecarp at tssun5.dsccc.com
Wed Jan 17 08:02:44 PST 1996



----- Begin Included Message -----

From: <monier at pa.dec.com>
Subject: Re: Alta Vista searches WHAT?!? 
Date: Tue, 16 Jan 96 14:47:51 -0800
X-Mts: smtp

Hum, one more time.

Scooter, the robot behind Alta Vista, follows links, and only follows links.
If the "directory browsing" option is enabled on a server, and someone publishes
the URL for a directory, then the robots gets back a page of HTML which lists
every file as a link, but that is not intentional.  And yes, this has led to
embarrassing situations, but again, it's not intentional.
In the absence of strong conventions about directory names or file extensions
it is hard for a robot to exclude anything a-priori.  I wish it was easier...

To keep a document private, list it in /robots.txt, password-protect it, change
the protection on the file, or simpler: do not leave it in your Web hierarchy.
Can you imagine what happens when someone uses / as web root, exposing for 
example the password file?  It has happened!

Remember that what a robot does, anyone with a browser can do: find this private
file and then post to usenet for example, robots have no magic powers!

The bottom line is that the usual danger is not aggressive robots, but 
clueless Web masters.


		--Louis


----- End Included Message -----







More information about the cypherpunks-legacy mailing list