----- Begin Included Message ----- From: <monier@pa.dec.com> Subject: Re: Alta Vista searches WHAT?!? Date: Tue, 16 Jan 96 14:47:51 -0800 X-Mts: smtp Hum, one more time. Scooter, the robot behind Alta Vista, follows links, and only follows links. If the "directory browsing" option is enabled on a server, and someone publishes the URL for a directory, then the robots gets back a page of HTML which lists every file as a link, but that is not intentional. And yes, this has led to embarrassing situations, but again, it's not intentional. In the absence of strong conventions about directory names or file extensions it is hard for a robot to exclude anything a-priori. I wish it was easier... To keep a document private, list it in /robots.txt, password-protect it, change the protection on the file, or simpler: do not leave it in your Web hierarchy. Can you imagine what happens when someone uses / as web root, exposing for example the password file? It has happened! Remember that what a robot does, anyone with a browser can do: find this private file and then post to usenet for example, robots have no magic powers! The bottom line is that the usual danger is not aggressive robots, but clueless Web masters. --Louis ----- End Included Message -----