Spiderspace

[Bill Stewart]

>... I was under the impression that the only documents that most web crawlers
>will search are documents that are link-accessible.  Are you saying that this
>isn't true?  Are you saying that Alta-Vista will search EVERYTHING that's
>publicly accessible, whether by anonymous FTP or web?

Don't archie servers already pick up the anonymous ftp fairly well?
Also, aside from no-robots conventions, you can build a cgi program for
access to files that might be more effective at blocking searches
while still preserving access.

Also, it wouldn't be hard for a web-crawler to follow ftp links,
as long as the root of an anon-ftp site is pointed to by a URL somewhere.
#--
#				Thanks;  Bill
# Bill Stewart, stewarts at ix.netcom.com, Pager/Voicemail 1-408-787-1281
#
# "Eternal vigilance is the price of liberty" used to mean us watching
# the government, not the other way around....