whitehouse.gov/robots.txt
Anatoly Vorobey
mellon at pobox.com
Wed Dec 10 04:59:07 PST 2003
On Wed, Dec 10, 2003 at 12:56:24PM +0100, Eugen Leitl wrote:
> Can somebody with a webspider crawl these documents, and put it up
> on the web?
>
> http://www.whitehouse.gov/robots.txt
All or nearly all of them are duplicates of same documents
elsewhere in the directory tree; "X/text/" and "X/iraq/" are
supposed to be copies of "X/", with images removed in the first
case. I suspect that downloading them all would just confirm that.
--
avva
More information about the Testlist
mailing list