Google and the Net

[Declan McCullagh]

But dynamic sites are a problem. Search engines are often
reluctant to index through cgi scripts because of the possibility
of infinite loops of on-the-fly generated pages with unique URLs.
Some folks call this the hidden web.

-Declan


On Sat, Oct 20, 2001 at 11:07:24PM -0400, Yeoh Yiu wrote:
> David Honig <honig at sprynet.com> writes:
> 
> > At 11:23 PM 10/9/01 -0400, Declan McCullagh wrote:
> > >Huh? There's little to search with when it comes to images, unless
> > >you pull from the surrounding content, which Google does.
> > 
> > But google doesn't save the images, including navigational images
> > (which tend to be .gifs) nor does it -traversal -crawl
> > 
> > And the dynamic (CGI) site problem.
> 
> Dynamic sites should not be a problem,
> unless they require users to submit forms
> before they see the real content.
> 
> A google cache of a dynamic site eg omor.com 
> might have something like
> 
> 'This page Optimized for 216.239.46.66, googlebot(at)googlebot.com,
>  using Googlebot/2.1 (+http://www.googlebot.com/bot.html) .
> 
> YY