[ntp:questions] Re: Lists of public NTP servers

Danny Mayer mayer at gis.net
Thu Nov 13 23:10:06 UTC 2003

Steve Kostecke <kostecke at ntp.org> wrote in message news:<slrnbr4nas.1a0.kostecke at stasis.kostecke.net>...
> On 2003-11-12, Dale R. Worley <worley at theworld.com> wrote:
> > Brad Knowles <brad.knowles at skynet.be> writes:
> >
> >> Once again, I will say that anything in the public twiki space is
> >> totally open to world-wide access, including any and all known search
> >> engines.
> >
> > I fear that this conversation is veering severely from what should be
> > the actual point: Despite that the TWiki can theoretically be indexed
> > by any spider, in practice it is not.
> The NTP TWiki is, with a few exceptions, open to spiders. I have tested
> this with wget (which, by default, obeys robots.txt directives).
> The NTP TWiki was last visited by crawler10.googlebot.com at
> 12/Nov/2003:04:16:05 +0000. But we're not appearing in the Google search 
> results yet.
> <snip>
It can take as much as a month for a page or pages to appear in Google.
That's because it's not practical to update the indexed database in real
time and they tend to only get updated once in a while. This is based
on my internal knowledge of what we used to do in reloading AltaVista
search. Things may have changed a great deal, but only news is likely to
be indexed and made available in real time.


More information about the questions mailing list