[ntp:hackers] U Wisconsin

David L. Mills mills at udel.edu
Thu Jun 26 18:17:42 PDT 2003


Dean,

I did remember your message, but don't believe the solutions you propose
are workable. The three computer time protocols have very different
names and user communities, even if some think they are all the same
thing. For instance, I am told TCP/TIME is the medium required for stock
trades. I am also told NIST servers cannot abandon the gadget protocols
and must support TCP versions of TIME and DAYTIME. So, changing names in
the standards or creating a new one are a surprisingly complex process.
Changing host address/port works for only the short term. If we build it
they will come.

We have had discussions on this issue many times in the last twenty
years, but your experience, mine and the U Wisconsin folks suggest we
might have to do something really serious. Just about every defense I
can imagine is already in NTPv4 with the only exception handing out
bogus time values to known perps. My official recommendation to the U
Wisconsin folks was to fire up their legal staff and sue the bastards.
Precedent is the Internet Worm some years ago when Robert Morris was
sued by the feds for theft of service from DoD computers and fined
bigtime.

In point of fact, as pointed out to the U Wisconsin folks, the current
NTPv4 defenses were never designed to work 700,000 packets per second
from 285,000 different perps, most of which didn't configure their
firewalls to accept NTP reply packets. I did recommend that every
product connected to the public Internet be equipped with a deadman
switch that periodically sends its version number to the manufacturer's
web site and sounds a klaxon if the site has a required update. Does
Windows Update come to mind? After a suitable interval and no update,
the deadman becomes a dead man. I've thought seriously about
incorporating this feature/punishment in NTP.

Probably the only good thing that really can be done is an easy to use
scheme that can be implemented blindly by folks having no idea how it
works and don't care about abuse. Remember, the problem is not
specifically ganging up on any particular server, but the flood the gang
creates. A good and useful thing would be a volunteer to join the Corps
and rewrite the LRU code as a patricia trie and provide for efficient
expansion to 200,000 addresses or so, assuming resources available.

I've lost track of the pool.ntp.org guys and the various scripts and
pearls of perl. This stuff should be on/linked at the web page. A useful
project would be to collect these tools on maccarony and build a web
tool to use them. This should be widely distributed via newsgroups and
especially to developers. Lots of stuff has already been done; the stuff
should be collected and made conveniently available.

So much preaching, so little time.

Dave 

"Dean Gibson (NTP Time Administrator)" wrote:
> 
> I must say, not only am I not surprised, but I'm surprised that you're
> surprised.
> 
> The abuse that has been heaped on my little NTP server from
> misconfigured SNTP clients is legion.  Just two weeks ago I found a
> single IP address (in Greece) that in 36 hours had sent over 100,000
> DNS lookups for "time.ultimeth.net", an NTP hostname that we abandoned
> almost three years ago, and mapped to 127.x.x.x to satisfy thirsty DNS
> requests.  We had to find a different secondary DNS provider due to
> complaints from one provider about DNS storms for "time.ultimeth.net",
> and I had to switch to an unlimited rate ISP rather than the cheaper
> metered-rate ISP I had been using.
> 
> My limited research shows that our hostname "time.ultimeth.net" is
> installed in the software as one of the choices.  The author of this
> package, not being content to just implement an SNTP that allows
> polling intervals as little as one second or less, apparently also
> decided to directly do his own DNS queries for each SNTP poll
> (complete w/ NO caching), rather than using the local resolver.  May
> SNTP rot in hell.
> 
> That's just one incident, and many of you on this mailing list have
> heard me play this record before.
> 
> So, what do we do about it?
> 
> In the case of U Wisconsin, it's very clear:  NorTel Networks gets to
> pay U Wisconsin's Internet access fees in perpetuity.
> 
> For the rest of the NTP world who will increasingly suffer under this
> nonsense, I'll repeat my suggestion made previously here on January
> 21st of this year (copied below):  NTP needs to be renamed (eg,
> "Precision Time Protocol") and the port number changed from 123, plus
> other changes intended to make it much more difficult for
> unauthorized, poorly-written clients to participate.  Dave suggested
> at the time making public-key encryption mandatory;  I'm coming around
> to that
> 
> -- Dean
> 
> David L. Mills wrote on 2003-06-26 10:55:
> 
> > Guys,
> >
> > I find myself on the review team for an incident taking place at U
> > Wisconsin/Madison.  Apparently, the Netgear folks have manufactured
> > some 700,000 routers with embedded SNTP clients configured to use
> > the public U Wisconsin NTP server.  The server address is
> > unchangeable and the client cannot be disabled.  If that isn't bad
> > enough, if the client gets no replies, it starts sending packets at
> > one-second intervals until forever and without backoff.
> >
> > The U Wisconsin folks determined some 285,000 different IP addresses
> > are now sending between 300 and 700 packets per second requiring
> > between 150 and 400 megabits per second.  Apparently, the prrincipal
> > eason for this flux is misconfiguration of the firewall component of
> > the router.  This is costing them $266 per megabit per day.
> >
> > The Netgear folks were slow to respond until U Wisconsin folks
> > emailed the entire senior management and others known to be U
> > Wisconsin alum.  Netgear says they have no way to recall those
> > routers and no way to insure the products are updated from the web
> > site.  The products cost between $20 and $40 depending on rebate.
> >
> > U Wisconsin have considered several ways to deflect the tide, the
> > most promising may be noting the source port 23457 unique to these
> > products and tossing them at the doorstep.  The products do not use
> > DNS and are not configurable.  Another way considered is to
> > configure a subnet visible to BGP and convince the ISPs to punch
> > holes in the routing fabric.  Send money.
> >
> > I never thought it could get as bad as that.  My reasoned
> > recommendation was to fire up the lawyers and sue the bastards for
> > costs and punitive damages and to injoin the company from selling
> > any products until proved safe.  There is apparently some standards
> > group that allegedly reviews and certifies new products for Internet
> > use.  The Netgear products were all certified, which surely says
> > nothing about the standards group.
> >
> > Include me in any replies; I am not on any ntp.org list.
> >
> > Dave
> >
> >> To: hackers at ntp.org
> >> From: "Dean Gibson (NTP Time Administrator)"
> >> <ntp-admin at ultimeth.net>
> >> Subject: Abuse ...
> >> Date: Tue, 21 Jan 2003 11:14:12 -0800
> >>
> >> This is the problem that I started warning about over a year ago.
> >> Our solution was to take our NTP servers "prior permission only"
> >> 15 months ago and change hostnames and IP addresses;
> >> nevertheless, we still have unauthorized hosts that attempt to
> >> access the old IP address, and their attempts top the list when
> >> "ntpdc -c monlist" is sorted by usage.
> >>
> >> The problem is complex:
> >>
> >> 1. A number of web sites have copied the ntp.org lists of stratum
> >> 1 & 2 servers to their own web site, making it virtually
> >> impossible to get off all the lists around the world.
> >>
> >> 2. Further, at least one software author, Graham Mainwaring
> >> <graham at mhn.org>, distributes (via SourceForge.net) a Windows NTP
> >> client named "NetTime", with a list of NTP servers built in to the
> >> configuration file;  the list of servers is from June, 2000!
> >> Despite my request to Mr. Mainwaring, and his agreement to do so
> >> one year ago, he has not yet removed my server entry from his
> >> configuration file, nor does he apparently see anything wrong with
> >> capturing the list of servers and putting them in his
> >> configuration file.  Requests to SourceForge.net to remove
> >> "NetTime" from distribution have gone unheeded.  SourceForge.net
> >> is particularly incompetent in this matter.
> >>
> >> 3. At least some of the software out there allows the user to
> >> specify very small polling intervals;  we have seen polling rates
> >> from a single client as high as 5 per second.
> >>
> >> 4. Users are confused because the "Daytime" (port 13), "Time"
> >> (port 37), and "NTP" (port 123) protocol all describe themselves
> >> as a "Network Time Protocol", leading naive users to believe that
> >> NTP servers will support their client software.  We had a number
> >> of "Daytime" and "Time" protocol clients sending massive amounts
> >> of packets.  We have had to ask potential users for the
> >> name/version of their NTP client software;  and we still have to
> >> reject requests that specify "rdate"!
> >>
> >> 5. Tracking down unauthorized users is difficult;  ISPs presently
> >> don't regard unauthorized NTP access as abuse.
> >>
> >> If my experience is any guide, I think that CSIRO.AU's request to
> >> remove their servers from the public listings, will (sadly) have
> >> little effect on their traffic.  Changing/removing DNS names will
> >> have little effect, as many NTP clients run for months without
> >> being rebooted, and many other NTP clients (for access control
> >> reasons) specify the IP address of an NTP server, despite the
> >> listing's request to use DNS.  Put bluntly, CSIRO.AU is hosed in
> >> the short run;  the best long term solution is to remove the DNS
> >> entries and move the servers to new IP addresses, and even that
> >> will leave behind a huge group of hosts around the world blindly
> >> sending packets to the old IP addresses for years to come.
> >>
> >> The best real solution that I can think of, is to fundamentally
> >> change the NTP protocol:
> >>
> >> 1. Change the name of the protocol so that it doesn't have the
> >> "generic" name of "Network Time Protocol".
> >>
> >> 2. Go to a new port number (is 24 available?).  This will allow
> >> network switches to reject old NTP packets wholesale;  this should
> >> kill off most of the unmaintained and poorly written 3rd-party NTP
> >> clients.
> >>
> >> 3. Design the new NTP so that the "restrict" command can be used
> >> to specify thousands of users, or some equivalent mechanism.
> >> Cause the "server" command to be able to create an associated
> >> "restrict" entry, so that DNS lookups for the "server" command
> >> result in the same IP address for both the "server" and associated
> >> "restrict" data entries in the server's memory.
> >>
> >> 4. Inform anyone who requests a public NTP server listing, that it
> >> is a fool's errand to not get contact information from EVERY
> >> client before allowing access.
> >>
> >> Because, if we don't do something soon, the situation will be even
> >> more out of control than it is now.
> >>
> >> Sincerely, Dean Gibson
> >>
> >> David L. Mills wrote on 2003-01-20 20:46:
> >>
> >> > Guys,
> >> >
> >> > I was afraid it would come to this. The most disturbing issue is
> >> > that the abuse gets worse when the perp is call-gapped. I've
> >> > heard this from other sites.
> >> >
> >> > Dave
> >> >
> >> >>  -------- Original Message --------
> >> >>  Subject: RE: Listing of CSIRO NML stratum 1 NTP servers
> >> >>  Date: Tue, 21 Jan 2003 09:50:39 +1100
> >> >>  From: Peter.Fisk at csiro.au
> >> >>  To: mills at udel.edu, Michael.Wouters at csiro.au
> >> >>  CC: Peter.Fisk at csiro.au
> >> >>
> >> >>  Dear Dr Mills,
> >> >>  Thank you for the response.
> >> >>
> >> >>  For reasons which I expand on below, I would appreciate it if
> >> >>  you could remove the entries from your listing as soon as
> >> >>  possible.
> >> >>
> >> >>  We are not leaving the timekeeping business or the NTP
> >> >>  business; our problem is that despite the service area being
> >> >>  shown as limited (by request) to AARNET on your listings, we
> >> >>  have more than 200000 clients accessing these servers each day
> >> >>  from outside Australia. The network charges for this traffic
> >> >>  are becoming prohibitive, the traffic is doubling over a
> >> >>  timescale of less than eight months and we must now take some
> >> >>  action.
> >> >>
> >> >>  A further issue we face is that denying service to clients
> >> >>  outside our service area either by failing to reply or sending
> >> >>  the "kiss of death" packet results in a massive (up to 100-fold
> >> >>  in some cases) increase in incoming traffic, further increasing
> >> >>  our costs. In fact, when we performed an experiment last year
> >> >>  to deny responses to NTP packets from Japan, this produced such
> >> >>  a massive increase in traffic that AARNET thought it was a DOS
> >> >>  attack on the Australian network.
> >> >>
> >> >>  It seems that some authors of ntp clients do not write their
> >> >>  software to deal with the lack of a response, or a "kiss of
> >> >>  death", packet in a sensible way. The potential financial
> >> >>  liability for this traffic, especially in the event that we
> >> >>  have to temporarily or permanently take one of our servers
> >> >>  off-line, is completely unacceptable.
> >> >>
> >> >>  We are therefore looking at ways to limit the traffic on our
> >> >>  servers from international sources, and the first and most
> >> >>  obvious method is to remove the addresses from internationally
> >> >>  recognised listings such as yours. The extent of our further
> >> >>  action will be dependent on the effect of this removal.
> >> >>
> >> >>  I think this problem will sooner or later face all providers of
> >> >>  open access NTP services, and that organisations such as ours
> >> >>  will be unable to provide such a service unless coordinated
> >> >>  action is taken by the network community to enforce reasonable
> >> >>  behaviour.
> >> >>
> >> >>  Regards, Peter Fisk
> >> >>
> >> >>  --------------
> >> >>  ------------------------------------------------------------
> >> >>  Dr Peter Fisk
> >> >>  Section Manager: Standards for RF, Microwave, Time and
> >> >>  Frequency
> >> >>  National Measurement Laboratory
> >> >>  CSIRO Division of Telecommunications and Industrial Physics
> >> >>  PO Box 218, Lindfield NSW 2070
> >> >>  Sydney, Australia
> >> >>  (street address: Bradfield Rd, Lindfield NSW 2070)
> >> >>  Phone: (61 2) 9413 7221
> >> >>  Fax (61 2) 9413 7631
> >> >>  Mobile (61) 0418 693 962
> >> >>  --------------------
> >> >>  ------------------------------------------------------
> >> >>
> >> >>
> >> >> > -----Original Message-----
> >> >> > From: David L. Mills [mailto:mills at udel.edu]
> >> >> > Sent: Tuesday, 21 January 2003 00:58
> >> >> > To: Wouters, Michael (CTIP, Lindfield)
> >> >> > Cc: Fisk, Peter (CTIP, Lindfield)
> >> >> > Subject: Re: Listing of CSIRO NML stratum 1 NTP servers
> >> >> >
> >> >> > Michael,
> >> >> >
> >> >> > Thanks. In the next update. I hope this doesn't mean you guys
> >> >> > are out of the timekeeping business.
> >> >> >
> >> >> > Dave
> >> >> >
> >> >> >>  "Wouters, Michael (CTIP, Lindfield)" wrote:
> >> >> >>
> >> >> >>  Dear Dr Mills
> >> >> >>
> >> >> >>  Could you please remove the entries for
> >> >> >>
> >> >> >>  ntp.mel.nml.csiro.au
> >> >> >>  ntp.per.nml.csiro.au
> >> >> >>  ntp.nml.csiro.au
> >> >> >>
> >> >> >>  from the list of public stratum 1 NTP servers as soon as is
> >> >> >>  convenient ?
> >> >> >>
> >> >> >>  Yours sincerely, Michael Wouters
> >> >> >>
> >> >> >>  -------------------------------------------
> >> >> >>  Dr Michael Wouters
> >> >> >>  Time and Frequency Section
> >> >> >>  National Measurement Laboratory
> >> >> >>  CSIRO Division of Telecommunications and Industrial Physics
> >> >> >>
> >> >> >>  PO Box 218, Lindfield NSW 2070
> >> >> >>  Sydney Australia
> >> >> >>
> >> >> >>  (street address: Bradfield Rd, Lindfield NSW 2070)
> >> >> >>
> >> >> >>  Ph 61 2 9413 7268
> >> >> >>  Fax 61 2 9413 7202
> >> >> >>  --------------------------------------------
> >> >> >



More information about the hackers mailing list