[ntp:hackers] U Wisconsin

Dean Gibson (NTP Time Administrator) ntp-admin at ultimeth.net
Thu Jun 26 13:10:23 PDT 2003


I must say, not only am I not surprised, but I'm surprised that you're surprised.

The abuse that has been heaped on my little NTP server from misconfigured SNTP clients is legion.  Just two weeks ago I found a single IP address (in Greece) that in 36 hours had sent over 100,000 DNS lookups for "time.ultimeth.net", an NTP hostname that we abandoned almost three years ago, and mapped to 127.x.x.x to satisfy thirsty DNS requests.  We had to find a different secondary DNS provider due to complaints from one provider about DNS storms for "time.ultimeth.net", and I had to switch to an unlimited rate ISP rather than the cheaper metered-rate ISP I had been using.

My limited research shows that our hostname "time.ultimeth.net" is installed in the software as one of the choices.  The author of this package, not being content to just implement an SNTP that allows polling intervals as little as one second or less, apparently also decided to directly do his own DNS queries for each SNTP poll (complete w/ NO caching), rather than using the local resolver.  May SNTP rot in hell.

That's just one incident, and many of you on this mailing list have heard me play this record before.

So, what do we do about it?

In the case of U Wisconsin, it's very clear:  NorTel Networks gets to pay U Wisconsin's Internet access fees in perpetuity.

For the rest of the NTP world who will increasingly suffer under this nonsense, I'll repeat my suggestion made previously here on January 21st of this year (copied below):  NTP needs to be renamed (eg, "Precision Time Protocol") and the port number changed from 123, plus other changes intended to make it much more difficult for unauthorized, poorly-written clients to participate.  Dave suggested at the time making public-key encryption mandatory;  I'm coming around to that

-- Dean

David L. Mills wrote on 2003-06-26 10:55:
>Guys,
>
>I find myself on the review team for an incident taking place at U Wisconsin/Madison.  Apparently, the Netgear folks have manufactured some 700,000 routers with embedded SNTP clients configured to use the public U Wisconsin NTP server.  The server address is unchangeable and the client cannot be disabled.  If that isn't bad enough, if the client gets no replies, it starts sending packets at one-second intervals until forever and without backoff.
>
>The U Wisconsin folks determined some 285,000 different IP addresses are now sending between 300 and 700 packets per second requiring between 150 and 400 megabits per second.  Apparently, the prrincipal eason for this flux is misconfiguration of the firewall component of the router.  This is costing them $266 per megabit per day.
>
>The Netgear folks were slow to respond until U Wisconsin folks emailed the entire senior management and others known to be U Wisconsin alum.  Netgear says they have no way to recall those routers and no way to insure the products are updated from the web site.  The products cost between $20 and $40 depending on rebate.
>
>U Wisconsin have considered several ways to deflect the tide, the most promising may be noting the source port 23457 unique to these products and tossing them at the doorstep.  The products do not use DNS and are not configurable.  Another way considered is to configure a subnet visible to BGP and convince the ISPs to punch holes in the routing fabric.  Send money.
>
>I never thought it could get as bad as that.  My reasoned recommendation was to fire up the lawyers and sue the bastards for costs and punitive damages and to injoin the company from selling any products until proved safe.  There is apparently some standards group that allegedly reviews and certifies new products for Internet use.  The Netgear products were all certified, which surely says nothing about the standards group.
>
>Include me in any replies; I am not on any ntp.org list.
>
>Dave
>
>>To: hackers at ntp.org
>>From: "Dean Gibson (NTP Time Administrator)" <ntp-admin at ultimeth.net>
>>Subject: Abuse ...
>>Date: Tue, 21 Jan 2003 11:14:12 -0800
>>
>>This is the problem that I started warning about over a year ago.  Our solution was to take our NTP servers "prior permission only" 15 months ago and change hostnames and IP addresses;  nevertheless, we still have unauthorized hosts that attempt to access the old IP address, and their attempts top the list when "ntpdc -c monlist" is sorted by usage.
>>
>>The problem is complex:
>>
>>1. A number of web sites have copied the ntp.org lists of stratum 1 & 2 servers to their own web site, making it virtually impossible to get off all the lists around the world.
>>
>>2. Further, at least one software author, Graham Mainwaring <graham at mhn.org>, distributes (via SourceForge.net) a Windows NTP client named "NetTime", with a list of NTP servers built in to the configuration file;  the list of servers is from June, 2000!  Despite my request to Mr. Mainwaring, and his agreement to do so one year ago, he has not yet removed my server entry from his configuration file, nor does he apparently see anything wrong with capturing the list of servers and putting them in his configuration file.  Requests to SourceForge.net to remove "NetTime" from distribution have gone unheeded.  SourceForge.net is particularly incompetent in this matter.
>>
>>3. At least some of the software out there allows the user to specify very small polling intervals;  we have seen polling rates from a single client as high as 5 per second.
>>
>>4. Users are confused because the "Daytime" (port 13), "Time" (port 37), and "NTP" (port 123) protocol all describe themselves as a "Network Time Protocol", leading naive users to believe that NTP servers will support their client software.  We had a number of "Daytime" and "Time" protocol clients sending massive amounts of packets.  We have had to ask potential users for the name/version of their NTP client software;  and we still have to reject requests that specify "rdate"!
>>
>>5. Tracking down unauthorized users is difficult;  ISPs presently don't regard unauthorized NTP access as abuse.
>>
>>If my experience is any guide, I think that CSIRO.AU's request to remove their servers from the public listings, will (sadly) have little effect on their traffic.  Changing/removing DNS names will have little effect, as many NTP clients run for months without being rebooted, and many other NTP clients (for access control reasons) specify the IP address of an NTP server, despite the listing's request to use DNS.  Put bluntly, CSIRO.AU is hosed in the short run;  the best long term solution is to remove the DNS entries and move the servers to new IP addresses, and even that will leave behind a huge group of hosts around the world blindly sending packets to the old IP addresses for years to come.
>>
>>The best real solution that I can think of, is to fundamentally change the NTP protocol:
>>
>>1. Change the name of the protocol so that it doesn't have the "generic" name of "Network Time Protocol".
>>
>>2. Go to a new port number (is 24 available?).  This will allow network switches to reject old NTP packets wholesale;  this should kill off most of the unmaintained and poorly written 3rd-party NTP clients.
>>
>>3. Design the new NTP so that the "restrict" command can be used to specify thousands of users, or some equivalent mechanism.  Cause the "server" command to be able to create an associated "restrict" entry, so that DNS lookups for the "server" command result in the same IP address for both the "server" and associated "restrict" data entries in the server's memory.
>>
>>4. Inform anyone who requests a public NTP server listing, that it is a fool's errand to not get contact information from EVERY client before allowing access.
>>
>>Because, if we don't do something soon, the situation will be even more out of control than it is now.
>>
>>Sincerely, Dean Gibson
>>
>>David L. Mills wrote on 2003-01-20 20:46:
>>>Guys,
>>>
>>>I was afraid it would come to this. The most disturbing issue is that the abuse gets worse when the perp is call-gapped. I've heard this from other sites.
>>>
>>>Dave
>>>
>>>>-------- Original Message --------
>>>>Subject: RE: Listing of CSIRO NML stratum 1 NTP servers
>>>>Date: Tue, 21 Jan 2003 09:50:39 +1100
>>>>From: Peter.Fisk at csiro.au
>>>>To: mills at udel.edu, Michael.Wouters at csiro.au
>>>>CC: Peter.Fisk at csiro.au
>>>>
>>>>Dear Dr Mills,
>>>>Thank you for the response. 
>>>>
>>>>For reasons which I expand on below, I would appreciate it if you could remove the entries from your listing as soon as possible.
>>>>
>>>>We are not leaving the timekeeping business or the NTP business; our problem is that despite the service area being shown as limited (by request) to AARNET on your listings, we have more than 200000 clients accessing these servers each day from outside Australia. The network charges for this traffic are becoming prohibitive, the traffic is doubling over a timescale of less than eight months and we must now take some action.
>>>>
>>>>A further issue we face is that denying service to clients outside our service area either by failing to reply or sending the "kiss of death" packet results in a massive (up to 100-fold in some cases) increase in incoming traffic, further increasing our costs. In fact, when we performed an experiment last year to deny responses to NTP packets from Japan, this produced such a massive increase in traffic that AARNET thought it was a DOS attack on the Australian network.
>>>>
>>>>It seems that some authors of ntp clients do not write their software to deal with the lack of a response, or a "kiss of death", packet in a sensible way. The potential financial liability for this traffic, especially in the event that we have to temporarily or permanently take one of our servers off-line, is completely unacceptable.
>>>>
>>>>We are therefore looking at ways to limit the traffic on our servers from international sources, and the first and most obvious method is to remove the addresses from internationally recognised listings such as yours. The extent of our further action will be dependent on the effect of this removal.
>>>>
>>>>I think this problem will sooner or later face all providers of open access NTP services, and that organisations such as ours will be unable to provide such a service unless coordinated action is taken by the network community to enforce reasonable behaviour.
>>>>
>>>>Regards, Peter Fisk
>>>>
>>>>---------------------------------------------------------------------------
>>>>Dr Peter Fisk
>>>>Section Manager: Standards for RF, Microwave, Time and Frequency
>>>>National Measurement Laboratory
>>>>CSIRO Division of Telecommunications and Industrial Physics
>>>>PO Box 218, Lindfield NSW 2070
>>>>Sydney, Australia
>>>>(street address: Bradfield Rd, Lindfield NSW 2070)
>>>>Phone: (61 2) 9413 7221
>>>>Fax (61 2) 9413 7631
>>>>Mobile (61) 0418 693 962
>>>>---------------------------------------------------------------------------
>>>> 
>>>>>-----Original Message-----
>>>>>From: David L. Mills [mailto:mills at udel.edu]
>>>>>Sent: Tuesday, 21 January 2003 00:58
>>>>>To: Wouters, Michael (CTIP, Lindfield)
>>>>>Cc: Fisk, Peter (CTIP, Lindfield)
>>>>>Subject: Re: Listing of CSIRO NML stratum 1 NTP servers
>>>>>
>>>>>Michael,
>>>>>
>>>>>Thanks. In the next update. I hope this doesn't mean you guys are out of the timekeeping business.
>>>>>
>>>>>Dave
>>>>>
>>>>>>"Wouters, Michael (CTIP, Lindfield)" wrote:
>>>>>>
>>>>>>Dear Dr Mills
>>>>>>
>>>>>>Could you please remove the entries for
>>>>>>
>>>>>>ntp.mel.nml.csiro.au
>>>>>>ntp.per.nml.csiro.au
>>>>>>ntp.nml.csiro.au
>>>>>>
>>>>>>from the list of public stratum 1 NTP servers as soon as is convenient ?
>>>>>>
>>>>>>Yours sincerely, Michael Wouters
>>>>>>
>>>>>>-------------------------------------------
>>>>>>Dr Michael Wouters
>>>>>>Time and Frequency Section
>>>>>>National Measurement Laboratory
>>>>>>CSIRO Division of Telecommunications and Industrial Physics
>>>>>>
>>>>>>PO Box 218, Lindfield NSW 2070
>>>>>>Sydney Australia
>>>>>>
>>>>>>(street address: Bradfield Rd, Lindfield NSW 2070)
>>>>>>
>>>>>>Ph 61 2 9413 7268
>>>>>>Fax 61 2 9413 7202
>>>>>>--------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://support.ntp.org/pipermail/hackers/attachments/20030626/c9485e95/attachment.html


More information about the hackers mailing list