[ntp:hackers] U Wisconsin

Brian Utterback Brian.Utterback at Sun.COM
Thu Jun 26 16:09:54 PDT 2003


I relayed the story to a friend of mine and he has this suggestion:

Netgear pays for an IETF fast-track RFC designating that single
IP as a non-routable reserved IP. All ISP would have to be encouraged
to adopt it.

UW-Madison places a brass and granite marker in the ground that says:

"Here lies xxx.xxx.xxx.xxx, an innocent IP address killed in its
prime by corporate short-sightedness and selfishness. RIP"

Perhaps RIP should be spelled out, there could be some confusion in
this context.

Dean Gibson (NTP Time Administrator) wrote:

> I must say, not only am I not surprised, but I'm surprised that you're 
> surprised.
> 
> The abuse that has been heaped on my little NTP server from 
> misconfigured SNTP clients is legion.  Just two weeks ago I found a 
> single IP address (in Greece) that in *36 hours* had sent over *100,000 
> DNS* lookups for "time.ultimeth.net", an NTP hostname that we abandoned 
> almost three years ago, and mapped to 127.x.x.x to satisfy thirsty DNS 
> requests.  We had to find a different secondary DNS provider due to 
> complaints from one provider about DNS storms for "time.ultimeth.net", 
> and I had to switch to an unlimited rate ISP rather than the cheaper 
> metered-rate ISP I had been using.
> 
> My limited research shows that our hostname "time.ultimeth.net" is 
> installed *in the software* as one of the choices.  The author of this 
> package, not being content to just implement an SNTP that allows polling 
> intervals as little as one second or less, apparently also decided to 
> directly do his own DNS queries for each SNTP poll (complete w/ *NO 
> *caching), rather than using the local resolver.  May SNTP rot in hell.
> 
> That's just one incident, and many of you on this mailing list have 
> heard me play this record before.
> 
> So, what do we do about it?
> 
> In the case of U Wisconsin, it's very clear:  NorTel Networks gets to 
> pay U Wisconsin's Internet access fees in perpetuity.
> 
> For the rest of the NTP world who will increasingly suffer under this 
> nonsense, I'll repeat my suggestion made previously here on January 21st 
> of this year (copied below):  NTP needs to be renamed (eg, "Precision 
> Time Protocol") and the port number changed from 123, plus other changes 
> intended to make it much more difficult for unauthorized, poorly-written 
> clients to participate.  Dave suggested at the time making public-key 
> encryption mandatory;  I'm coming around to that
> 
> -- Dean
> 
> David L. Mills wrote on 2003-06-26 10:55:
> 
>> Guys,
>>
>> I find myself on the review team for an incident taking place at U 
>> Wisconsin/Madison.  Apparently, the Netgear folks have manufactured 
>> some 700,000 routers with embedded SNTP clients configured to use the 
>> public U Wisconsin NTP server.  The server address is unchangeable and 
>> the client cannot be disabled.  If that isn't bad enough, if the 
>> client gets no replies, it starts sending packets at one-second 
>> intervals until forever and without backoff.
>>
>> The U Wisconsin folks determined some 285,000 different IP addresses 
>> are now sending between 300 and 700 packets per second requiring 
>> between 150 and 400 megabits per second.  Apparently, the prrincipal 
>> eason for this flux is misconfiguration of the firewall component of 
>> the router.  This is costing them $266 per megabit per day.
>>
>> The Netgear folks were slow to respond until U Wisconsin folks emailed 
>> the entire senior management and others known to be U Wisconsin alum.  
>> Netgear says they have no way to recall those routers and no way to 
>> insure the products are updated from the web site.  The products cost 
>> between $20 and $40 depending on rebate.
>>
>> U Wisconsin have considered several ways to deflect the tide, the most 
>> promising may be noting the source port 23457 unique to these products 
>> and tossing them at the doorstep.  The products do not use DNS and are 
>> not configurable.  Another way considered is to configure a subnet 
>> visible to BGP and convince the ISPs to punch holes in the routing 
>> fabric.  Send money.
>>
>> I never thought it could get as bad as that.  My reasoned 
>> recommendation was to fire up the lawyers and sue the bastards for 
>> costs and punitive damages and to injoin the company from selling any 
>> products until proved safe.  There is apparently some standards group 
>> that allegedly reviews and certifies new products for Internet use.  
>> The Netgear products were all certified, which surely says nothing 
>> about the standards group.
>>
>> Include me in any replies; I am not on any ntp.org list.
>>
>> Dave
>>
>>> To: hackers at ntp.org
>>> From: "Dean Gibson (NTP Time Administrator)" <ntp-admin at ultimeth.net>
>>> Subject: Abuse ...
>>> Date: Tue, 21 Jan 2003 11:14:12 -0800
>>>
>>> This is the problem that I started warning about over a year ago.  
>>> Our solution was to take our NTP servers "prior permission only" 15 
>>> months ago and change hostnames and IP addresses;  nevertheless, we 
>>> still have unauthorized hosts that attempt to access the old IP 
>>> address, and their attempts top the list when "ntpdc -c monlist" is 
>>> sorted by usage.
>>>
>>> The problem is complex:
>>>
>>> 1. A number of web sites have copied the ntp.org lists of stratum 1 & 
>>> 2 servers to their own web site, making it virtually impossible to 
>>> get off all the lists around the world.
>>>
>>> 2. Further, at least one software author, Graham Mainwaring 
>>> <graham at mhn.org>, distributes (via SourceForge.net) a Windows NTP 
>>> client named "NetTime", with a list of NTP servers built in to the 
>>> configuration file;  the list of servers is from June, 2000!  Despite 
>>> my request to Mr. Mainwaring, and his agreement to do so one year 
>>> ago, he has not yet removed my server entry from his configuration 
>>> file, nor does he apparently see anything wrong with capturing the 
>>> list of servers and putting them in his configuration file.  Requests 
>>> to SourceForge.net to remove "NetTime" from distribution have gone 
>>> unheeded.  SourceForge.net is particularly incompetent in this matter.
>>>
>>> 3. At least some of the software out there allows the user to specify 
>>> very small polling intervals;  we have seen polling rates from a 
>>> single client as high as 5 per second.
>>>
>>> 4. Users are confused because the "Daytime" (port 13), "Time" (port 
>>> 37), and "NTP" (port 123) protocol all describe themselves as a 
>>> "Network Time Protocol", leading naive users to believe that NTP 
>>> servers will support their client software.  We had a number of 
>>> "Daytime" and "Time" protocol clients sending massive amounts of 
>>> packets.  We have had to ask potential users for the name/version of 
>>> their NTP client software;  and we still have to reject requests that 
>>> specify "rdate"!
>>>
>>> 5. Tracking down unauthorized users is difficult;  ISPs presently 
>>> don't regard unauthorized NTP access as abuse.
>>>
>>> If my experience is any guide, I think that CSIRO.AU's request to 
>>> remove their servers from the public listings, will (sadly) have 
>>> little effect on their traffic.  Changing/removing DNS names will 
>>> have little effect, as many NTP clients run for months without being 
>>> rebooted, and many other NTP clients (for access control reasons) 
>>> specify the IP address of an NTP server, despite the listing's 
>>> request to use DNS.  Put bluntly, CSIRO.AU is hosed in the short 
>>> run;  the best long term solution is to remove the DNS entries and 
>>> move the servers to new IP addresses, and even that will leave behind 
>>> a huge group of hosts around the world blindly sending packets to the 
>>> old IP addresses for years to come.
>>>
>>> The best real solution that I can think of, is to fundamentally 
>>> change the NTP protocol:
>>>
>>> 1. Change the name of the protocol so that it doesn't have the 
>>> "generic" name of "Network Time Protocol".
>>>
>>> 2. Go to a new port number (is 24 available?).  This will allow 
>>> network switches to reject old NTP packets wholesale;  this should 
>>> kill off most of the unmaintained and poorly written 3rd-party NTP 
>>> clients.
>>>
>>> 3. Design the new NTP so that the "restrict" command can be used to 
>>> specify thousands of users, or some equivalent mechanism.  Cause the 
>>> "server" command to be able to create an associated "restrict" entry, 
>>> so that DNS lookups for the "server" command result in the same IP 
>>> address for both the "server" and associated "restrict" data entries 
>>> in the server's memory.
>>>
>>> 4. Inform anyone who requests a public NTP server listing, that it is 
>>> a fool's errand to not get contact information from EVERY client 
>>> before allowing access.
>>>
>>> Because, if we don't do something soon, the situation will be even 
>>> more out of control than it is now.
>>>
>>> Sincerely, Dean Gibson
>>>
>>> David L. Mills wrote on 2003-01-20 20:46:
>>>
>>>> Guys,
>>>>
>>>> I was afraid it would come to this. The most disturbing issue is 
>>>> that the abuse gets worse when the perp is call-gapped. I've heard 
>>>> this from other sites.
>>>>
>>>> Dave
>>>>
>>>>> -------- Original Message --------
>>>>> Subject: RE: Listing of CSIRO NML stratum 1 NTP servers
>>>>> Date: Tue, 21 Jan 2003 09:50:39 +1100
>>>>> From: Peter.Fisk at csiro.au
>>>>> To: mills at udel.edu, Michael.Wouters at csiro.au
>>>>> CC: Peter.Fisk at csiro.au
>>>>>
>>>>> Dear Dr Mills,
>>>>> Thank you for the response.
>>>>>
>>>>> For reasons which I expand on below, I would appreciate it if you 
>>>>> could remove the entries from your listing as soon as possible.
>>>>>
>>>>> We are not leaving the timekeeping business or the NTP business; 
>>>>> our problem is that despite the service area being shown as limited 
>>>>> (by request) to AARNET on your listings, we have more than 200000 
>>>>> clients accessing these servers each day from outside Australia. 
>>>>> The network charges for this traffic are becoming prohibitive, the 
>>>>> traffic is doubling over a timescale of less than eight months and 
>>>>> we must now take some action.
>>>>>
>>>>> A further issue we face is that denying service to clients outside 
>>>>> our service area either by failing to reply or sending the "kiss of 
>>>>> death" packet results in a massive (up to 100-fold in some cases) 
>>>>> increase in incoming traffic, further increasing our costs. In 
>>>>> fact, when we performed an experiment last year to deny responses 
>>>>> to NTP packets from Japan, this produced such a massive increase in 
>>>>> traffic that AARNET thought it was a DOS attack on the Australian 
>>>>> network.
>>>>>
>>>>> It seems that some authors of ntp clients do not write their 
>>>>> software to deal with the lack of a response, or a "kiss of death", 
>>>>> packet in a sensible way. The potential financial liability for 
>>>>> this traffic, especially in the event that we have to temporarily 
>>>>> or permanently take one of our servers off-line, is completely 
>>>>> unacceptable.
>>>>>
>>>>> We are therefore looking at ways to limit the traffic on our 
>>>>> servers from international sources, and the first and most obvious 
>>>>> method is to remove the addresses from internationally recognised 
>>>>> listings such as yours. The extent of our further action will be 
>>>>> dependent on the effect of this removal.
>>>>>
>>>>> I think this problem will sooner or later face all providers of 
>>>>> open access NTP services, and that organisations such as ours will 
>>>>> be unable to provide such a service unless coordinated action is 
>>>>> taken by the network community to enforce reasonable behaviour.
>>>>>
>>>>> Regards, Peter Fisk
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Dr Peter Fisk
>>>>> Section Manager: Standards for RF, Microwave, Time and Frequency
>>>>> National Measurement Laboratory
>>>>> CSIRO Division of Telecommunications and Industrial Physics
>>>>> PO Box 218, Lindfield NSW 2070
>>>>> Sydney, Australia
>>>>> (street address: Bradfield Rd, Lindfield NSW 2070)
>>>>> Phone: (61 2) 9413 7221
>>>>> Fax (61 2) 9413 7631
>>>>> Mobile (61) 0418 693 962
>>>>> ---------------------------------------------------------------------------
>>>>>  
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: David L. Mills [mailto:mills at udel.edu]
>>>>>> Sent: Tuesday, 21 January 2003 00:58
>>>>>> To: Wouters, Michael (CTIP, Lindfield)
>>>>>> Cc: Fisk, Peter (CTIP, Lindfield)
>>>>>> Subject: Re: Listing of CSIRO NML stratum 1 NTP servers
>>>>>>
>>>>>> Michael,
>>>>>>
>>>>>> Thanks. In the next update. I hope this doesn't mean you guys are 
>>>>>> out of the timekeeping business.
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>>> "Wouters, Michael (CTIP, Lindfield)" wrote:
>>>>>>>
>>>>>>> Dear Dr Mills
>>>>>>>
>>>>>>> Could you please remove the entries for
>>>>>>>
>>>>>>> ntp.mel.nml.csiro.au
>>>>>>> ntp.per.nml.csiro.au
>>>>>>> ntp.nml.csiro.au
>>>>>>>
>>>>>>> from the list of public stratum 1 NTP servers as soon as is 
>>>>>>> convenient ?
>>>>>>>
>>>>>>> Yours sincerely, Michael Wouters
>>>>>>>
>>>>>>> -------------------------------------------
>>>>>>> Dr Michael Wouters
>>>>>>> Time and Frequency Section
>>>>>>> National Measurement Laboratory
>>>>>>> CSIRO Division of Telecommunications and Industrial Physics
>>>>>>>
>>>>>>> PO Box 218, Lindfield NSW 2070
>>>>>>> Sydney Australia
>>>>>>>
>>>>>>> (street address: Bradfield Rd, Lindfield NSW 2070)
>>>>>>>
>>>>>>> Ph 61 2 9413 7268
>>>>>>> Fax 61 2 9413 7202
>>>>>>> --------------------------------------------
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hackers mailing list
> hackers at ntp.org
> http://mailman.ntp.org/mailman/listinfo/hackers




More information about the hackers mailing list