[ntp:hackers] Use MSG_CONFIRM flag to keep OS ARP table up to date?

Heiko Gerstung (mobil) heiko.gerstung at meinberg.de
Wed May 2 21:12:31 UTC 2012


Yes, gc_thres3 is 1024 per default, so you will see ARP resolving when you deal with more than 1024 clients. But without MSG_CONFIRM there will be unnecessary ARP resolving with every ntp client - no matter what gc_thres3 is set to. All that 'helps' is trying to keep the kernel from starting the garbage collection by prolonging the minimum interval and the gc_stale time parameter.  

Regards,
 Heiko

Am 02.05.2012 um 22:30 schrieb Greg Dowd <gdowd at symmetricom.com>:

> I definitely agree it would be useful to have at the application layer.  I think one of the challenges there is that the default arp table depth is quite small although I can't remember it now.   Maybe gc_thresh3 is like 1024.  So you will still get thrashing as you boot old entries off the queue, yes?
> 
> 
> Greg Dowd
> gdowd at symmetricom dot com (antispam format)
> Symmetricom, Inc.
> www.symmetricom.com
> "A clever person solves a problem.  A wise person avoids it." Albert Einstein
> 
> 
> -----Original Message-----
> From: Heiko Gerstung (Home Office) [mailto:heiko.gerstung at meinberg.de] 
> Sent: Wednesday, May 02, 2012 12:16 PM
> To: Greg Dowd
> Cc: hackers at lists.ntp.org
> Subject: Re: [ntp:hackers] Use MSG_CONFIRM flag to keep OS ARP table up to date?
> 
> I agree that this is a workaround that can be applied on a dedicated 
> appliance (and we did this as well as a first measure to improve 
> performance and reduce unnecessary ARP requests), but in general I would 
> say that ntpd should follow the - IMHO - approach that is recommended by 
> the OS. On the arp(7) manpage the MSG_CONFIRM flag is mentioned when it 
> comes to UDP traffic.
> 
> Regards,
>   Heiko
> 
> Am 02.05.12 18:53, schrieb Greg Dowd:
>> We didn't modify the app but instead modified the size and stale timeout values for the arp table under linux to prevent the arp spam.
>> 
>> Greg Dowd
>> gdowd at symmetricom dot com (antispam format)
>> Symmetricom, Inc.
>> www.symmetricom.com
>> "A clever person solves a problem.  A wise person avoids it." Albert Einstein
>> 
>> 
>> -----Original Message-----
>> From: hackers-bounces+gdowd=symmetricom.com at lists.ntp.org [mailto:hackers-bounces+gdowd=symmetricom.com at lists.ntp.org] On Behalf Of Heiko Gerstung
>> Sent: Wednesday, May 02, 2012 3:10 AM
>> To: hackers at lists.ntp.org
>> Subject: [ntp:hackers] Use MSG_CONFIRM flag to keep OS ARP table up to date?
>> 
>> Hi!
>> 
>> We are currently running stress tests with more than 100k simulated NTP clients against one single NTP server and found out
>> that Linux is sending ARP queries for every single NTP client every 30s, putting significant load onto the NTP server
>> (running NTP 4.2.6p3) we use as the NTP reference for the simulated clients.
>> 
>> This seems to be caused by Linux not automatically updating the ARP cache when NTP sends its NTP responses to the clients,
>> resulting in all ARP table entries to become stale after a period of 30s (this can be changed but would require users to
>> mess with their kernel / network configuration). This triggers a new ARP request for each stale table entry
>> 
>> It is possible to use a flag called MSG_CONFIRM when calling send(), sendmsg() or sendto() that basically tells the kernel
>> that the ARP table entry for the given MAC / IP Address pair is still valid, avoiding that the corresponding ARP table entry
>> becomes stale.
>> 
>> My question: Did anyone use the MSG_CONFIRM flag with a UDP based socket communication in other projects? Is there any
>> obvious technical/security disadvantage using this flag within NTP? I patched my ntpd and applied this and it immediately
>> improved overall performance of the NTP server (and obviously reduced ARP related traffic dramatically).
>> 
>> I am not sure if Linux is the only affected OS as it is the only one I was able to test/look at.
>> 
>> Another big problem with this large-scale simulation: NTP only uses one of the CPU cores, which also dramatically limits the
>> performance on multi-processor machines. Do you know of anyone who looked at this and probably tried to add multi CPU
>> support to ntpd?
>> 
>> Thanks in advance for any help, comments or feedback!
>> 
>> Regards,
>>   Heiko
>> 
>> 
> 


More information about the hackers mailing list