[ntp:security] memory leak

geir.guldstein at no.abb.com geir.guldstein at no.abb.com
Fri Jan 11 15:54:21 UTC 2008


Hi and happy new year to all.

I have now investigated this problem using Rational Purify. I have not used
this tool previously, but found it to be very useful in this case (thanks
Heiko).

There is a memory leak bug in ntpd that is triggered when it receives
random garbage data, e.g. those produced by ISIC.

Bug explanation, the source code version used is 4.2.4p4:

When "attacked" by ISIC as described in our previous mail ntpd frequently
executes the FormatMessage call at line 73 in i ntp-4.2.4
p4\ports\winnt\libisc\isc_strerror.c. The supplied error number is 234
("More data is available"). Call stack:

>     ntpd.exe!FormatError(int error=234)  Line 85    C
      ntpd.exe!isc__NTstrerror(int err=234, int * bfreebuf=0x00cef724)
Line 114 + 0x9 bytes    C
      ntpd.exe!NTstrerror(int errnum=234)  Line 40 + 0xd bytes    C
      ntpd.exe!format_errmsg(char * nfmt=0x00cef92c, int lennfmt=256, const
char * fmt=0x0053601c, int errval=234)  Line 103 + 0x9 bytes      C
      ntpd.exe!msyslog(int level=3, const char * fmt=0x0053601c, ...)  Line
162 + 0x1c bytes  C
      ntpd.exe!iocompletionthread(void * NotUsed=0x00000000)  Line 192 +
0x10 bytes  C
      ntpd.exe!_callthreadstart()  Line 293 + 0xf bytes     C
      ntpd.exe!_threadstart(void * ptd=0x00ad3578)  Line 277      C
      kernel32.dll!7c80b683()

The leak happens because ntpd supplies the FORMAT_MESSAGE_ALLOCATE_BUFFER
flag to the FormatMessage OS API. The application is then responsible for
deallocating (LocalFree) the returned buffer when it is no longer needed
(http://msdn2.microsoft.com/en-us/library/ms679351.aspx). ntpd fails to do
this, there is a "freebuf" flag that is signaled back to the calling
routines but the flag is thrown away unused in the NTstrerror function
(line 38 in ntp-4.2.4p4\ports\winnt\libisc\isc_strerror.c).

Best regards
Geir





                                                                           
             Heiko Gerstung                                                
             <heiko.gerstung at m                                             
             einberg.de>                                                To 
                                       mayer at ntp.isc.org                   
             21.12.2007 08:48                                           cc 
                                       Kevin McGrath/NOABB/ABB at ABB,        
                                       security at ntp.org, Kai               
                                       Hansen/NOCRC/ABB at ABB, Geir          
                                       Guldstein/NOINA/ABB at ABB, Martin     
                                       Burnicki                            
                                       <martin.burnicki at meinberg.de>,      
                                       Frank Kardel <kardel at ntp.org>       
                                                                   Subject 
                                       Re: [ntp:security] memory leak      
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




Hi Guys,

I installed a trial version of Rational Purify 7 from IBM on my laptop and
ran
NTPD in debug mode on it. As Danny correctly described, it runs a lot
slower in
this environment and all I got out of it was ~600 req/s (compared to
>40,000
req/s running the release version without Purify).

Purify revealed that a library called BMNET.DLL shows a memory leak. After
some
investigation I found out that this DLL belongs to the driver suite of my
mobile
UMTS data card and is produced by Byte Mobile ("BM"). It is hooked into the
IP
stack of Windows ("LSP") and redirects Windows system API calls (one of
them
being "BMGetPeer").

I did not want to uninstall my whole data card stuff therefore I renamed
that
library (which causes all TCP/IP connections to fail, NTPD will not start
because it cannot bind to the wildcard interface and Firefox simply does
not
return anything when you try to access any web page). I then ran LSPfix
(http://www.cexx.org/lspfix.htm), a small tool that finds problems in the
winsock protocol stack. It immediately detected the missing DLL and removed
it
from the stack (fixing the registry). That restored my IP stack and I was
able
to access the net and start ntpd.

When I now fire up NTPLOAD I get around 30,000 req/s and the memory leak is
not
there anymore.

Kevin, could you please check if you have such a library in your SYSTEM32\
path?
If not, I would suggest to download and install the trial version of
Rational
Purify 7 from IBM (http://www.ibm.com/developerworks/downloads/r/rpp/) and
run a
debug version of ntpd within it (if you need a binary version of the 4.2.4
p4
with debugging enabled, please let me know).

So, after all it seems that this memory leak you see is caused by something
else
in the Windows IP stack.

I am available via email throughout the holidays in case of emergency.

Wishing all of you at ABB Norway (especially Dr. Hansen who I met last year
in
Oslo )and (of course!) Danny

a Merry Christmas and a Happy New Year,

    Heiko



Danny Mayer schrieb:
> Heiko Gerstung wrote:
>> Danny Mayer schrieb:
>>> Heiko Gerstung wrote:
>>>> Danny et al,
>>>>
>>>> I can confirm a memory leak with 4.2.4p4, penetrating it with ntpload
at
>>>> 33000 req/s results in ~0.5M/s increased memory consumption rate which
>>>> is not freed as it seems. I did not crash it, but this would be the
>>>> result if I just let ntpload do its ugly job :-)
>>> Heiko,
>>>
>>> Can you test this with the build, as is, from the tarball. I know you
>>> still have made some minor changes to your build and I want to make
sure
>>> that those changes are not affecting this.
>> No, we were using a vanilla ntpd in the 4.2.4p4 version (AKA "Modena")
>> of the installer.
>>
>>> Just remember that the
>>> recvbuf list expands to accommodate the incoming influx of packets and
>>> does not release them. There used to be a limit and I had removed it
but
>>> that's true of the Unix version as well. The version I have has barely
>>> changed its footprint since I started to run ntpload (from another
>>> system) against it. The current syntax I'm using is:
>>> ntpload-2.2\Release>ntpload -c -t 10 -u 200 10.60.98.32
>>>
>>> and I'm running debug mode which does slow things down somewhat.
>> Today I checked again and found out that the memory leak seems to be
>> appearing on my laptop (which I used for my tests so far) but not on my
>> desktop machine, which I use for testing and building the installer and
>> the included ntpd and openssl. That seems to indicate that this memory
>> leak is somehow related to different hardware or software platforms.
>>
>> Both my machines run XP Professional SP2 and patches are up to date
>> (last patch installed is KB944653). The laptop has IE7 installed and the
>> desktop still runs IE6, while I am typing this I am installing IE7 on
>> the desktop machine to find out if this has something to do with it.
>>
>
> Nothing about IE should make a difference to ntp. I'm also running an
> Oracle DB, tomcat, IIS, Firefox and IE, antivirus, VS 2005, pidgin,
> Acrobat reader, named, Microsoft Office products all at the same time on
> my system.
>
>> There is nothing special with the network interfaces of both systems, I
>> have GigE connections on both of them (Intel chip on the desktop,
>> Marvell Yukon on the lappy) and they both are connected to the same
>> switch and subnet plus they use the same DHCP, DNS and other servers.
>>
>
> None of which should make a difference.
>
>> I will keep you posted. We are trying to analyze the memory leak with
>> some special debugger (Rational Purify) in order to hunt it down.
>>
>
> Purify works best if ntpd is built with debug. That also slows it down
> of course.
>
>> Of course we are open for ideas and if anyone reading this has the
>> chance to test 4.2.4p4-modena on a Windows machine, please do so and let
>> us know the results as well as details regarding hard- and software
>> configuration of that system.
>>
>
> The security list is a very small closed list that only a few people see.
>
> One thing you might try is adding -U 0 to the command line in case the
> dynamic interface code is causing a problem. We have one bug report on
> HP/UX which indicated a problem that we haven't been able to track down
> yet. Turning off dynamic reconfiguration fixed the problem but we still
> don't know why. See Bug #885 starting with Comment #3.
>
> Danny


--
------------------------------------------------------------------------

*MEINBERG Funkuhren GmbH & Co. KG*
Lange Wand 9
D-31812 Bad Pyrmont, Germany
Tel.: ++49 (0)5281 9309-25
Fax: ++49 (0)5281 9309-30
eMail: heiko.gerstung at meinberg.de <mailto:heiko.gerstung at meinberg.de>
Internet: www.meinberg.de <http://www.meinberg.de/>

------------------------------------------------------------------------

Meinberg radio clocks: 25 years of accurate time worldwide





More information about the security mailing list