[ntp:hackers] Time constant too large?

Brian Utterback Brian.Utterback at Sun.COM
Sun Jan 6 15:30:38 UTC 2008


Okay, new data. I ran the binary actually running currently on pogo with 
the same
behavior. So, it doesn't appear to be a build problem after all. I 
looked closer at the difference
between the microkernel code and the nanokernel code and I discovered 
something interesting.

In the microkernel reference code from udel.edu, the new frequency is 
set when MOD_FREQ is
given before the offset is set with MOD_OFFSET. Since the setting of the 
offset triggers a
recalculation of frequency, that means that the frequency is not 
actually set to the value given in the
call, but is modified.

Now, in the case of ntpd, it calculates the frequency and tried to load 
that into the kernel. Then
the kernel also calculates the frequency and adds the correction to the 
new amount, effectively
setting it to double what it should be.

In the nanokernel, on the other hand, the offset is set first, and the 
frequency recalculated. After
that, then the frequency is set. So, for the nanokernel, the frequency 
gets the actual value
given.

Since the magnitude of the error in the frequency is also a function of 
the time between setting
the frequency, all of the impulse tests pass because they usually 
involve setting the frequency
to 0 and then immediately setting it to some larger value. In that case, 
it behaves as it should.

So, what should ntpd do about this? While it seems reasonable to change 
the kernel, this is
something that can't easily be detected so it would be prudent to modify 
ntpd to handle all
cases. One possiblity would be to never set the frequency and offset in 
the same call.
If you need to do both, then make sure to do the offset first.

Brian Utterback wrote:
> What I did in every case was:
> 1. Stop ntpd
> 2. delete the drift file
> 3. Run "ntptime -f 0 -e 0 -o 0"
> 4. Start ntpd
> 5. observe.
>
> In the case of ntpd, it set the kernel frequency and offset both to 0. 
> Then later set the offset, and
> then even later set the frequency and offset. After that only the offset 
> is set.
>
> In the case of xntpd, it set the frequency and offset both to 0, then 
> after that only set the offset.
>
> Since in both cases the drift file is not in existence but both set the 
> frequency to 0 at start up,
> it would seem that your statement about the behavior when there is no 
> drift file is incorrect.
> Unless by specified, you mean not configured. In both tests the drift 
> file is configured but
> does not exist, as I would expect at initial start up of NTP on a new 
> system.
>
> You did not answer by questions though. Is it normal for NTP to set the 
> frequency a second
> time, and if so, is it normal for the kernel to modify this value with 
> the frequency calculated?
>
> Brian Utterback
> David L. Mills wrote:
>   
>> Brian,
>>
>> Forward progess. If the drift file is not specified and the kernel is in 
>> use, there is no record of prior adjustments, either by the daemon or 
>> the kernel. The question is whether to clobber any prior history of the 
>> frequency or to let it be. There are argurments both ways. Now, repeat 
>> your experiment with a specified frequency file, then make argument what 
>> to do otherwise. The original design presumes if there is no frequency 
>> file the kernel is disabled in order to use whatever prior setting was 
>> used to train its frequency.
>>
>> Dave
>>
>> Brian Utterback wrote:
>>
>>   
>>     
>>> I do appreciate your help. Thanks Dave.
>>>
>>> To answer the questions, the problem does no manifest with xntpd. I 
>>> have built many of
>>> the last 60 versions or so and they all seem to have the problem, 
>>> while I don't recall
>>> them having this problem at the time they were released, so I too 
>>> suspect a change in
>>> the build. As I said, there was no change (IIRC) in the kernel time 
>>> procedures between
>>> Solaris 10 and Nevada. By the way, Solaris is now open source and the 
>>> kernel is
>>> buildable by anyone. You once expressed interest in porting the 
>>> nanokernel to Solaris.
>>>
>>> Anyhow, I will try building on 10 and running on Nevada. But one thing 
>>> I noticed that
>>> looked odd in the dtrace data is that if there is no drift file, xntpd 
>>> set the kernel
>>> frequency once to zero at start up and then let the kernel handle the 
>>> frequency,
>>> but ntpd set it once to zero at startup, then later set the offset 
>>> once and then sometime
>>> later set both the offset and the frequency. After that it behaves the 
>>> same, only setting
>>> the offset.
>>>
>>> It appears to be this second call to set the frequency that is giving 
>>> the frequency a
>>> double kick. When the frequency is set by ntpd, the kernel also added 
>>> in the calculated
>>> frequency based on the offset, effectively doubling the magnitude of 
>>> the frequency
>>> desired.  So, the questions I need answered are whether it is normal 
>>> for ntpd to set
>>> the frequency twice in this manner, and if so, whether it is normal 
>>> for the kernel to
>>> use that frequency and add in the calculated frequency based on the 
>>> offset.
>>>
>>> Again, thanks for you help.
>>>
>>> Brian Utterback
>>>
>>> David L. Mills wrote:
>>>
>>>     
>>>       
>>>> Brian,
>>>>
>>>> I'm doing all I can to help and I don't have a budget either. The PPS 
>>>> is a red herring; as I said I confirmed Sol10 works fine with and 
>>>> without PPS. But, you haven't told me you confirmed your exatnt V4 
>>>> does or does not show the effects in a Sol 10. What I'm lookinf for 
>>>> here is some defect in the build procedure.
>>>>
>>>> I'm sorry, but my dim eyes can't make out the blizzard of numbers you 
>>>> show. Mucheasier to eyeball the loopstats. The dtrace is not the way 
>>>> to do it. Stop the daemon and use ntptime to set the frequency to 
>>>> maybe 200 PPM. Let the machine alone long enough for the ofset to 
>>>> climg to 50-100 ms. Set minpoll to 6, start the daemon and watch the 
>>>> loopstats. What you should see is an exponential decrease in offset 
>>>> with characteristic crossing zero in about 3000 s, overshooting a few 
>>>> percent and then slowly converging to zero in several hours.
>>>>
>>>> If it doesn't do that, check the kernel time constant with a 
>>>> debugger. With minpoll 6 and your old kernel modifications, it should 
>>>> be 2 as it should show with ntptime. A double check. Catch the sucker 
>>>> when at an extreme, kill the daemon, put noselect on the server line 
>>>> and restart the daemon. The offset should show the same behavior as 
>>>> with the previouis experiment, but this time without a forcing function.
>>>>
>>>> I forgot to ask if the behavior happens with xntpd. If it doesn't, 
>>>> there is something wrong with the ntpd build process. Again, try 
>>>> building on Sol 10 and running on Sol 11.
>>>>
>>>> Dave
>>>>
>>>> Brian Utterback wrote:
>>>>
>>>>  
>>>>
>>>>       
>>>>         
>>>>> Ahh, Nevada differs from other states in oh, so many ways 8-).
>>>>>
>>>>> However, that is as may be, but Nevada is the code name for the
>>>>> next version of Solaris, what experience tells us will be called
>>>>> Solaris 11, but our marketing department hates committing to these
>>>>> things in advance.
>>>>>
>>>>> As far as I know, there is no difference in the clock update loop
>>>>> between Solaris 10 and Nevada.
>>>>>
>>>>> I do not have a PPS installed because I:
>>>>> a) have no budget for NTP work
>>>>> b) have no access to outside walls or roof to use
>>>>>    what GPS equipment I can afford on my own.
>>>>>
>>>>> As Hal mentioned, is there any test case I can use to determine
>>>>> if the kernel is behaving correctly? I have been using dtrace to
>>>>> track all calls to ntp_adjtime and what the values being set were.
>>>>> Perhaps this data can offer a clue. Here is the first two hours of
>>>>> calls to ntp_adjtime: (the first line has the time stamp and the
>>>>> mode flags, the next has the lines as input to ntp_adjtime and
>>>>> the last has the values returned)
>>>>>
>>>>> 2008 Jan  3 08:00:40 mode=OFFSET,FREQ,
>>>>>         stat=0 off=0us freq=0 ppb tc=0 me=0us ee=0us
>>>>>         stat=40 off=0us freq=0 ppb tc=0 me=16us ee=16us
>>>>> 2008 Jan  3 08:03:56 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=13651us tc=2 me=34464us ee=4826us
>>>>>         stat=1 off=13651us freq=3371 ppb tc=2 me=34464us ee=4826us
>>>>> 2008 Jan  3 08:19:20 mode=OFFSET,FREQ,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-28616us freq=-42371 ppb tc=2 me=110416us ee=30436us
>>>>>         stat=1 off=-28616us freq=-67587 ppb tc=2 me=110416us ee=30436us
>>>>> 2008 Jan  3 08:20:24 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-19994us tc=2 me=98093us ee=28633us
>>>>>         stat=1 off=-19994us freq=-68808 ppb tc=2 me=98093us ee=28633us
>>>>> 2008 Jan  3 08:24:37 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-21799us tc=2 me=109303us ee=26791us
>>>>>         stat=1 off=-21799us freq=-74067 ppb tc=2 me=109303us ee=26791us
>>>>> 2008 Jan  3 08:26:54 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=1802us tc=2 me=81573us ee=26413us
>>>>>         stat=1 off=1802us freq=-73832 ppb tc=2 me=81573us ee=26413us
>>>>> 2008 Jan  3 08:28:58 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=12527us tc=2 me=102697us ee=24997us
>>>>>         stat=1 off=12527us freq=-72351 ppb tc=2 me=102697us ee=24997us
>>>>> 2008 Jan  3 08:32:36 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=23685us tc=3 me=101668us ee=23713us
>>>>>         stat=1 off=23685us freq=-71120 ppb tc=3 me=101668us ee=23713us
>>>>> 2008 Jan  3 08:37:46 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=25540us tc=3 me=101684us ee=22191us
>>>>>         stat=1 off=25540us freq=-69232 ppb tc=3 me=101684us ee=22191us
>>>>> 2008 Jan  3 08:41:02 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=24777us tc=3 me=109321us ee=20759us
>>>>>         stat=1 off=24777us freq=-68074 ppb tc=3 me=109321us ee=20759us
>>>>> 2008 Jan  3 08:43:08 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=27474us tc=3 me=103794us ee=19442us
>>>>>         stat=1 off=27474us freq=-67249 ppb tc=3 me=103794us ee=19442us
>>>>> 2008 Jan  3 08:44:21 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=25319us tc=3 me=105589us ee=18202us
>>>>>         stat=1 off=25319us freq=-66808 ppb tc=3 me=105589us ee=18202us
>>>>> 2008 Jan  3 08:44:41 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=26858us tc=4 me=98078us ee=17035us
>>>>>         stat=1 off=26858us freq=-66776 ppb tc=4 me=98078us ee=17035us
>>>>> 2008 Jan  3 08:45:41 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=25334us tc=4 me=118274us ee=15944us
>>>>>         stat=1 off=25334us freq=-66686 ppb tc=4 me=118274us ee=15944us
>>>>> 2008 Jan  3 08:50:09 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=31714us tc=4 me=99238us ee=15084us
>>>>>         stat=1 off=31714us freq=-66179 ppb tc=4 me=99238us ee=15084us
>>>>> 2008 Jan  3 08:55:04 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=31798us tc=4 me=125330us ee=14110us
>>>>>         stat=1 off=31798us freq=-65620 ppb tc=4 me=125330us ee=14110us
>>>>> 2008 Jan  3 08:55:19 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=34767us tc=5 me=104301us ee=13240us
>>>>>         stat=1 off=34767us freq=-65612 ppb tc=5 me=104301us ee=13240us
>>>>> 2008 Jan  3 08:59:44 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=37557us tc=5 me=125748us ee=12424us
>>>>>         stat=1 off=37557us freq=-65464 ppb tc=5 me=125748us ee=12424us
>>>>> 2008 Jan  3 09:00:48 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=37690us tc=5 me=115392us ee=11622us
>>>>>         stat=1 off=37690us freq=-65428 ppb tc=5 me=115392us ee=11622us
>>>>> 2008 Jan  3 09:01:53 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=37954us tc=5 me=115454us ee=10872us
>>>>>         stat=1 off=37954us freq=-65391 ppb tc=5 me=115454us ee=10872us
>>>>> 2008 Jan  3 09:04:45 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=34776us tc=6 me=145640us ee=10231us
>>>>>         stat=1 off=34776us freq=-65369 ppb tc=6 me=145640us ee=10231us
>>>>> 2008 Jan  3 09:05:24 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=43728us tc=6 me=121610us ee=10080us
>>>>>         stat=1 off=43728us freq=-65362 ppb tc=6 me=121610us ee=10080us
>>>>> 2008 Jan  3 09:07:25 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=43233us tc=6 me=129558us ee=9431us
>>>>>         stat=1 off=43233us freq=-65343 ppb tc=6 me=129558us ee=9431us
>>>>> 2008 Jan  3 09:12:15 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-42658us tc=6 me=218649us ee=31622us
>>>>>         stat=1 off=-42658us freq=-65389 ppb tc=6 me=218649us ee=31622us
>>>>> 2008 Jan  3 09:15:09 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-34545us tc=6 me=180866us ee=29718us
>>>>>         stat=1 off=-34545us freq=-65411 ppb tc=6 me=180866us ee=29718us
>>>>> 2008 Jan  3 09:16:16 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=33025us tc=6 me=189657us ee=36653us
>>>>>         stat=1 off=33025us freq=-65403 ppb tc=6 me=189657us ee=36653us
>>>>> 2008 Jan  3 09:17:20 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=32978us tc=6 me=140863us ee=34286us
>>>>>         stat=1 off=32978us freq=-65395 ppb tc=6 me=140863us ee=34286us
>>>>> 2008 Jan  3 09:20:34 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=32359us tc=6 me=165259us ee=32072us
>>>>>         stat=1 off=32359us freq=-65372 ppb tc=6 me=165259us ee=32072us
>>>>> 2008 Jan  3 09:23:52 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=32404us tc=6 me=152001us ee=30001us
>>>>>         stat=1 off=32404us freq=-65348 ppb tc=6 me=152001us ee=30001us
>>>>> 2008 Jan  3 09:23:59 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-35375us tc=6 me=156564us ee=36902us
>>>>>         stat=1 off=-35375us freq=-65349 ppb tc=6 me=156564us ee=36902us
>>>>> 2008 Jan  3 09:24:24 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-42791us tc=6 me=143992us ee=34618us
>>>>>         stat=1 off=-42791us freq=-65353 ppb tc=6 me=143992us ee=34618us
>>>>> 2008 Jan  3 09:28:42 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-43537us tc=6 me=172839us ee=32384us
>>>>>         stat=1 off=-43537us freq=-65395 ppb tc=6 me=172839us ee=32384us
>>>>> 2008 Jan  3 09:31:26 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=51581us tc=6 me=189296us ee=45261us
>>>>>         stat=1 off=51581us freq=-65363 ppb tc=6 me=189296us ee=45261us
>>>>> 2008 Jan  3 09:34:04 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=57662us tc=6 me=198472us ee=42625us
>>>>>         stat=1 off=57662us freq=-65342 ppb tc=6 me=198472us ee=42625us
>>>>> 2008 Jan  3 09:35:08 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=57305us tc=6 me=202333us ee=39872us
>>>>>         stat=1 off=57305us freq=-65328 ppb tc=6 me=202333us ee=39872us
>>>>> 2008 Jan  3 09:38:25 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-865us tc=6 me=238985us ee=42591us
>>>>>         stat=1 off=-865us freq=-65329 ppb tc=6 me=238985us ee=42591us
>>>>> 2008 Jan  3 09:40:37 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-67929us tc=6 me=222502us ee=46362us
>>>>>         stat=1 off=-67929us freq=-65362 ppb tc=6 me=222502us ee=46362us
>>>>> 2008 Jan  3 09:46:51 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=-31570us tc=6 me=205502us ee=45233us
>>>>>         stat=1 off=-31570us freq=-65406 ppb tc=6 me=205502us ee=45233us
>>>>> 2008 Jan  3 09:51:15 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=5742us tc=6 me=135807us ee=44320us
>>>>>         stat=1 off=5742us freq=-65400 ppb tc=6 me=135807us ee=44320us
>>>>> 2008 Jan  3 09:53:26 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=38830us tc=6 me=142775us ee=43077us
>>>>>         stat=1 off=38830us freq=-65381 ppb tc=6 me=142775us ee=43077us
>>>>> 2008 Jan  3 09:59:58 mode=OFFSET,MERR,EERR,STATUS,TIMECONST,
>>>>>         stat=1 off=10957us tc=6 me=115393us ee=41482us
>>>>>         stat=1 off=10957us freq=-65365 ppb tc=6 me=115393us ee=41482us
>>>>>
>>>>>
>>>>>
>>>>> David L. Mills wrote:
>>>>>
>>>>>    
>>>>>
>>>>>         
>>>>>           
>>>>>> Brian,
>>>>>>
>>>>>> Does v4 work okay on platforms other than Nevada? The kernel works 
>>>>>> fine here with Sol 10 on Blade 1500, Ultra 60, etc., both with and 
>>>>>> without PPS. Just for grins I checked the impulse response on pogo 
>>>>>> with PPS and it behaved exactly as predicted.
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>> How does Nevada differ from other states? Are you using something 
>>>>>> other than Sol 10?
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>> Brian Utterback wrote:
>>>>>>
>>>>>>      
>>>>>>
>>>>>>           
>>>>>>             
>>>>>>> Double checked. It fires each 10ms.
>>>>>>>
>>>>>>> David L. Mills wrote:
>>>>>>>
>>>>>>>        
>>>>>>>
>>>>>>>             
>>>>>>>               
>>>>>>>> Brian,
>>>>>>>>
>>>>>>>> P l e a s e check the tick interrupt frequency. I suspect it is 
>>>>>>>> higher than 100 Hz, in which case as I said before, the frequncy 
>>>>>>>> gain is way too high.. The Alpha Tru64 has been running the 
>>>>>>>> microkernel since 1992 with no ill effects. Note that its timer 
>>>>>>>> frequency is 1024 Hz and that the frequency gain has been 
>>>>>>>> calibrated for that frequency.
>>>>>>>>
>>>>>>>> Dave
>>>>>>>>
>>>>>>>> Brian Utterback wrote:
>>>>>>>>
>>>>>>>>          
>>>>>>>>
>>>>>>>>               
>>>>>>>>                 
>>>>>>>>> I have been testing the V4 code on Solaris Nevada, and have notice
>>>>>>>>> something rather disturbing. On one system I have that has a 
>>>>>>>>> measured
>>>>>>>>> frequency correction of 28.5 ppm, it takes the version 3 xntpd 
>>>>>>>>> about
>>>>>>>>> an hour to get to plus or minus 2 or 3, and then settles on 28.5 
>>>>>>>>> in about
>>>>>>>>> 10 hours. However, the V4 code jumps to something between 40 and 
>>>>>>>>> 60,
>>>>>>>>> and then slowly moved towards 28.5 over about a week and a half. 
>>>>>>>>> This
>>>>>>>>> is not acceptable performance and is a showstopper for upgrading to
>>>>>>>>> the latest builds in Solaris.
>>>>>>>>>
>>>>>>>>> Now, looking at some of the documentation that Dave has 
>>>>>>>>> published, there
>>>>>>>>> is a mention that if you run a micro kernel with the V4 code the 
>>>>>>>>> time constant
>>>>>>>>> will be too big. Could this be the problem? Solaris is indeed 
>>>>>>>>> using the micro
>>>>>>>>> kernel ntp_adjtime. If it is, I would think that it would be 
>>>>>>>>> possible to correct
>>>>>>>>> for this at run time.
>>>>>>>>>
>>>>>>>>> Oh, and if :disable kernel" is configured, then the V4 code gets 
>>>>>>>>> the right value
>>>>>>>>> in just a few hours, just like the V3 code. Any ideas on how to 
>>>>>>>>> fix this?
>>>>>>>>>
>>>>>>>>> Brian Utterback
>>>>>>>>>             
>>>>>>>>>                 
>>>>>>>>>                   
>>>>>>>> _______________________________________________
>>>>>>>> hackers mailing list
>>>>>>>> hackers at lists.ntp.org
>>>>>>>> https://lists.ntp.org/mailman/listinfo/hackers
>>>>>>>>           
>>>>>>>>               
>>>>>>>>                 
>>>>>>>         
>>>>>>>             
>>>>>>>               
>>>>>> _______________________________________________
>>>>>> hackers mailing list
>>>>>> hackers at lists.ntp.org
>>>>>> https://lists.ntp.org/mailman/listinfo/hackers
>>>>>>       
>>>>>>           
>>>>>>             
>>>>>     
>>>>>         
>>>>>           
>>>> _______________________________________________
>>>> hackers mailing list
>>>> hackers at lists.ntp.org
>>>> https://lists.ntp.org/mailman/listinfo/hackers
>>>>   
>>>>       
>>>>         
>>>     
>>>       
>> _______________________________________________
>> hackers mailing list
>> hackers at lists.ntp.org
>> https://lists.ntp.org/mailman/listinfo/hackers
>>   
>>     
>
> _______________________________________________
> hackers mailing list
> hackers at lists.ntp.org
> https://lists.ntp.org/mailman/listinfo/hackers
>   



More information about the hackers mailing list