[ntp:questions] Isolated Network Drift Problem

Unruh unruh-spam at physics.ubc.ca
Fri Nov 21 02:48:42 UTC 2008


cwebster at ec.rr.com (Cal Webster) writes:

>Our NTP servers are slowly loosing time. All are in nearly perfect sync
>but collectively drift backwards over time. Is there a way to apply a
>bias to the drift calculations?



>We had to disconnect from the Internet several months ago. Since then we
>have had serious drift problems. Shortly after the disconnect I
>discovered that we were predictably loosing 10 minutes every 15 days. I
>tried several things but not until I zeroed out the
>"driftfile" (/var/lib/ntp/drift) 10 days ago [Mon Nov 10 18:10:00 2008]
>did this large drift abate.

You might want to try using chrony assuming your machines are Linux
machines. One of the features of chrony is the ability to put in "watch"
time-- ie manually set the time. Now this will NOT give you ms accuracy,
but it will use those times to discipline the drift of the clocks just as
it would with regular ntp servers. 10min/15days is about 1000PPM which
ntp  would not be able to fix even if it were running. 

alternatively you can do the drift yourself. Look at the adjtimex console
command. You can alter the tick rate to try to bring the main  server
(jato?) rate into  reasonable agreemetn with the truth.  

>Although it is much improved, we are still steadily loosing time. Three
>days after I zeroed the drift file [Thu Nov 13 15:04:00 EST 2008] we
>were 32 seconds behind. Today, 10 days later [Thu Nov 20 09:05:00 2008]
>we are 1 min 54 secs behind. This works out to roughly 12 secs per day -
>not bad I guess but still requires regular monitoring.

That is about 25PPM.


>Any suggestions would be welcome. I've outlined our NTP/Network topology
>and configuration below, as well as the output of some time reference
>tools that might be relevant.

>Let me know if you need more info. Thank you in advance!

>Cal Webster

>##############
>## Topology ##
>##############

>Notes:

>1. Links between routers are busy ADSL lines with approx 512 Mbps
>throughput.
>2. Host names are shown in parenthesis for cross-referencing data.

>[router]---[LAN A]
>    |         |
>    |         +--[NTP 1 (axl)]
>    |
>[router]---[LAN B]---[LAN B.1]
>    |         |
>    |         +--[NTP 2 (pegasus)]
>    |
>[router]---[LAN C]
>              |
>              +--[NTP 3 (jato)]
>              +--[NTP 4 (fluid)]


>###################
>## Configuration ##
>###################

>[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict default notrust nomodify notrap noquery
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.1.30 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>peer 192.168.3.132
>peer 192.168.1.30
>peer 192.168.2.6
>server 127.127.1.0
>fudge   127.127.1.0 stratum 5
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict default notrust nomodify notrap noquery
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>restrict 192.168.1.30 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.3.132
>peer 192.168.1.30
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 8
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>restrict 192.168.1.30 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.1.30
>peer 192.168.2.6
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 5
>driftfile /etc/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.3.132
>peer 192.168.2.6
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 5
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------
>##############
>## Topology ##
>##############

>Notes:

>1. Links between routers are ADSL lines with approx 512 Mbps throughput.
>2. Host names are shown in parenthesis for cross-referencing data.

>[router]---[LAN A]
>    |         |
>    |         +--[NTP 1 (axl)]
>    |
>[router]---[LAN B]---[LAN B.1]
>    |         |
>    |         +--[NTP 2 (pegasus)]
>    |
>[router]---[LAN C]
>              |
>              +--[NTP 3 (jato)]
>              +--[NTP 4 (fluid)]


>###################
>## Configuration ##
>###################

>[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict default notrust nomodify notrap noquery
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.1.30 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>peer 192.168.3.132
>peer 192.168.1.30
>peer 192.168.2.6
>server 127.127.1.0
>fudge   127.127.1.0 stratum 5
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict default notrust nomodify notrap noquery
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>restrict 192.168.1.30 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.3.132
>peer 192.168.1.30
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 8
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>restrict 192.168.1.30 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.1.30
>peer 192.168.2.6
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 5
>driftfile /etc/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------

>[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
>---------------------------------------------------------
>restrict 127.0.0.1
>restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
>restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
>restrict 192.168.3.9 mask 255.255.255.255
>restrict 192.168.3.132 mask 255.255.255.255
>restrict 192.168.2.6 mask 255.255.255.255
>peer 192.168.3.9
>peer 192.168.3.132
>peer 192.168.2.6
>server  127.127.1.0     # local clock
>fudge   127.127.1.0 stratum 5
>driftfile /var/lib/ntp/drift
>broadcastdelay  0.008
>keys            /etc/ntp/keys
>---------------------------------------------------------


>##########################
>## Current Drift Values ##
>##########################

>[root at jato /]# cat /var/lib/ntp/drift
>-------------------------------------
>-99.534
>-------------------------------------

>[root at pegasus /]# cat /var/lib/ntp/drift
>-------------------------------------
>-26.817
>-------------------------------------

>[root at fluid /]# cat /etc/ntp/drift
>-------------------------------------
>-81.140
>-------------------------------------

>[root at axl /]# cat /var/lib/ntp/drift
>-------------------------------------
>17.653
>-------------------------------------


>##################
>## NTPQ Queries ##
>##################

>[root at jato backup]# ntpq
>------------------------
>ntpq> pe
>     remote           refid      st t when poll reach   delay   offset
>jitter
>==============================================================================
> fluid           192.168.3.9      7 u  831 1024  376    0.368   -0.896
>1.275
> axl             192.168.3.9      7 u  951 1024  376   25.263   -0.776
>0.464
> pegasus         192.168.3.9      7 u   28 1024  377   24.597   -1.228
>0.298
>*LOCAL(0)        LOCAL(0)         5 l    8   64  377    0.000    0.000
>0.004
>ntpq> as
> 
>ind assID status  conf reach auth condition  last_event cnt
>===========================================================
>  1 46140  9014   yes   yes  none    reject   reachable  1
>  2 46141  9034   yes   yes  none    reject   reachable  3
>  3 46142  9054   yes   yes  none    reject   reachable  5
>  4 46143  9614   yes   yes  none  sys.peer   reachable  1
>------------------------

>[root at pegasus ECP Data]# ntpq
>------------------------
>ntpq> pe
>     remote           refid      st t when poll reach   delay   offset
>jitter
>==============================================================================
>*jato            LOCAL(0)         6 u   59 1024  376   24.853    1.356
>0.976
>+fluid           192.168.3.9      7 u  352 1024  377   24.737    0.568
>0.159
>+axl             192.168.3.9      7 u  194 1024  377   32.219    0.475
>0.342
> LOCAL(0)        LOCAL(0)         8 l   41   64  377    0.000    0.000
>0.001
>ntpq> as
> 
>ind assID status  conf reach auth condition  last_event cnt
>===========================================================
>  1 37660  96f4   yes   yes  none  sys.peer   reachable 15
>  2 37661  94f4   yes   yes  none  candidat   reachable 15
>  3 37662  94f4   yes   yes  none  candidat   reachable 15
>  4 37663  90b4   yes   yes  none    reject   reachable 11
>------------------------

>[root at fluid cwebster]# ntpq
>------------------------
>ntpq> pe
>     remote           refid      st t when poll reach   delay   offset
>jitter
>==============================================================================
>*jato            LOCAL(0)         6 u  138 1024  377    0.395    0.883
>0.494
>+axl             jato             7 u  897 1024  376   27.010   -1.833
>0.974
>+pegasus         jato             7 u  884 1024  376   25.082   -0.396
>0.298
> LOCAL(0)        LOCAL(0)         5 l   35   64  377    0.000    0.000
>0.008
>ntpq> as
>ind assID status  conf reach auth condition  last_event cnt
>===========================================================
>  1 47588  9694   yes   yes  none  sys.peer   reachable  9
>  2 47589  94f4   yes   yes  none  candidat   reachable 15
>  3 47590  9494   yes   yes  none  candidat   reachable  9
>  4 47591  9074   yes   yes  none    reject   reachable  7
>------------------------

>[root at axl /]# ntpq
>------------------------
>ntpq> pe
>     remote           refid      st t when poll reach   delay   offset
>jitter
>==============================================================================
>*jato            LOCAL(0)         6 u  674 1024  377   24.492    1.161
>0.082
> fluid           192.168.3.9      7 u  846 1024  377   24.673    0.665
>1.095
> pegasus         192.168.3.9      7 u  523 1024  376   32.235   -0.018
>0.340
> LOCAL(0)        .LOCL.           5 l   23   64  377    0.000    0.000
>0.001
>ntpq> as
> 
>ind assID status  conf reach auth condition  last_event cnt
>===========================================================
>  1 23414  96a4   yes   yes  none  sys.peer   reachable 10
>  2 23415  9064   yes   yes  none    reject   reachable  6
>  3 23416  9084   yes   yes  none    reject   reachable  8
>  4 23417  9064   yes   yes  none    reject   reachable  6
>------------------------



>#########################
>## Hardware Clock Info ##
>#########################

>Note: There are no precision clocks. All NTP servers use standard quartz
>based clocks on system boards.


>[root at jato backup]# cat /etc/adjtime
>------------------------------------
>0.146704 1226952522 0.000000
>1226952522
>LOCAL
>------------------------------------

>[root at jato backup]# hwclock --test --debug
>------------------------------------------
>hwclock from util-linux-2.12a
>Using /dev/rtc interface to clock.
>Last drift adjustment done at 1226952522 seconds after 1969
>Last calibration done at 1226952522 seconds after 1969
>Hardware clock is on local time
>Assuming hardware clock is kept in local time.
>Waiting for clock tick...
>...got clock tick
>Time read from Hardware Clock: 2008/11/20 09:07:56
>Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
>Thu 20 Nov 2008 09:07:56 AM EST  -0.852202 seconds
>------------------------------------------



>[root at pegasus ECP Data]# cat /etc/adjtime
>------------------------------------
>27.446035 1226358312 0.000000
>1226358312
>LOCAL
>------------------------------------

>[root at pegasus ECP Data]# hwclock --test --debug
>------------------------------------------
>hwclock from util-linux-2.12a
>Using /dev/rtc interface to clock.
>Last drift adjustment done at 1226358312 seconds after 1969
>Last calibration done at 1226358312 seconds after 1969
>Hardware clock is on local time
>Assuming hardware clock is kept in local time.
>Waiting for clock tick...
>...got clock tick
>Time read from Hardware Clock: 2008/11/20 09:07:55
>Hw clock time : 2008/11/20 09:07:55 = 1227190075 seconds since 1969
>Thu 20 Nov 2008 09:07:55 AM EST  -0.016559 seconds
>------------------------------------------



>[root at fluid cwebster]# cat /etc/adjtime
>------------------------------------
>-0.025420 1226360186 0.000000
>1226360186
>LOCAL
>------------------------------------

>[root at fluid cwebster]# hwclock --test --debug
>------------------------------------------
>hwclock from util-linux-2.11y
>Using /dev/rtc interface to clock.
>Last drift adjustment done at 1226360186 seconds after 1969
>Last calibration done at 1226360186 seconds after 1969
>Hardware clock is on local time
>Assuming hardware clock is kept in local time.
>Waiting for clock tick...
>...got clock tick
>Time read from Hardware Clock: 2008/11/20 09:07:56
>Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
>Thu 20 Nov 2008 09:07:56 AM EST  -0.232158 seconds
>------------------------------------------



>[root at axl /]# cat /etc/adjtime
>------------------------------------
>44.508790 1226358437 0.000000
>1226358437
>LOCAL
>------------------------------------

>[root at axl /]# hwclock --test --debug
>------------------------------------------
>hwclock from util-linux-ng 2.13
>Using /dev interface to clock.
>Last drift adjustment done at 1226358437 seconds after 1969
>Last calibration done at 1226358437 seconds after 1969
>Hardware clock is on local time
>Assuming hardware clock is kept in local time.
>Waiting for clock tick...
>...got clock tick
>Time read from Hardware Clock: 2008/11/20 09:07:56
>Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
>Thu 20 Nov 2008 09:07:56 AM EST  -0.001092 seconds
>------------------------------------------




More information about the questions mailing list