[ntp:questions] Isolated Network Drift Problem

Cal Webster cwebster at ec.rr.com
Thu Nov 20 15:25:13 UTC 2008


Our NTP servers are slowly loosing time. All are in nearly perfect sync
but collectively drift backwards over time. Is there a way to apply a
bias to the drift calculations?

We had to disconnect from the Internet several months ago. Since then we
have had serious drift problems. Shortly after the disconnect I
discovered that we were predictably loosing 10 minutes every 15 days. I
tried several things but not until I zeroed out the
"driftfile" (/var/lib/ntp/drift) 10 days ago [Mon Nov 10 18:10:00 2008]
did this large drift abate.

Although it is much improved, we are still steadily loosing time. Three
days after I zeroed the drift file [Thu Nov 13 15:04:00 EST 2008] we
were 32 seconds behind. Today, 10 days later [Thu Nov 20 09:05:00 2008]
we are 1 min 54 secs behind. This works out to roughly 12 secs per day -
not bad I guess but still requires regular monitoring.

Any suggestions would be welcome. I've outlined our NTP/Network topology
and configuration below, as well as the output of some time reference
tools that might be relevant.

Let me know if you need more info. Thank you in advance!

Cal Webster

##############
## Topology ##
##############

Notes:

1. Links between routers are busy ADSL lines with approx 512 Mbps
throughput.
2. Host names are shown in parenthesis for cross-referencing data.

[router]---[LAN A]
    |         |
    |         +--[NTP 1 (axl)]
    |
[router]---[LAN B]---[LAN B.1]
    |         |
    |         +--[NTP 2 (pegasus)]
    |
[router]---[LAN C]
              |
              +--[NTP 3 (jato)]
              +--[NTP 4 (fluid)]


###################
## Configuration ##
###################

[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.1.30 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
peer 192.168.3.132
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0
fudge   127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.1.30
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 8
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.1.30
peer 192.168.2.6
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 5
driftfile /etc/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.2.6
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------
##############
## Topology ##
##############

Notes:

1. Links between routers are ADSL lines with approx 512 Mbps throughput.
2. Host names are shown in parenthesis for cross-referencing data.

[router]---[LAN A]
    |         |
    |         +--[NTP 1 (axl)]
    |
[router]---[LAN B]---[LAN B.1]
    |         |
    |         +--[NTP 2 (pegasus)]
    |
[router]---[LAN C]
              |
              +--[NTP 3 (jato)]
              +--[NTP 4 (fluid)]


###################
## Configuration ##
###################

[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.1.30 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
peer 192.168.3.132
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0
fudge   127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.1.30
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 8
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.1.30
peer 192.168.2.6
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 5
driftfile /etc/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------

[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.2.6
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay  0.008
keys            /etc/ntp/keys
---------------------------------------------------------


##########################
## Current Drift Values ##
##########################

[root at jato /]# cat /var/lib/ntp/drift
-------------------------------------
-99.534
-------------------------------------

[root at pegasus /]# cat /var/lib/ntp/drift
-------------------------------------
-26.817
-------------------------------------

[root at fluid /]# cat /etc/ntp/drift
-------------------------------------
-81.140
-------------------------------------

[root at axl /]# cat /var/lib/ntp/drift
-------------------------------------
17.653
-------------------------------------


##################
## NTPQ Queries ##
##################

[root at jato backup]# ntpq
------------------------
ntpq> pe
     remote           refid      st t when poll reach   delay   offset
jitter
==============================================================================
 fluid           192.168.3.9      7 u  831 1024  376    0.368   -0.896
1.275
 axl             192.168.3.9      7 u  951 1024  376   25.263   -0.776
0.464
 pegasus         192.168.3.9      7 u   28 1024  377   24.597   -1.228
0.298
*LOCAL(0)        LOCAL(0)         5 l    8   64  377    0.000    0.000
0.004
ntpq> as
 
ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1 46140  9014   yes   yes  none    reject   reachable  1
  2 46141  9034   yes   yes  none    reject   reachable  3
  3 46142  9054   yes   yes  none    reject   reachable  5
  4 46143  9614   yes   yes  none  sys.peer   reachable  1
------------------------

[root at pegasus ECP Data]# ntpq
------------------------
ntpq> pe
     remote           refid      st t when poll reach   delay   offset
jitter
==============================================================================
*jato            LOCAL(0)         6 u   59 1024  376   24.853    1.356
0.976
+fluid           192.168.3.9      7 u  352 1024  377   24.737    0.568
0.159
+axl             192.168.3.9      7 u  194 1024  377   32.219    0.475
0.342
 LOCAL(0)        LOCAL(0)         8 l   41   64  377    0.000    0.000
0.001
ntpq> as
 
ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1 37660  96f4   yes   yes  none  sys.peer   reachable 15
  2 37661  94f4   yes   yes  none  candidat   reachable 15
  3 37662  94f4   yes   yes  none  candidat   reachable 15
  4 37663  90b4   yes   yes  none    reject   reachable 11
------------------------

[root at fluid cwebster]# ntpq
------------------------
ntpq> pe
     remote           refid      st t when poll reach   delay   offset
jitter
==============================================================================
*jato            LOCAL(0)         6 u  138 1024  377    0.395    0.883
0.494
+axl             jato             7 u  897 1024  376   27.010   -1.833
0.974
+pegasus         jato             7 u  884 1024  376   25.082   -0.396
0.298
 LOCAL(0)        LOCAL(0)         5 l   35   64  377    0.000    0.000
0.008
ntpq> as
ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1 47588  9694   yes   yes  none  sys.peer   reachable  9
  2 47589  94f4   yes   yes  none  candidat   reachable 15
  3 47590  9494   yes   yes  none  candidat   reachable  9
  4 47591  9074   yes   yes  none    reject   reachable  7
------------------------

[root at axl /]# ntpq
------------------------
ntpq> pe
     remote           refid      st t when poll reach   delay   offset
jitter
==============================================================================
*jato            LOCAL(0)         6 u  674 1024  377   24.492    1.161
0.082
 fluid           192.168.3.9      7 u  846 1024  377   24.673    0.665
1.095
 pegasus         192.168.3.9      7 u  523 1024  376   32.235   -0.018
0.340
 LOCAL(0)        .LOCL.           5 l   23   64  377    0.000    0.000
0.001
ntpq> as
 
ind assID status  conf reach auth condition  last_event cnt
===========================================================
  1 23414  96a4   yes   yes  none  sys.peer   reachable 10
  2 23415  9064   yes   yes  none    reject   reachable  6
  3 23416  9084   yes   yes  none    reject   reachable  8
  4 23417  9064   yes   yes  none    reject   reachable  6
------------------------



#########################
## Hardware Clock Info ##
#########################

Note: There are no precision clocks. All NTP servers use standard quartz
based clocks on system boards.


[root at jato backup]# cat /etc/adjtime
------------------------------------
0.146704 1226952522 0.000000
1226952522
LOCAL
------------------------------------

[root at jato backup]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.12a
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226952522 seconds after 1969
Last calibration done at 1226952522 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST  -0.852202 seconds
------------------------------------------



[root at pegasus ECP Data]# cat /etc/adjtime
------------------------------------
27.446035 1226358312 0.000000
1226358312
LOCAL
------------------------------------

[root at pegasus ECP Data]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.12a
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226358312 seconds after 1969
Last calibration done at 1226358312 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:55
Hw clock time : 2008/11/20 09:07:55 = 1227190075 seconds since 1969
Thu 20 Nov 2008 09:07:55 AM EST  -0.016559 seconds
------------------------------------------



[root at fluid cwebster]# cat /etc/adjtime
------------------------------------
-0.025420 1226360186 0.000000
1226360186
LOCAL
------------------------------------

[root at fluid cwebster]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.11y
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226360186 seconds after 1969
Last calibration done at 1226360186 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST  -0.232158 seconds
------------------------------------------



[root at axl /]# cat /etc/adjtime
------------------------------------
44.508790 1226358437 0.000000
1226358437
LOCAL
------------------------------------

[root at axl /]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-ng 2.13
Using /dev interface to clock.
Last drift adjustment done at 1226358437 seconds after 1969
Last calibration done at 1226358437 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST  -0.001092 seconds
------------------------------------------





More information about the questions mailing list