[ntp:questions] Isolated Network Drift Problem
Cal Webster
cwebster at ec.rr.com
Thu Nov 20 15:25:13 UTC 2008
Our NTP servers are slowly loosing time. All are in nearly perfect sync
but collectively drift backwards over time. Is there a way to apply a
bias to the drift calculations?
We had to disconnect from the Internet several months ago. Since then we
have had serious drift problems. Shortly after the disconnect I
discovered that we were predictably loosing 10 minutes every 15 days. I
tried several things but not until I zeroed out the
"driftfile" (/var/lib/ntp/drift) 10 days ago [Mon Nov 10 18:10:00 2008]
did this large drift abate.
Although it is much improved, we are still steadily loosing time. Three
days after I zeroed the drift file [Thu Nov 13 15:04:00 EST 2008] we
were 32 seconds behind. Today, 10 days later [Thu Nov 20 09:05:00 2008]
we are 1 min 54 secs behind. This works out to roughly 12 secs per day -
not bad I guess but still requires regular monitoring.
Any suggestions would be welcome. I've outlined our NTP/Network topology
and configuration below, as well as the output of some time reference
tools that might be relevant.
Let me know if you need more info. Thank you in advance!
Cal Webster
##############
## Topology ##
##############
Notes:
1. Links between routers are busy ADSL lines with approx 512 Mbps
throughput.
2. Host names are shown in parenthesis for cross-referencing data.
[router]---[LAN A]
| |
| +--[NTP 1 (axl)]
|
[router]---[LAN B]---[LAN B.1]
| |
| +--[NTP 2 (pegasus)]
|
[router]---[LAN C]
|
+--[NTP 3 (jato)]
+--[NTP 4 (fluid)]
###################
## Configuration ##
###################
[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.1.30 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
peer 192.168.3.132
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0
fudge 127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.1.30
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 8
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 5
driftfile /etc/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.2.6
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
##############
## Topology ##
##############
Notes:
1. Links between routers are ADSL lines with approx 512 Mbps throughput.
2. Host names are shown in parenthesis for cross-referencing data.
[router]---[LAN A]
| |
| +--[NTP 1 (axl)]
|
[router]---[LAN B]---[LAN B.1]
| |
| +--[NTP 2 (pegasus)]
|
[router]---[LAN C]
|
+--[NTP 3 (jato)]
+--[NTP 4 (fluid)]
###################
## Configuration ##
###################
[root at jato ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.1.30 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
peer 192.168.3.132
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0
fudge 127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at pegasus ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict default notrust nomodify notrap noquery
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.1.30
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 8
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at fluid root]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
restrict 192.168.1.30 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.1.30
peer 192.168.2.6
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 5
driftfile /etc/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
[root at axl ~]# egrep -v '^[[:space:]]*$|^#' /etc/ntp.conf
---------------------------------------------------------
restrict 127.0.0.1
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
restrict 10.1.150.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.3.9 mask 255.255.255.255
restrict 192.168.3.132 mask 255.255.255.255
restrict 192.168.2.6 mask 255.255.255.255
peer 192.168.3.9
peer 192.168.3.132
peer 192.168.2.6
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 5
driftfile /var/lib/ntp/drift
broadcastdelay 0.008
keys /etc/ntp/keys
---------------------------------------------------------
##########################
## Current Drift Values ##
##########################
[root at jato /]# cat /var/lib/ntp/drift
-------------------------------------
-99.534
-------------------------------------
[root at pegasus /]# cat /var/lib/ntp/drift
-------------------------------------
-26.817
-------------------------------------
[root at fluid /]# cat /etc/ntp/drift
-------------------------------------
-81.140
-------------------------------------
[root at axl /]# cat /var/lib/ntp/drift
-------------------------------------
17.653
-------------------------------------
##################
## NTPQ Queries ##
##################
[root at jato backup]# ntpq
------------------------
ntpq> pe
remote refid st t when poll reach delay offset
jitter
==============================================================================
fluid 192.168.3.9 7 u 831 1024 376 0.368 -0.896
1.275
axl 192.168.3.9 7 u 951 1024 376 25.263 -0.776
0.464
pegasus 192.168.3.9 7 u 28 1024 377 24.597 -1.228
0.298
*LOCAL(0) LOCAL(0) 5 l 8 64 377 0.000 0.000
0.004
ntpq> as
ind assID status conf reach auth condition last_event cnt
===========================================================
1 46140 9014 yes yes none reject reachable 1
2 46141 9034 yes yes none reject reachable 3
3 46142 9054 yes yes none reject reachable 5
4 46143 9614 yes yes none sys.peer reachable 1
------------------------
[root at pegasus ECP Data]# ntpq
------------------------
ntpq> pe
remote refid st t when poll reach delay offset
jitter
==============================================================================
*jato LOCAL(0) 6 u 59 1024 376 24.853 1.356
0.976
+fluid 192.168.3.9 7 u 352 1024 377 24.737 0.568
0.159
+axl 192.168.3.9 7 u 194 1024 377 32.219 0.475
0.342
LOCAL(0) LOCAL(0) 8 l 41 64 377 0.000 0.000
0.001
ntpq> as
ind assID status conf reach auth condition last_event cnt
===========================================================
1 37660 96f4 yes yes none sys.peer reachable 15
2 37661 94f4 yes yes none candidat reachable 15
3 37662 94f4 yes yes none candidat reachable 15
4 37663 90b4 yes yes none reject reachable 11
------------------------
[root at fluid cwebster]# ntpq
------------------------
ntpq> pe
remote refid st t when poll reach delay offset
jitter
==============================================================================
*jato LOCAL(0) 6 u 138 1024 377 0.395 0.883
0.494
+axl jato 7 u 897 1024 376 27.010 -1.833
0.974
+pegasus jato 7 u 884 1024 376 25.082 -0.396
0.298
LOCAL(0) LOCAL(0) 5 l 35 64 377 0.000 0.000
0.008
ntpq> as
ind assID status conf reach auth condition last_event cnt
===========================================================
1 47588 9694 yes yes none sys.peer reachable 9
2 47589 94f4 yes yes none candidat reachable 15
3 47590 9494 yes yes none candidat reachable 9
4 47591 9074 yes yes none reject reachable 7
------------------------
[root at axl /]# ntpq
------------------------
ntpq> pe
remote refid st t when poll reach delay offset
jitter
==============================================================================
*jato LOCAL(0) 6 u 674 1024 377 24.492 1.161
0.082
fluid 192.168.3.9 7 u 846 1024 377 24.673 0.665
1.095
pegasus 192.168.3.9 7 u 523 1024 376 32.235 -0.018
0.340
LOCAL(0) .LOCL. 5 l 23 64 377 0.000 0.000
0.001
ntpq> as
ind assID status conf reach auth condition last_event cnt
===========================================================
1 23414 96a4 yes yes none sys.peer reachable 10
2 23415 9064 yes yes none reject reachable 6
3 23416 9084 yes yes none reject reachable 8
4 23417 9064 yes yes none reject reachable 6
------------------------
#########################
## Hardware Clock Info ##
#########################
Note: There are no precision clocks. All NTP servers use standard quartz
based clocks on system boards.
[root at jato backup]# cat /etc/adjtime
------------------------------------
0.146704 1226952522 0.000000
1226952522
LOCAL
------------------------------------
[root at jato backup]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.12a
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226952522 seconds after 1969
Last calibration done at 1226952522 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST -0.852202 seconds
------------------------------------------
[root at pegasus ECP Data]# cat /etc/adjtime
------------------------------------
27.446035 1226358312 0.000000
1226358312
LOCAL
------------------------------------
[root at pegasus ECP Data]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.12a
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226358312 seconds after 1969
Last calibration done at 1226358312 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:55
Hw clock time : 2008/11/20 09:07:55 = 1227190075 seconds since 1969
Thu 20 Nov 2008 09:07:55 AM EST -0.016559 seconds
------------------------------------------
[root at fluid cwebster]# cat /etc/adjtime
------------------------------------
-0.025420 1226360186 0.000000
1226360186
LOCAL
------------------------------------
[root at fluid cwebster]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-2.11y
Using /dev/rtc interface to clock.
Last drift adjustment done at 1226360186 seconds after 1969
Last calibration done at 1226360186 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST -0.232158 seconds
------------------------------------------
[root at axl /]# cat /etc/adjtime
------------------------------------
44.508790 1226358437 0.000000
1226358437
LOCAL
------------------------------------
[root at axl /]# hwclock --test --debug
------------------------------------------
hwclock from util-linux-ng 2.13
Using /dev interface to clock.
Last drift adjustment done at 1226358437 seconds after 1969
Last calibration done at 1226358437 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2008/11/20 09:07:56
Hw clock time : 2008/11/20 09:07:56 = 1227190076 seconds since 1969
Thu 20 Nov 2008 09:07:56 AM EST -0.001092 seconds
------------------------------------------
More information about the questions
mailing list