[ntp:questions] Syncing and peering for a multi-continent deployment

David Lord snews at lordynet.org
Mon Apr 8 13:53:28 UTC 2013


Blair Zajac wrote:
> I'm working on deploying a large number of servers to data centers on 
> three continents, say US, Europe and India.  Because each data center 
> will be writing data and timestamping the data, I want them all to think 
> they are at the same time, even if it isn't exactly the correct UTC 
> time.  The data will be synced to other data centers, so relative 
> ordering of when something happened can be important.
> 
> To not ping stratum-1 and -2 level servers from a large number of boxes, 
> I'm setting up three stratum-3 NTP servers in each data center.
> 
> Here's what I've read:
> 
> 1) Each ntp server should have 4 to 7 upstream clocks [1], [2], [3].
> 
> 2) The three stratum-3 NTP servers in each data center should be peers 
> of each other [4].  So each NTP server would have at most 2+7=9 
> connections.
> 
> Questions:
> 
> 1) Should each data center use nearby stratum-2 clocks or pick a set of 
> stratum-2 clocks that are network wise in the center between all data 
> centers (it may not be possible to get a true center)?
> 
> 2) Should each stratum-3 server in a single data center use all 4-7 
> upstream clocks?  Or should the 4-7 be split between the three stratum-3 
> servers?  The diagram at [4] suggests they should be split?  If that's 
> the case, then how are falsetickers identified?
> 
> 3) Should I have all the stratum-3 servers in each data center be peers 
> of each other, so each would have 8 peers?  This would ensure that all 
> my clocks think they are at the same time.  If I do this, then I would 
> need to increase maxclocks to support up to 7 (upstream) + 8 (peers)=15 
> clocks?
> 
> 4) Or, should I not peer data centers to each other and trust that the 
> stratum-2 clocks near each data center will be close to the other clocks?
> 
> Other random questions:
> 
> 1) It doesn't appear that its necessary to set up symmetric keys for peers?
> 
> 2) Is there a way to tell which peer is the "master" peer?  Looking 
> through ntpdc and ntpq I didn't see anything.
> 
> Thanks,
> Blair
> 
> [1] 
> http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Secti
> [2] 
> http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Secti
> [3] http://support.ntp.org/bin/view/Support/StartingNTP4#Section_7.1.4.3.2.
> [4] http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3101


What level of time offsets would you find acceptable?

 From peer_summary                           mean     rms  maximum
                                             (us)    (us)     (us)

Internet - ntp3.lordynet.org                 500     500     1000

Lan - ntp3.lordynet.org                       50     250      700
       (no temperature control)

PPS - me6000e.home.lordynet.org                0       5       33
       (no temperature control)

Informed guesses:
PPS (constant ambient temperature)             0       4       15
PPS (OCXO system clock)                    0.050   0.100

Most of variation on my lan is due to change of ambient
temperature whilst for my PPS synced pc it is from both changes
of system temperature under load and ambient temperature.


Below, for ntp3, the selected peer is shown from "ntpq -c rv" by
refid=81.187.61.78 and from "ntpq -p" by the "*" in the first
column.

For "me6000e", refid=PPSb and from "ntpq -p" by the "o" in the
first column whilst the preferred peer indicated by the "*" has a
large offset of 8.666 msec.


David

#----------------------------------------------------------------

# ntp3.lordynet.org
# ntpq -c rv -p ntp3.lordynet.org
associd=0 status=061d leap_none, sync_ntp, 1 event, kern,
version="ntpd 4.2.6p5-o Wed Feb  1 07:49:06 UTC 2012 (import)",
processor="i386", system="NetBSD/6.1_RC2", leap=00, stratum=2,
precision=-18, rootdelay=0.405, rootdisp=11.529, refid=81.187.61.78,
                                                  ------------------
reftime=d50d3ca0.b65a4f90  Mon, Apr  8 2013 12:54:56.712,
clock=d50d3eab.e868dd3e  Mon, Apr  8 2013 13:03:39.907, peer=1276, tc=6,
mintc=3, offset=0.204, frequency=-10.411, sys_jitter=0.079,
clk_jitter=0.029, clk_wander=0.004, tai=35, leapsec=201207010000,
expire=201306280000

Mon Apr  8 12:53:00 GMT 2013
      remote           refid      st t when poll reach   delay   offset 
  jitter
==============================================================================
  ns0.lordynet.or 81.187.61.65     3 u   41   64  376    0.640   -0.546 
   0.517
*ns2.lordynet.or .MSFa.           1 u   12   64  377    0.457    0.293 
  0.036
+??????????????? ??????????????   2 u  124  256  377   15.147    0.131 
  0.179
+??????????????? ??????????????   2 u   82  256  377   15.688    0.096 
  0.112
-??????????????? ??????????????   2 u   44  256  377   21.732   -0.078 
  0.115
-??????????????? ??????????????   2 u   19  256  377   23.610    0.048 
  0.164
-??????????????? ??????????????   3 u  210  256  377   26.161   -1.134 
  1.266

#----------------------------------------------------------------

# me6000e
# peerstats.me6000e.20130407
        ident     cnt     mean     rms      max     delay     dist     disp
==========================================================================
127.127.20.2    1350   -0.256   17.910   68.071    0.000    2.564    1.742
127.127.22.2    1350    0.000    0.005    0.033    0.000    0.930    0.930
81.187.61.65      70    1.089    0.297    0.580    0.974   23.814   11.149
81.187.61.74     288    1.037    0.555    1.457    1.481    7.191    3.117
81.187.61.69     292    1.042    0.196    0.619    0.826    6.208    2.852
?????????????     69    0.775    0.269    0.702   16.096   30.449   11.337
81.187.61.78      75    1.035    0.227    0.515    0.749   23.093   11.335

# ntpq -c rv -p me6000e
associd=0 status=01fd leap_none, sync_pps, 15 events, kern,
version="ntpd 4.2.6p5-o Wed Feb  1 07:49:06 UTC 2012 (import)",
processor="i386", system="NetBSD/6.1_RC2", leap=00, stratum=1,
precision=-18, rootdelay=0.000, rootdisp=1.705, refid=PPSb,
reftime=d50d40c7.76bd4a0f  Mon, Apr  8 2013 13:12:39.463,
clock=d50d40fa.eede15ee  Mon, Apr  8 2013 13:13:30.933, peer=58635, tc=6,
mintc=3, offset=0.005, frequency=-35.769, sys_jitter=0.005,
clk_jitter=0.004, clk_wander=0.009, tai=35, leapsec=201207010000,
expire=201306280000

      remote           refid      st t when poll reach   delay   offset 
  jitter
==============================================================================
*GPS_NMEA(2)     .GPSb.           4 l   53   64  377    0.000    8.666 
11.460
oPPS(2)          .PPSb.           0 l   51   64  377    0.000    0.005 
  0.005
+ns0.lordynet.or 81.187.61.65     3 u   18   64  377    1.424    0.421 
  0.472
+ns2.lordynet.or .MSFa.           1 u   40  256  377    0.439    1.111 
  0.562
+ns1.lordynet.or 81.187.61.78     2 u   40   64  377    0.699    1.219 
  0.528
+ns3.lordynet.or 81.187.61.78     2 u  193  256  377    1.081    0.751 
  0.599
+??????????????? ?????????????    2 u  181  256  377   15.920    1.074 
  0.760
#----------------------------------------------------------------



More information about the questions mailing list