[ntp:questions] Sub-millisecond NTP synchronization for local network

Kevin Oberman oberman at es.net
Sun Dec 7 16:50:06 UTC 2008


> From: Terje Mathisen <terje.mathisen at hda.hydro.com>
> Date: Sun, 07 Dec 2008 11:44:18 +0100
> Sender: questions-bounces+oberman=es.net at lists.ntp.org
> 
> 
> Maarten Wiltink wrote:
> > "Unruh" <unruh-spam at physics.ubc.ca> wrote in message
> > news:S5y_k.2887$yK5.1576 at edtnps82...
> > [...]
> >>> What should I be doing to get 20 us? Buy all new computers with
> >>> gigabit Ethernet?
> >> I suspect buying better switches. And it looks to me like you should
> >> definitely NOT go to gigabit Ethernet if you want good timing.
> > 
> > Seeing as how gigabit Ethernet uses jumbo frames to reach decent
> > throughput, I suspect it's subject to higher jitter at the network
> > level. NTP frames are short, but if it's contending with a large
> > download, there will be a relatively large unpredictable delay on
> > the line.
> 
> I wrote a very similar msg, then I started to consider the math:
> 
> 100 Mbit uses 1.5 kB packets, while Gbit jumbo frames are typically 9 
> kB, right?
> 
> The thing is that as long as the jumbo frames are less than 10 times the 
> size of regular packets, the wire will be busy for a shorter time than 
> for a 1.5 K packet on 100 Mbit!
> 
> Hybrid interrupt/polled hardware & drivers are a much more likely 
> culprit: If the hardware waits a little bit before interrupting, hoping 
> to be able to bunch up several packets in a single task, then jitter 
> will suffer badly.

Before this gets too far off track, standards compliant Ethernet uses
frames of 1500 bytes, regardless of speed. This is true for 10M or 10G
Ethernet. 

Most modern cards have support for jumbo frames. These are, by
definition, non-standard, and are available only when specifically
configured. Unless you turn them on, frames are 1500 data bytes plus
framing of up to 22 bytes. When jumbos are enabled, the size currently
recommended by the US federal Joint Engineering Team is 9000 bytes to
allow the transmission of 8K of actual user data when allowing for
framing, VLAN and MPLS tags and such with a comfortable margin over
that. (Most modern systems page memory in 8K chunks, so this optimizes
performance.) 

So, if you have not set your card to use jumbo frames, you are not using
them. You are using 1500 byte MTUs. (On a Unix system, ifconfig(1) should
show the MTU.)

Interrupt coalescing is available on many modern 1G or faster Ethernet
interfaces. It is probably universal in any PCI-e 10G interface. It
requires driver support, but is available on most modern systems. It
will delay interrupts for a defined time so that only a single interrupt
is needed for multiple packets. For 10GE cards, a default of about 30-60
microseconds is normal, although this is usually controllable in around
1 microsecond units.

1G cards based on Intel chips may support coalescing in similar
increments, but the default delay is highly variable. This could explain
some of the jitter. But the jitter introduced should be fairly low, so I
doubt it can explain what is seen here.

Out network uses Intel based 1G interfaces and we do not see any
excessive jitter on FreeBSD. But the interrupt delay for coalescing is
driver controlled and other OSes my use a greater delay. In FreeBSD, the
delay defaults to 0 and may be adjusted with a sysctl. On other OSes,
you will need to read the documentation to see what, if any, delay is
being used.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman at es.net			Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751



More information about the questions mailing list