[Pool] My NTP rate limiting setup.

Andreas Krüger timekeeper at famsik.de
Tue Dec 17 14:26:33 UTC 2013


Hello, Marc and all,

here's my 2c on NTP rate limiting.


planning rate limiting
======================

My rate-limiting goal is not so much to protect my own server from
misbehaving clients.  If misbehaved clients _increase_ their knocking
at my door after I have shut it into their face, so be it.  Let them
knock.

What I am really concerned about is reflection attacks.  Someone rogue
sends NTP request parcels to my server with a forged sender address,
pointing to some victim's computer.  Without rate limiting, my sever
would flood the victim's computer with NTP answers.

Like Marc and Rob, I turn away from the rate-limiting facilities build
into ntpd.  My reason: ntpd rate-limiting keeps a list of only 600
clients.  That's not so bad for a start.

But now, I set up an IP-tables list which has slots for 30000 unique
clients.  When all is normal, an individual client drops out of that
list after roughly an hour of inactivity.  Feels better than 600.
A botnet would need 50000 slaves instead of 1000 to have a certain
percentage of their packets come through.  Should that ever happen, I
can increase that 30000 to a larger number much easier than I
could the 600.  Feels good.

If I understand correctly, an ntpd - client could legitimatly send a
burst of 8 requests every 64 seconds, if configured with "burst".  So
I decided to rate limit at 10 requests every 50 seconds.

For the record: I deliberately decided against using the "time to
live" of the incoming requests in my rate limiting.  Doing so would
make it a little harder for someone rogue to forge parcels that cause
my server to ignore the legitimate (victim's) request.  So far, so
good - but doing so would also make it easier for a rogue botnet to
launch a distributed reflection attack against a single victim's
computer.


Installing rate limiting
========================

I wanted to use iptables rate limiting via the "recent" module.

I wanted 30000 slots (unique clients tracked).

As I limit at 10 requests, I need at least that many per client.  Many
client seem to send out requests in bursts of 4.  So 12 seemed like a
reasonable number.

This boils down to the following modprobe configuration line:

options xt_recent ip_list_tot=30000 ip_pkt_list_tot=12

On my box, I added a file /etc/modprobe.d/xt_recent.conf with that
line.  That might also work on your box, consult "man modprobe.conf".

In passing: These numbers are compatible with all other xt_recent uses
on that box.

At this point, I checked xt_recent was not yet on the list of loaded
modules, as displayed by lsmod .

Next, I added a couple of IP-tables firewall rules:

iptables -A INPUT -p udp --dport 123 -m recent --name toomuchntp --set
iptables -A INPUT -p UDP --dport 123 -m recent --name toomuchntp \
  --rcheck --seconds 50 --hitcount 10 -j DROP


testing rate limiting - does it work?
=====================================

This automatically loads the xt_remote kernel module.  I checked via
lsmod.

After a cup of tea, the customary

    iptables -vnxL INPUT

gives some initial staticstics.

Does it work as intended?  For that, I go to a different Linux box and
issue, in more or less rapid succession, several

ntpdate -qv -p 1 myserver

Higher values for "-p" (parcels per burst) can also be used.


Doing reports
=============

Back on the ntp server machine, looking at

/proc/net/xt_recent/toomuchntp

yields some interesting per-client information.  The timestamps are in
Kernel jiffies from some (seemingly arbitrary?) point in the past.
There are 250 such jiffies per second.

I wrote a little Ruby script, which I'm happy to provide below, to run
some reports on that file.

Regards, Andreas



#!/usr/bin/env ruby
# -*- coding: utf-8 -*-

# Copyright 2013 by Andreas Krüger timekeeper at famsik.de.

# This is Open Source software that you may use under the Apache
# License 2.0 http://www.apache.org/licenses/LICENSE-2.0.html .

# Copy some file below /proc/net/xt_recent/ to
# toomuchntp.copy and run this script.

# The output contains several reports. Search for "***".

# This is a quick hack, originally intended for NTP rate limiting.
# Have fun.

# An array of arrays. One slot for each client found.
# data[i][0]: IP
# data[i][1]: packets per second (or some high bogus value)
# data[i][2]: number of packets on record (also data[i][3].length)
# data[i][3]: An array, that has the timestamps of the last packets received,
#             assumed to be in kernel jiffis of 4 ms each.
data = []

File.open 'toomuchntp.copy' do |f|
  f.each_line do |l|
    if l =~ 
/^src=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+ttl\:\s+\d+\s+last_seen\:\s+(\d+)\s+oldest_pkt\:\s+\d+\s+(((\d+)\,\s+)*\d+)\s*$/
      ip = $1
      times = $3.split(/\, /).map {|t| t.to_i}.sort
      packets = times.length
      delta_time = times[packets - 1] - times[0]
      packets_per_second_seen = \
        (0 < delta_time) ? (packets * 250.0 / delta_time) : \
        (1 < packets ? packets * 5000.0 : nil)
      data << [ip, packets_per_second_seen, packets, times]
    else
      raise "Cannot parse: #{l}"
    end
  end
end

# Sort descending by packets per seconds,
# or, if that's equal, by number of packets on record,
# or, if that's equal, by timestamp of oldest package.
data.sort! do |a, b|
  r = (a[1].nil? or b[1].nil?) ? 0 : b[1] <=> a[1]
  if 0 == r
    r = b[2] <=> a[2]
  end
  if 0 == r
    r = b[3][0] <=> a[3][0]
  end
  r
end

i = 0
$stdout.puts "***\nALL clients:\n\n(number, IP, packets_per_second (or some high
bogus value), "  +
  "number of packets known, timestamps in seconds relative to this client's
oldest packet)\n\n"
data.each do |d|
  $stdout.printf "%5d %15s %13.5f %2d %s\n", i += 1, d[0], d[1].nil? ? -1.0 :
d[1], \
    d[2], d[3].map{|t| (t - d[3][0])/250.0}.to_s
end

$stdout.puts "\n\n***\nClients that were done within 2.5 seconds, sending 2 to 8
requests:"
i = 0
data.each do |d|
  if d[2] <= 8 and 2 <= d[2] and d[3][-1] - d[3][0] <= 2.5 * 250
    $stdout.printf "%5d %15s %13.5f %2d %s\n", i += 1, d[0], d[1].nil? ? -1.0 :
d[1], \
      d[2], d[3].map{|t| (t - d[3][0])/250.0}.to_s
  end
end

$stdout.puts "\n\n***\nClients sending two requests within the same 8 µs:"
i = 0
data.each do |d|
  if 2 <= d[3].length
    rapid_fire = false
    (1 .. d[3].length - 1).each do |i|
      if d[3][i] - d[3][i-1] < 250 * 0.08
        rapid_fire = true
        break
      end
    end
    if rapid_fire
      $stdout.printf "%5d %15s %13.5f %2d %s\n", i += 1, d[0], d[1].nil? ? -1.0
: d[1], \
        d[2], d[3].map{|t| (t - d[3][0])/250.0}.to_s
    end
  end
end

# For each client, consider the youngest parcel on record.
# Find out what's the earliest and the latest of these.
# This tells us how far back our client record is complete.

earliest = data[0][3][-1]
latest = data[0][3][-1]

data.each do |d|
  e = d[3][-1]
  l = d[3][-1]
  if e < earliest
    earliest = e
  end
  if latest < l
    latest = l
  end
end

earliest = earliest / 250.0
latest = latest / 250.0

$stdout.printf "\n\nTable spans clients that last sent between %f and %f s.
Delta is %f s.\n", earliest, latest, latest - earliest



More information about the pool mailing list