[Pool] 8-10k pps in Brazil
mwaggy at gmail.com
Fri May 22 18:28:05 UTC 2015
Does anyone else here run an NTP server in Brazil? I'm wondering if you are
seeing the same crazy load I am.
For a long time I saw maybe 400 queries/second, but I got email last
weekend that I had fallen out of the pool for being unreachable. Indeed, I
couldn't even SSH in. It turns out that it's because my server (a t1.micro
instance) was dying under the load, which is close to 10,000 queries per
second right now. For giggles, I upsized to a larger instance and moved the
IP to watch what was happening on a machine that could handle the load.
Yes, I'm patched against the old monlist exploit.
$ /usr/local/bin/ntpq -c sysstat
sysstats reset: 77729
packets received: 670434339
current version: 10573419
older version: 659857017
bad length or format: 3276
authentication failed: 7916
rate limited: 60293937
KoD responses: 10096867
processed for time: 636
There are definitely some abusive clients, but it's not a crazy DoS from
one IP or anything. Less than 10% of requests hit rate limits, and if I
watch tcpdump or something, it's from a huge range of IPs. Only a handful
of clients have made more than 50,000 requests (over the ~77000 second
uptime), and none are way over that. Trying to profile random IPs from
tcpdump, none seem to be behaving too wildly. It seems like I'm just
serving a huge number of clients.
My bandwidth is set at 100 Mbps, which it has been at for a while. The jump
from a few hundred queries/second to 10,000 queries/second seems to have
come out of nowhere.
Is anyone else seeing this? I'm happy to keep soaking up some of the load,
but I'm not eager to pay for 50GB of NTP traffic a day for too long.
More information about the pool