[Pool] 8-10k pps in Brazil

Maximiliano Valdez mvg at ier.unam.mx
Fri May 22 18:47:44 UTC 2015


Matt, sorry for the duplicity, I forgot to reply to the list

here it goes

--------------------
Hi Matt

I wonder if this behaviour is the result of some kind of load balancing
error on ntp.org DNS or something.

I have seen that kind of traffic a few times the last 6 months, I even had
a complain from NOC since they could not reach the peripheral router
because of NTP traffic. I had a couple of issues with firewall reaching
connection tracking limit, but that is easily corrected.

It would be great to know if this is actually a DoS (I think it is) or it
is just an error on the round-robin or similar at the DNS level that sends
way too much traffic to one host on the pool.

Regards !
Max

2015-05-22 13:28 GMT-05:00 Matt Wagner <mwaggy at gmail.com>:

2015-05-22 13:28 GMT-05:00 Matt Wagner <mwaggy at gmail.com>:

> Does anyone else here run an NTP server in Brazil? I'm wondering if you are
> seeing the same crazy load I am.
>
> For a long time I saw maybe 400 queries/second, but I got email last
> weekend that I had fallen out of the pool for being unreachable. Indeed, I
> couldn't even SSH in. It turns out that it's because my server (a t1.micro
> instance) was dying under the load, which is close to 10,000 queries per
> second right now. For giggles, I upsized to a larger instance and moved the
> IP to watch what was happening on a machine that could handle the load.
>
> Yes, I'm patched against the old monlist exploit.
>
> $ /usr/local/bin/ntpq -c sysstat
> uptime:                 77729
> sysstats reset:         77729
> packets received:       670434339
> current version:        10573419
> older version:          659857017
> bad length or format:   3276
> authentication failed:  7916
> declined:               3
> restricted:             126
> rate limited:           60293937
> KoD responses:          10096867
> processed for time:     636
>
> There are definitely some abusive clients, but it's not a crazy DoS from
> one IP or anything. Less than 10% of requests hit rate limits, and if I
> watch tcpdump or something, it's from a huge range of IPs. Only a handful
> of clients have made more than 50,000 requests (over the ~77000 second
> uptime), and none are way over that. Trying to profile random IPs from
> tcpdump, none seem to be behaving too wildly. It seems like I'm just
> serving a huge number of clients.
>
> My bandwidth is set at 100 Mbps, which it has been at for a while. The jump
> from a few hundred queries/second to 10,000 queries/second seems to have
> come out of nowhere.
>
> Is anyone else seeing this? I'm happy to keep soaking up some of the load,
> but I'm not eager to pay for 50GB of NTP traffic a day for too long.
> _______________________________________________
> pool mailing list
> pool at lists.ntp.org
> http://lists.ntp.org/listinfo/pool
>


More information about the pool mailing list