DNS traffic query limits?
So I am gathering information on a large number of domains for a project I'm working. I need to run DNS queries to grab the A record for each domain (millions of them).
Is there a recommended timing or practice in order to prevent throttling or disabling of my network traffic? Something like, "X amount of queries per period of time?"
I would like to do this without breaking any rules and follow best practices. I know this kind of stuff might look malicious if I do it in the wrong way, so was hoping to avoid that if possible.
My understanding of the limits on outgoing requests is that they are arbitrarily set by the registrar. Doing some googling I found this post at serverfault that talks about how Google Public DNS handles this sort of stuff:
To protect other DNS nameservers against DoS attacks that could be launched from our resolver servers, Google Public DNS enforces per-nameserver QPS limits on outgoing requests from each serving cluster.
As far as Linode is concerned, I am not aware of any specific limits on the number of DNS queries you can perform, but I would probably recommend you to start low (maybe something closer to 10 queries a minute) and then work your way up slowly from there in order to ensure you are not hitting any limits. A more conservative rate of querying should help mitigate any negative repercussions on your IP (like getting throttled).