Object storage rate limiting
Three questions:
If i were to reverse proxy requests to object storage on my server, will it count as a request from the client or my server?
I am using minio s3 gateway to connect to linode object storage, it makes many tcp requests i'm not familiar with - is there any way in linode to profile these requests to see how they are contributing toward my rate limits, and are tcp requests rate limited the same way http requests are?
Is there bandwidth limiting or any other rate limiting besides requests per second/bucket?
Thanks!
2 Replies
hphillips
Linode Staff
✓ Best Answer
Object Storage is rate limited to 750 S3 API requests per second per bucket at the endpoint. Each part of a multi-part upload for large files would count individually.
The S3 API requests are made using the HTTP protocol which in turn relies on TCP to provide reliable transport over IP. Each individual TCP segment does not count as part of the rate limit.
To gather statistics on HTTP Requests, you can use a tool such as Wireshark or another protocol analyzer to monitor the connections between your reverse proxy and the S3 API endpoint such as us-east-1.linodeobjects.com.
It does not matter if the request to Object Storage is a GET, POST or some other operation.
If your reverse proxy is making connections to Linode Object Storage on behalf of a client, it is the requests from the proxy to the Object Storage endpoint that are counted.
Requests in excess of the rate limit will fail with an HTTP 503 (Service Unavailable) Error. Presumably the client or proxy will implement some sort of retry mechanism.
There is no other bandwidth or rate limiting in place on the Object Storage endpoint.
Note that you may run up against a Linode's bandwidth rate limit, or if you are using methods from the Linode API to access the contents of your bucket, your requests would be subject to both rate limits.
