8MB picture takes 9 sec to download. How can I speed this up?
I am using a object bucket storage in us-southeast-1, and I am using a computer on the west coast. It takes me 8.95sec to download a 8.29MB picture file or 6.7sec to download 5.1MB picture file.
This is very slow for my work. I am wondering if there is any way to speed this up.
The process of downloading a file can decomposed into:
- reading the file from the remote system
- transferring the file over the from the remote system's network to the internet and finally to your local network
- writing the file to the local system
To get a baseline for how long it takes to download files of that size from a system local to the Atlanta data center, I created a bucket and Linode in the us-southeast-1 region. I then generated a files of the same size as you describe:
dd if=/dev/random of=5.1MB_file bs=5100000 count=1 iflag=dsync,fullblock
dd if=/dev/random of=8.29MB_file bs=8920000 count=1 iflag=dsync,fullblock
I used s3cmd to upload the files:
The 5.1MB_file took 0.447s and the 8.29MB_file took 2.098s to upload in total, but both took under 0.5s for the transfer to occur.
root@localhost:~# time s3cmd put 5.1MB_file s3://bucket/5.1MB_file upload: '5.1MB_file' -> 's3://bucket/5.1MB_file' [1 of 1] 5100000 of 5100000 100% in 0s 22.47 MB/s done real 0m0.447s user 0m0.146s sys 0m0.030s root@localhost:~# time s3cmd put 8.29MB_file s3://bucket/8.29MB_file upload: '8.29MB_file' -> 's3://bucket/8.29MB_file' [1 of 1] 8290000 of 8290000 100% in 0s 26.31 MB/s done real 0m2.098s user 0m0.153s sys 0m0.034s
The 5.1MB_file took 0.399s and the 8.29MB_file took 3.198s to download from a Linode in Atlanta, but both took under 0.5s for the transfer to occur. The rest is just connection overhead.
root@localhost:~# time s3cmd get s3://bucket/8.29MB_file download: 's3://bucket/8.29MB_file' -> './8.29MB_file' [1 of 1] 8920000 of 8920000 100% in 0s 51.53 MB/s done real 0m3.198s user 0m0.159s sys 0m0.034s root@localhost:~# time s3cmd get s3://bucket/5.1MB_file download: 's3://bucket/5.1MB_file' -> './5.1MB_file' [1 of 1] 5100000 of 5100000 100% in 0s 25.09 MB/s done real 0m0.399s user 0m0.133s sys 0m0.045s
The 5.1MB_file took 2.747s and the 8.29MB_file took 7.639s to download from a Linode in Fremont, the actual transfers took 2s and 3s respectively however.
root@localhost:~# time s3cmd get s3://bucket/8.29MB_file download: 's3://bucket/8.29MB_file' -> './8.19MB_file' [1 of 1] 8290000 of 8290000 100% in 3s 2.17 MB/s done real 0m7.639s user 0m0.225s sys 0m0.061s root@localhost:~# time s3cmd get s3://bucket/5.1MB_file download: 's3://bucket/5.1MB_file' -> './5.1MB_file' [1 of 1] 5100000 of 5100000 100% in 2s 2.14 MB/s done real 0m2.747s user 0m0.181s sys 0m0.074s
It looks like connection overhead is the biggest part of the time you are seeing. You may wish to use a tool such as CyberDuck to process your downloads in one session or in parallel.
Some other possible contributing factors
- Variability in Object Storage performance due to demand
- Internet Transit Problems: https://www.linode.com/docs/guides/diagnosing-network-issues-with-mtr/
- Downloading from the Cloud Manager can be much slower due to the possibility that it might need to relay the data from object storage for you
- IO overhead on your local system
- Network overhead on your local network
I hope this helps!