Block storage cause high loads with rsync
I'm trying to copy files from one block volume to another. They are both mounted on the same machine.
I'm using rsync to do this:
sudo rsync -a /volume1/* /volume2/
uptime looks like this:
$ uptime
08:59:31 up 469 days, 12:21, 5 users, load average: 1.94, 1.62, 1.37
iostats looks like this:
avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 4.29 48.23 0.00 46.97
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
loop0 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
sda 1.00 0.00 26.00 0 52
sdc 166.50 1476.00 0.00 2952 0
sdd 0.00 0.00 0.00 0 0
dm-0 0.00 0.00 0.00 0 0
sde 0.00 0.00 0.00 0 0
sdf 0.00 0.00 0.00 0 0
dm-1 0.00 0.00 0.00 0 0
With load of 1.5 machine is still responsive, but at 2 it starts to struggle.
But sometimes load gets out of control and makes everything using this machine goes down with it.
I would like to run a rsync cron job without the fear of the server going down while I'm sleeping peacefully.
Any ideas how I can prevent this from happening.
1 Reply
rgerke
Linode Staff
Hello!
The first thing that pops into mind is that you have both volumes attached to the same Linode. This might be causing some resource bottlenecks and a workaround for that might be to mount one of these volumes to a different Linode (in the same data center) while doing the rsync; you can always reattach that volume when you need to.
The reason I mention that is because typically your Block Storage will be slower than your Linode's local storage - and that can be accounting for the issues you're running into. Separating them onto two different Linodes for the rsync might help you out in this situation.
I also wanted to give you a list of commands you can run to check the performance even deeper:
DD test - 4M write performance
dd if=/dev/zero of=test_file bs=4M count=4000
DD test - 4K write performance
dd if=/dev/zero of=test_file bs=4k count=4000000
Read performance
dd if=test_file of=/dev/null bs=4M
FIO test for many small files
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite --ramp_time=4
FIO test for one large file
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4M --iodepth=64 --size=4G --readwrite=write --ramp_time=4