Best method of linode <--> linode file transfer?

if you need to move things right now, you can use tar and netcat:

server1$ tar cf - pathtofiles | nc -l -p 12345

server2$ nc server1 12345 | tar xf -

If you want a more permanent solution, set up rsyncd and tell it to listen only on the private IPs. rsyncd doesn't do encryption, and you get the bonus of changes-only transmission if you're routinely moving the same files around.

Even faster, use ssh:

For the first copy, use tar, it's fastest:

$ tar -cpf - -C /path/to/files . | ssh user@otherlinode "tar -xvf - -C /path/to/files"

For incremental updates, use rsync over ssh, no need for rsyncd:

$ rsync -avz /path/to/files/* user@otherlinode:/path/to/files/

Doh.

That'll teach me to read the whole post. You specifically didn't want ssh…

For the best raw file transfer speeds, I usually fall back to good 'ol FTP. lftp is a great command-line FTP client that's fully scriptable, and available in most every distro I've ever used.

If these files are largely the same each time, with say 20% of the file contents changing each sync, then you really need to setup and rsyncd on the source host. This will allow rsync to do binary deltas - ie if you have a 1GB file where only 10MB of the file has changed, it will only transfer 10MB.

HTH,

Justin

We did benchmarks about the "double encryption" stuff at my old job… and we discovered it really doesn't cause a slowdown at all.

I wouldn't fix it if it ain't broke.

@kbrantley:

What would be the best way of moving data from linode to linode here? Namely largeish files, anywhere from 100MB to 1GB.

I have heard system administrators at my office rave about some data transfer program named SneakerNet. It seems they use SneakerNet all the time and it works well.

James

@zunzun:

I have heard system administrators at my office rave about some data transfer program named SneakerNet. It seems they use SneakerNet all the time and it works well.

Sneakernet at the Newark datacenter has huge latency - you have to wait while caker puts on his sneakers and drives up there.

@Jay:

We did benchmarks about the "double encryption" stuff at my old job… and we discovered it really doesn't cause a slowdown at all.

I wouldn't fix it if it ain't broke.

That and how old it gets moving data with long and nasty ssh commands, that's the real problem here.

And sneakernet from the midwest to Atlanta is amazingly poor latency :)

I can hate on NFS as much as anyone, but for reducing typing, it's the way to go :-). If you don't mind having to move files on the destination, rsyncd isn't a bad solution.

That said, mostly I just rsync via ssh. Mostly I do NOT use the -z option; on a fast net, it doesn't buy much, and a lot of big files are already compressed.

@zunzun:

I have heard system administrators at my office rave about some data transfer program named SneakerNet. It seems they use SneakerNet all the time and it works well.

Never underestimate the bandwidth of a trunk full of disks (to update an old joke).

Latency may suck, but bandwidth is massive!