Do You Recommend RSYNC?

[Carlwill]

OK - I have a task at hand on my LAN and I was wondering if you guys felt Rsync was the best solution at this point. Please let know what I should do or what you think is the best course of action.

I have a server "fs1" which is a current Linux file server running Samba to all Windows XP clients on the LAN. The directory being shared on Samba is simply /share. Now this server is old and no longer under warranty so we purchased a nice new server and I configured RHEL4 on it using the entitlement from "fs1" to update it. Now the new server is called "fs-new". My question is how do you recommend I dump all the data from "fs1" to "fs-new" that is located under /share? Both servers are on the LAN and have SSH running on them if that means anything. I still need to configure SAMBA on "fs-new" however I thought I would worry about data migration 1st. I made an identical /share directory on the new "fs-new" so now its a simple matter of grabbing the data however I was told not to use scp.

When the data has been moved over and Samba is configured, I will simply change the hostname and IP after hours that way when everyone comes back in the next day, fs-new will now be fs1 w/o anyone noticing.

Thanks for any info and or suggestions.

[Spankin Partier]

Rsync is great to keep file systems in sync. As far as speed goes nothing can touch it...IF you have already copied most of the files over and are just looking to resync the files.

But seeing as you are transfering the files for the first time, TAR would be a much better option. For not only is it way more efficient at copying entire file systems, you can also add compression into the mix to reduce the network bottle neck. Here's a sample of the command I'd use:

tar -czf - /some/file | ssh host.name tar -xzf - -C /destination

This compresses the files and sends them via ssh to the destination computer where the destination computer uncompresses the files and places them onto the hard drive. I've used a similar command many times.

[Ned Slider]

Presumably the reason you were told not to use SCP is that you want the file permissions/ownerships left intact?

Anyhow, I'd use tar too as Spanky suggests as it has the facility to preserve file ownerships

[Carlwill]

Thanks all. The directory I want to copy from one machine to another is roughly 300GB in size. I think I will run this command over the weekend and let it go...

I don't know how long it will take to tar everything up and then transfer the tar ball and then un tar the file but I would assume a weekend would be enough time, no?

Thanks so much!

[Spankin Partier]

It should be. Here try this just for kicks:

time tar -czf - /some/file | ssh host.name tar -xzf - -C /destination

It'll report how long it took to do the command in the end.

[Carlwill]

Any idea what I did wrong with my command?

[root@lt2fs1 ctia]# time tar -cvpjf - cvs/ | ssh cwilliams@10.1.1.204 tar -xcf - -C /home/cwilliams/Desktop
cvs/
cvs/TENA/
cvs/TENA/Use Case Template.ZIP
cvs/TENA/Inst. Dispaly Dist Use Case.xls
cvs/master/
cvs/master/CTIADev/
cvs/master/CTIADeliverables/
cvs/master/CTIADeliverables/ReUseAnalysis/
cvs/master/dia-ctia/
cvs/master/Instrumentation/
cvs/master/Instrumentation/TESS-OTB/
cvs/master/Instrumentation/TESS-OTB/Component Contract/
cvs/master/Instrumentation/TESS-OTB/Component Contract/Component Contract TESS - OTB.doc,v
cvs/master/Instrumentation/TESS-OTB/Package_Class Template/
cvs/master/Instrumentation/TESS-OTB/Package_Class Template/Package_Class Template - Read Me.txt,v
cvs/master/Instrumentation/TESS-OTB/Design/
cvs/master/Instrumentation/TESS-OTB/Design/Component Design Document/
cvs/master/Instrumentation/TESS-OTB/Design/Component Design Document/TESSOTBUSeCase.doc,v
cwilliams@10.1.1.204's password: 
tar: You may not specify more than one `-Acdtrux' option
Try `tar --help' or `tar --usage' for more information.

real    0m2.703s
user    0m0.021s
sys     0m0.009s

[Spankin Partier]

Here's a quick run down on the theory of the command I gave you. The first tar statement creates the archive to send over the network, and the second one extracts it on the fly. The ssh command in the middle acts as the 'media' that the two tar commands are using (rather then an actual tape or hard drive).

That being said, the second tar only needs to extract data from an 'existing tar file'. So the small c is not needed as that's for creating tar files. This will elimiate the error that was displayed. But after fixing that you'll still run into a couple of other issues.

The next problem you'd have would be with data compression. You are compressing the data prior to sending it over the network but you are not telling tar to decompress it before writing the data to disk. Seeing as you wish to use bzip2, you'll need to add j to extracting tar. I don't know if you'll notice any speed increases using bzip2 vers bzip. It'll depend on both system processing speeds as bzip2 requires more CPU processing to function.

Another problem you'll run into is with the p command. It should only be used while extracting files (the -x command). Hence it should be on the second tar.

The next issue would be the verbose command. Due to this non-standard use of tar, a strange 'quirk' appears if you use verbose on the first tar. You output gets doubled up. To fix this, you should only use verbose on the second tar.

After taking all of this into account, here's the new command you should try:

time tar -cjf - cvs/ | ssh cwilliams@10.1.1.204 tar -xpjvf - -C /home/cwilliams/Desktop

I hope this helped out more then me just simply writing the correct command. Good luck!

[Carlwill]

Thanks for the break down and that made perfect sense. I am now testing everything between 2 servers on the LAN.

I am trying to replicate home directories from 10.1.1.24 > 10.1.1.25.

When I ran the command, It yelled at me because I would assume during the first tar command, after the "f", it wants the name or tar file you're created in the "c" tag and then a space and the path of the file you're generating the tar from.

Here was my error:

[root@lt2fs1 /]# tar -cjf home/ | ssh root@10.1.1.25 tar -xpjvf - - C /
tar: Cowardly refusing to create an empty archive
Try `tar --help' for more information.
root@10.1.1.25's password: 

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Invalid argument
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Child returned status 2
tar: -: Not found in archive
tar: C: Not found in archive
tar: /: Not found in archive
tar: Error exit delayed from previous errors

I then ran the following command with a basic generic name for the tar:

[root@lt2fs1 /]# tar -cjf home.tar.bz2 home/ | ssh root@10.1.1.25 tar -xpjvf - - C /

From that command it sits there forever and does not time out or do anything...

Any thoughts on what I am doing wrong? I notice that in server 10.1.1.24 where I issued the command from, I have a home.tar.bz2 file under /. I don't know why SSH did not send it to 10.1.1.25 and extract it there unless I just killed the process prematurely.

[Spankin Partier]

Well the thing is you're not really creating a tar file that needs a file name. This is similar to when you use tar to backup a system to a tape drive. You are just sending the raw data to a device (or in our case piping the raw data to ssh). I think the piping is accomplished by using the second minus (-) in the command. I'm not sure but I'm pretty sure that represents some pseudo device that allows for piping. You'll notice it's also used in the second tar as well. In that case it would telling tar that the data would be coming from a piped command.

I think the reason you were getting stuck with your second try would be due to one of two possibilities:

1. You trying to create a tar file called home.tar.bz2 in your home directory while backing up everything in the home directory. Since tar would be writing to the home directory it would include it's own backup file in the backup. This will lead to a very interesting tar file...that is until the drive is full (I've been known to do that once or twice! ).
   
2. It just simply got messed up seeing as there would be no need for piping to ssh since there was no - shown after the f command.

#2 is most likely what happened since it appears you were in the root directory when you typed the command.

[Carlwill]

Guys - can automate this RSYNC over SSH to run nightly? I have the following cron job set on the new server to run nightly at 2 am after my backups run. Is there any way I can use the following command in cron w/o it needing a password? How do you guys get by this because when I run this command, I am always required to enter in a password.

[Ned Slider]

Create a public/private key pair to automatically authenticate your SSH connection:

http://forums.pcper.com/showthread.php?t=433216

See #4

[Spankin Partier]

Why that's exactly what I did recently. I was running the backups manually using rsync (since they were only incremental backups over the internet) via a small script since I needed to type in the server password. But after I got comfortable with that, I looked up Ned's thread to finish everything off by key pairing the accounts and then adding a cron.

[Carlwill]

I want to make sure I am correct here guys. Please let me know if I am wrong.

If I run the command :

rsync -avz --delete -e ssh cwilliams@server2:/share/ctia /share/

This would delete any files only server1 that are not present on server1, correct?

[Ned Slider]

That will sync server 2 with server 1. Using --delete will remove any files present on server2 (from previous backups, for example) that are now no longer present on server1.