Thursday, February 28, 2008

Rsync and Wget

This tool helps you to copy files or directories between a local and a remote host.

rsync attempts to identify differences between source and destination files prior to initiating a transfer, and (assuming differences exist) tries only to copy the changes, rather than the entire file.
http://www.samba.org/rsync/

It woks like

On the same machine

rsync -a sourceDir/ destinationDir/
rsync care about the trailing slash, but only on the source argument










say dir1 and dir2 are two directories and file1 is the file that reside under dir1

command:-
rsync -a dir1 dir2 ==> will give the output as dir2/dir1/file1
command:-
rsync -a dir1/ dir2 ==> will give the output as dir2/file1

command:-
rsync -a --delete sourceDir/ destinationDir/ ==> will delete extra file from destinationDir which does not have presence in sourcedir



For Copying data by Rsync, it should be installed on both hosts.

Example:- How to copy data between localhost and remote host

You are sitting on localhost machine where you want to copy data from a remote machine

rsync -avz -e ssh userName@remoteHost:/remote/outgoingData/ /local/incomingData

here "/" after the outgoing data means that the localhost already have the directory and you just just want the data from the outgoingData directory.

If we remove the "/" then it means that you want to create the outgoingData directory first on
the remote machine then want to copy data from it. It should be like below:-

rsync -avz -e ssh userName@remoteHost:/fullPath/outgoingData/ /fullPath/incomingData/outgoingData



Mirroring the websites:-
you are giving the below command from server2.xyz.com (you want the backup to this server)
rsync -avz -e ssh userName@server1.xyz.com:/var/www/ /var/www/


rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html' (Mirror web site (using compression and encryption))


rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ (Synchronize current directory with remote one(

WGET
wget -c http://www.example.com/big.file [Continue downloading a partially downloaded file ]
wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/ [Download a set of files to the current directory ]
wget ftp://remote/file[1-9].iso/ [FTP supports globbing directly ]
wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head [Process output directly ]
echo 'wget url' | at 01:00 [Download url at 1AM to current dir]
wget --limit-rate=20k url [Do a low priority download (limit to 20KB/s in this case) ]
wget -nv --spider --force-html -i bookmarks.html [Check links in a file ]
wget --mirror http://www.example.com/ [Efficiently update a local copy of a site (handy from cron)]



copy files from remote server called server1 into /backup directory:
rsync -arvz -e ssh user@server1:/var/www/html /backup



a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive - means it copies directories and sub directories
v = verbose - means that it prints on the screen what is being copied



Use of "/" at the end of path:
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.
When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.

Friday, February 22, 2008

Redundant Array of Individual Disks (RAID)

Types of RAID


RAID 0 or Disk Striping


Number of Minimum Disks Required=2


RAID 0 provides the highest performance but no redundancy. Data in the logical drive is striped (distributed) across several physical drives.

RAID 1 or Disk Mirroring


Number of Minimum Disks Required=2

Capacity=N/2

RAID 1 mirrors the data stored in one hard drive to another. RAID 1 can only be performed with two hard drives. If there are more than two hard drives, RAID (0+1) will be performed automatically.

RAID 3 or Disk Striping with Dedicated Parity Disk
Minimum Disk Required=3

Capacity=N-1

Redundancy=Yes

RAID 3 performs Block Striping with Dedicated Parity. One drive member is dedicated to storing the parity data. When a drive member fails, the controller can recover/ regenerate the lost data of the failed drive from the dedicated parity drive.

Raid 5 or Striping with Interspersed Parity
Minimum Disk Required=3

Capacity=N-1

Redundancy=Yes

RAID 5 is similar to RAID 3 but the parity data is not stored in one dedicated hard drive. Parity information is interspersed across the drive array. In the event of a failure, the controller can recover/regenerate the lost data of the failed drive from the other surviving drives.