Monday, April 9, 2012

Linux- copy methods

Linux- copy methods

Download a file

Download a file whose URL contains a “&” (note quotes)
Draw an entire site (recursive download)
Resume a download
Download files only an extension (here,. Jpg)
Limit download speed
wget – limit-rate = 30k
Download via ftp (With authentication)
wget-r l4 ftp://username:password @ /

If there are a list of pages, files, and so on that I want to grab for the script, I can list one URL-formatted item per line within a file. For example, if the file was ~/bin/getme, I would use:

wget -i ~/bin/data/getme

To download a copy of a complete web site, up to five levels deep, you use the -r option (for recursive):

wget -r -o logfile

Instead of having the progress messages displayed on the standard output, you can save it to a log file with the -o option:


Copy the file "foobar.txt" from a remote host to the local host
$ scp /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host

Copy the directory "foo" from the local host to a remote host's directory "bar"

Copy the file "foobar.txt" from remote host "" to remote host ""

$ scp \

Copying the files "foo.txt" and "bar.txt" from the local host to your home directory on the remote host

$ scp foo.txt bar.txt

Copy multiple files from the remote host to your current directory on the local host

$ scp\{foo.txt,bar.txt\}  .


$ rsync options source destination

Source and destination could be either local or remote. In case of remote, specify the login name, remote server name and location.

Synchronize Two Directories in a Local Server

$ rsync -zvr /var/dir1/ /root/dir2
-z  :enable compression
-v :verbose
-r    : recursive
-a : preserve time stamp
-d : sync the directory tree structure

Preserve timestamps during Sync using rsync -a

rsync -azv /var/dir1/ /root/dir2

rsync -arzv /var/dir1/ /root/dir2       (this will do recursively)
a = archive - means it preserves permissions (owners, groups), times, symbolic links, and devices.
r = recursive - means it copies directories and sub directories
v = verbose - means that it prints on the screen what is being copied 

Synchronize Only One File

To copy only one file, specify the file name to rsync command
rsync -v /var/singlefile /root/myfile/

Synchronize only the Directory Tree Structure (not the files)

rsync -v -d abc@ .

Use of "/" at the end of path:
When using "/" at the end of source, rsync will copy the content of the last folder.
When not using "/" at the end of source, rsync will copy the last folder and the content of the folder.

When using "/" at the end of destination, rsync will paste the data inside the last folder.
When not using "/" at the end of destination, rsync will create a folder with the last destination folder name and paste the data inside that folder.

Synchronize with the Directory Name
What if you wanted to not just copy all of the files under /myDir/, but also the /myDir/ directory name. Just leave off the last "/" after the "/myDir" directory name. This will tell rsync to copy the directory name and all files under that directory. On the remote server you will then see the directory structure /backups/myDir/.
rsync -avz /myDir user@remotemachine:/backups/

Incremental Backups using rsync

The following example will make an incremental backup of the directory /data/working/ and put a copy of any file that changes into a dated directory in /BACKUP/ . This can be used to keep a daily backup tree of any changed files and not have to overwrite the previous days files. Note that this method does need to copy the entire file if it changes as the new files are made in the directory named under current day.
rsync --backup --backup-dir=`date +%Y.%m.%d` -a /data/working/ /BACKUP/

lftp -c "set ftp:list-options -a;
open ftp://$USER:$PASS@$HOST;
lcd $LCD;
cd $RCD; 
mirror --reverse \
        --delete \
        --verbose \
        --exclude-glob a-dir-to-exclude/ \
        --exclude-glob a-file-to-exclude \
        --exclude-glob a-file-group-to-exclude* \
        --exclude-glob other-files-to-exclude"

Upload a file

lftp -e 'put /local/path/yourfile.txt; bye' -u user,password 

Download a file

lftp -e 'set net:timeout 10; get yourfile.txt; bye' -u user,password 

Mirroring / Recursive upload

When using the mirror command, LFTP is recursive by default.
Mirror everything from the /local/path to the root of the remote FTP site, including all subdirectories and their files. The -R switch means “reverse mirror” which means “put” [upload]. Remote path is simply /, the root.

lftp -e 'mirror -R /local/path/ /' -u user,password

Recursive Download

To mirror remote to local, just omit the -R param and swap remote path with local. Be careful with this. Don’t overwrite your local changes.

lftp -e 'mirror / /local/path/' -u user,password  

Special Characters in Username or Password

Special characters in your username or password must be escaped with backslashes:

lftp -e 'put /pathToyourfile.txt; bye' -u user,password\!\!

1 comment:

  1. This comment has been removed by a blog administrator.