Recently I was moving data between an old server and new web server. I had some 250 GB of data and thought about just using Secure Copy (scp). When I started the process, I walked away
thinking “No Problem”, but after a few files, I got an error due to file permissions and I did not want to figure out what files had not been copied yet or where to try to restart.
So I used rsync to mirror the drive. I was able to set it up and due to my slow network speed, it still took several days, but I was certain that I had gotten all the data.

rsync -azuv -e ssh  user@originserver:public_html/* public_html/

The options I used were for :

This took some time to run, but I did not have to babysit the command and it duplicated the data from my old server to the new one.

I use find and grep quite a bit. I found that I was regularly searching for strings in file and was typing commands like:

find . -exec grep "somestring" {} \; --print

This find command will search from the current directory and for each file will run grep on it and if it finds a matching string, it will print the string and then the filename. So I took this and moved it into a script.

#!/bin/sh

if [ $# -lt 1 ]; then
  echo 1>&2 Usage: $0 ""
  exit 127
fi
icase=
search=

while [ $# -ge 1 ]; do
  case $1 in
     -i) icase=$1;;
      *) search=$1 ;;
  esac
  shift
done

find . -type f -print0 | xargs -0 grep $icase $search

This allows me to run a command such as

efind -i Super

This command will search all files starting from the current directory and recurse directories and the “-i” will tell it to ignore case.

Tags: scripting

There are times that I get on a Linux box and want to know about some of the resources available to me. There are tools that can accomplish this, but there are some files that contain basic settings about the resources available to the system. One of those file is:

/proc/cpuinfo

This file contains information about the Processors and you can do a simple:

more /proc/cpuinfo

to see all the details or you can grep for certain fields:

processor       -- This will show you how many processor entries there are.
cpu MHz  or MHz -- This will show you the speed of the processors.
cache size      -- This will show you the cache size available to the processor.
model           -- This will show you the type and model of the processor you are using.

There are quite a few more fields, but these are the ones that I check the most.

Tags: Linux, processor, Utilities

I was asked about system utilization on Linux systems. I found thesystatset of tools which consist of: sar, sadf, mpstat, iostat, nfsiostat, cifsiostat, pidstat and sa tools.

I will talk about the iostat tool in this article and follow up about some of the other tools.

On Ubuntu or debian systems you can install this toolset with:

sudo apt-get install sysstat

Then you will have access to the iostat command. The command output from executing with no options is below:

# iostat
Linux 2.6.32-33-generic (sumo)  09/01/2011      _i686_  (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.11    0.00    0.47    0.01    0.00   99.41

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.54         1.85         8.91    1611385    7740024

This displays the avg-cpu utilization and a device usage list with basic utilization.
Some options you can use are

One more way to use the iostat tool is to specify an interval and count to the command

iostat -d -x 10 5

This will display the output 5 more times with 10 seconds between each run.

Tags: Linux, system tools, utilization

If you ever need to schedule a job regularly on a Unix system, they have a great utility called cron. This will run a job at a regularly scheduled time.

Here is the format:

* * * * *  command to be executed
_ _ _ _ _
| | | | |
| | | | +--- day of week (0 = Sunday - 6 = Saturday)
| | | +----- month (1 - 12)
| | +------- day of month (1-31)
| +--------- hour(0-23)
+----------- minute (0-59)

example of a script that runs every day at 7:

* 7 * * * sendmeupdate

I regularly have to find where a file is located and usually use find:

find . | grep 

Where is replaced with the filename or partial name to match. So I finally decided to write a script to just run this and allow me to print the matching names and even potentially pass the -i flag to grep.

#!/bin/sh

if [ $# -lt 1 ]; then
  echo 1>&2 Usage: $0 ""
  exit 127
fi
icase=

while [ $# -ge 1 ]; do
   case $1 in
     -i)  icase=$1;;
      *)  search=$1 ;;
   esac
   shift
done

find .  | grep $icase $search

This is just a simple script which I have named ffind that searches starting from the current directory. This lets me search with the following commands:

ffind bak$

This will search for all files that end with “bak” .

ffind -i edr

This will search for all files with “edr” regardless of case

Tags: find, Linux, scripting, Shell, Utilities

So, after a Linux crash that caused my previous ddrescue to fail(after 350GB), I started to look into how to speed it up, because 30+ hours did not sound like a good idea to me.
I look into ways to speed it up and using a larger block size did the trick. I experimented with a few different sizes and figured out that using 2M gave me the best throughput. So instead of a couple of megs a second i was getting over 24 MB/s. So about a 4 1/2 hour recovery after that.

ddrescue -b 2M /dev/sda2 /media/newdisk/olddata /media/newdisk/logfile

The ddrescue process will attempt to recover any lost sectors and retrim the lost data. Luckily, I only lost about 3100k of data so the most of my data is safe.

You can attempt to rerun ddrescue with the -r <# of times to retry> and it will continue to attempt to read those failed sectors that were unrecoverable. There is no guarantee that you will get back all the data.

Once this data is recovered, you can (hopefully) mount this file as a disk under linux.

 mount /media/newdisk/olddata /mnt/newdisk 

Now remember to back up your data and you will not go through the pain that I did to recover this data.

Part 1 of article

Tags: dd, ddrescue, Linux

Sometimes you have a drive that your want to mount under Linux. The mount command can make the drive available to you. But you need to also create a directory to hold the mount point.

mkdir /mnt/drivepoint

Then you can mount the drive to that mount point by using the mount command and giving the device identifier, in this case it is /dev/sda1, and the mount point which we created as /mnt/drivepoint. One thing to note, is that the mount will use whatever directory you specify, so if there are files located in the directory you use as a mount point, they will no longer be visible.

mount /dev/sda1 /mnt/drivepoint

This must be run as root or with the sudo command.

Tags: Linux, OS, Utilities

Did you know? You can Set file timestamp using touch command. One way to do this is:
touch -c -t YYMMDDhhmm filename.txt

So every once in a while I want to see the progress of an upload or backup. So I turn to the watch command in Linux. The watch command gives the user the ability to re-run a command and see the updated output.

The syntax is ‘watch -n -d .

So for watching a file downloads progress you could run the following:

watch -n 30 ls -l filedownloading

This would show the ls output and you could watch the size of the file grow as it is dowload progress.