Bash

Which files are touched by bash

echo exit | strace bash -li |& grep '^open'

getting history of running process

echo -e "attach 23450\ncall write_history(\"/tmp/blah.txt\")\n" > cmd.txt ; gdb -batch -x cmd.txt

useful links

 http://mywiki.wooledge.org/BashFAQ/

moving around on bash prompt

C-a     Go to beginning
C-e     Go to end
C-f     Forward one character
C-b     Backward one character
M-f     Forward one word
M-b     Backward one word
C-k     Kill to end of line.
C-d     Kill word forward
C-w     Kill word backward
C-y     Yank word

bash Shell

For using multiple filenames in one command in Unix commandline use the following command:

for i in <equal part of all filenames>*;do <command> $i;done

$i is the variable set by the loop.

Example:

for i in sync_m4606_danube_sg_org_*;do /usr/atria/etc/mkorder -data $i -fship blrsxa16.blr.infineon.com;done

More examples:

for i  in `ct lsvob |grep systemq | cut -d" " -f4`;do /usr/atria/etc/utils/albd_list -s $i;done

All Files starting with sync_m4606_danube_sg_org_ are used to perform a mkorder -data to forward these packets to Bangalore server.

for i in `cut -f2 -d" " viewlist.del`;do rm -r /home/memhub.view/$i;done

Use viewtags of column 2 of the file viewlist.del to remove the viewstorage.

Alternative

Use xargs for simply passing over results to a simple command:

ps- elf | grep <string> |tr -s " " |cut f<row nr.> -d" "|xargs <command>

tr - text replace with -s for substitute

e.g.:

ps -elf | grep CCMS | tr -s " " | cut -f5 -d" " |xargs kill

loop number

i=0; while [ $i -lt 10 ]; do echo $i; i=`expr $i + 1`; done

interactive shell

echo $-                If shell is interactive variable will contain i.

explanation

operand		3, 3 + 6 = 9 '+' is the operator and '3' and '6' are the operands.
operator		+, 3 + 6 = 9 '+' is the operator and '3' and '6' are the operands.
primary		That which stands first in order, rank, or importance
Arithmetic		most elementary branch of mathematics,
Command control	Testing exit status of a command in order to determine whether a portion of the program should be executed.
Conditional branch	Logical point in the program when a condition determines what happens next.
Logic flow		The overall design of the program. Determines logical sequence of tasks so that the result is successful and controlled.
Loop			Part of the program that is performed zero or more times.
User input		Information provided by an external source while the program is running, can be stored and recalled when needed.

debug

set -x			# activate debugging from here
w
set +x			# stop debugging from here

debug login

Find out what is blocking login.

bash --login --verbose

variables

Character	Definition
$*	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the  value of each parameter separated by the first character of the IFS special variable.
$@	Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word.
$#	Expands to the number of positional parameters in decimal.
$?	Expands to the exit status of the most recently executed foreground pipeline.
$-	A hyphen expands to the current option flags as specified upon invocation, by the set built-in command, or those set by the shell itself (such as the -i).
$$	Expands to the process ID of the shell.
$!	Expands to the process ID of the most recently executed background (asynchronous) command.
$0	Expands to the name of the shell or shell script.
$_	The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. It is also set to the full pathname of each command executed and placed in the environment exported to that command. When checking mail, this parameter holds the name of the mail file.

quoting

Single quotes

Single quotes () are used to preserve the literal value of each character enclosed within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

Double quotes

Using double quotes the literal value of all characters enclosed is preserved, except for the dollar sign, the backticks (backward single quotes, ``) and the backslash.

The dollar sign and the backticks retain their special meaning within the double quotes.

The backslash retains its meaning only when followed by dollar, backtick, double quote, backslash or newline. Within double quotes, the backslashes are removed from the input stream when followed by one of these characters. Backslashes preceding characters that don't have a special meaning are left unmodified for processing by the shell interpreter.

A double quote may be quoted within double quotes by preceding it with a backslash.

eight kinds of expansion performed

1 Brace expansion
echo {01..3}{a..c}
1a 1b 1c 2a 2b 2c 3a 3b 3c

2 Tilde expansion

If a word begins with an unquoted tilde character ("~"), all of the characters up to the first unquoted slash (or all characters, if there is no   unquoted slash) are considered a tilde-prefix. If none of the characters in the tilde-prefix are quoted, the characters in the tilde-prefix following the tilde are treated as a possible login name. If this login name is the null string, the tilde is replaced with the value of the HOME shell variable. 
If HOME is unset, the home directory of the user executing the shell is substituted instead. Otherwise, the tilde-prefix is replaced with the home  directory associated with the specified login name.
If the tilde-prefix is "~+", the value of the shell variable PWD replaces the tilde-prefix. If the tilde-prefix is "~-", the value of the shell variable OLDPWD, if it is set, is substituted.
If the characters following the tilde in the tilde-prefix consist of a number N, optionally prefixed by a "+" or a "-", the tilde-prefix is replaced  with the corresponding element from the directory stack, as it would be displayed by the dirs built-in invoked with the characters following tilde in  the tilde-prefix as an argument. If the tilde-prefix, without the tilde, consists of a number without a leading "+" or "-", "+" is assumed.
If the login name is invalid, or the tilde expansion fails, the word is left unchanged.
Ech variable assignment is checked for unquoted tilde-prefixes immediately following a ":" or "=". In these cases, tilde expansion is also performed.  Consequently, one may use file names with tildes in assignments to PATH, MAILPATH, and CDPATH, and the shell assigns the expanded value.

3 Shell parameter and variable expansion

echo ${!N*}
NNTPPORT NNTPSERVER NPX_PLUGIN_PATH

The following construct allows for creation of the named variable if it does not yet exist:
${VAR:=value}

4. Command substitution

Command substitution allows the output of a command to replace the command itself. Command substitution occurs when a command is enclosed like this:
$(command) or like this using backticks:`command`
Bash performs the expansion by executing COMMAND and replacing the command substitution with the standard output of the command, with any trailing   newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
When the old-style backquoted form of substitution is used, backslash retains its literal meaning except when followed by "$", "`", or "\". The first  backticks not preceded by a backslash terminates the command substitution. When using the "$(COMMAND)" form, all characters between the parentheses make  up the command; none are treated specially.
Command substitutions may be nested. To nest when using the backquoted form, escape the inner backticks with backslashes.
If the substitution appears within double quotes, word splitting and file name expansion are not performed on the results.

5. Arithmetic expansion

$(( EXPRESSION ))
$[ EXPRESSION ]

6. Process substitution

Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. It takes the form of
<(LIST)
or
>(LIST)
The process LIST is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the   current command as the result of the expansion. If the ">(LIST)" form is used, writing to the file will provide input for LIST. If the "<(LIST)" form is  used, the file passed as an argument should be read to obtain the output of LIST. Note that no space may appear between the < or > signs and the left  parenthesis, otherwise the construct would be interpreted as a redirection.

7. Word splitting

The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word  splitting.
The shell treats each character of $IFS as a delimiter, and splits the results of the other expansions into words on these characters. If IFS is unset,  or its value is exactly "'<space><tab><newline>'",

8. File name expansion 

File name expansion Bash scans each word for the characters "*", "?", and "[". If one of these characters appears, then the word is regarded as a  PATTERN, and replaced with an alphabetically sorted list of file names matching the pattern.

regular expression metacharacters

Operator	Effect
.		Matches any single character.
?		The preceding item is optional and will be matched, at most, once.
*		The preceding item will be matched zero or more times.
+		The preceding item will be matched one or more times.
{N}		The preceding item is matched exactly N times.
{N,}		The preceding item is matched N or more times.
{N,M}		The preceding item is matched at least N times, but not more than M times.
-		represents the range if it's not first or last in a list or the ending point of a range in a list.
^		Matches the empty string at the beginning of a line; also represents the characters not in the range of a list.
$		Matches the empty string at the end of a line.
\b		Matches the empty string at the edge of a word.
\B		Matches the empty string provided it's not at the edge of a word.
\<		Match the empty string at the beginning of word.
\>		Match the empty string at the end of word.

Two regular expressions may be concatenated; the resulting regular expression matches any string formed by concatenating two substrings that respectively match the concatenated subexpressions.
Two regular expressions may be joined by the infix operator "|"; the resulting regular expression matches any string matching either subexpression.
Repetition takes precedence over concatenation, which in turn takes precedence over alternation. A whole subexpression may be enclosed in parentheses to override these precedence rules.
Parentheses -- ( ) -- enclose groups of REs. They are useful with the following "|" operator and in substring extraction using expr.

The postfix question mark (?) operator makes the preceding regexp optional (i.e, it can occur zero or one times). This shorthand is equivalent to a common use of summing. These two regexps are equivalent:

a(foo)?
afoo|a

Basic versus extended regular expressions
In basic regular expressions the metacharacters "?", "+", "{", "|", "(", and ")" lose their special meaning; instead use the backslashed versions "\?", "\+", "\{", "\|", "\(", and "\)".

POSIX Character Classes. [:class:]

This is an alternate method of specifying a range of characters to match.
#
[:alnum:] matches alphabetic or numeric characters. This is equivalent to [A-Za-z0-9].
[:alpha:] matches alphabetic characters. This is equivalent to [A-Za-z].
[:blank:] matches a space or a tab.
[:cntrl:] matches control characters.
[:digit:] matches (decimal) digits. This is equivalent to [0-9].
[:graph:] (graphic printable characters). Matches characters in the range of ASCII 33 - 126. This is the same as [:print:], below, but excluding the space character.
[:lower:] matches lowercase alphabetic characters. This is equivalent to [a-z].
[:print:] (printable characters). Matches characters in the range of ASCII 32 - 126. This is the same as [:graph:], above, but adding the space character.
[:space:] matches whitespace characters (space and horizontal tab).
[:upper:] matches uppercase alphabetic characters. This is equivalent to [A-Z].
[:xdigit:] matches hexadecimal digits. This is equivalent to [0-9A-Fa-f].

Grep multiple words. 21 and 23.

grep --color=always -vE '10\.[0-9]{1,3}\.(21|23)\.[0-9]{1,3}'

Character classes

A bracket expression is a list of characters enclosed by "[" and "]". It matches any single character in that list; if the first character of the list is the caret, "^", then it matches any character NOT in the list. For example, the regular expression "[0123456789]" matches any single digit.
Within a bracket expression, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, inclusive, using the locale's collating sequence and character set. For example, in the default C locale, "[a-d]" is equivalent to "[abcd]". Many locales sort characters in dictionary order, and in these locales "[a-d]" is typically not equivalent to "[abcd]"; it might be equivalent to "[aBbCcDd]", for example. To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value "C".

command examples

Does you account exist locally?

if grep $USER /etc/passwd;then echo "your user account is managed locally"; fi

Matching lines starting with a given pattern and ending in a second pattern are showed like this:

sandy ~> sed -n '/^This.*errors.$/p' example
This is a line not containing any errors.

Are you root?

if [ "$(whoami)" != 'root' ]; then
        echo "You have no permission to run $0 as non-root user."
        exit 1;
fi

With Bash, you can shorten this type of construct. The compact equivalent of the above test is as follows:

[ "$(whoami)" != 'root' ] && ( echo you are using a non-privileged account; exit 1 )
Similar to the "&&" expression which indicates what to do if the test proves true, "||" specifies what to do if the test is false.

real programmer...

test "$(whoami)" != 'root' && (echo you are using a non-privileged account; exit 1)

Scriptname.

#!/bin/bash
basename=${0##*/}
echo $basename

type of input.

#!/bin/bash
case $1 in
  *[!a-zA-Z]*) echo -e "\nContains only numbers.\n";;
  *) echo -e "\nIs not only numbers.\n";;
esac

1. repeat a number of time.

for (( i = 1 ; i <= 10 ; i++ ))
do
  echo "i is ${i}"
done

Character classes

Character classes can be specified within the square braces, using the syntax [:CLASS:], where CLASS is defined in the POSIX standard and has one of the values
"alnum", "alpha", "ascii", "blank", "cntrl", "digit", "graph", "lower", "print", "punct", "space", "upper", "word" or "xdigit".
ls -ld [[:digit:]]*

sed commands

The sed program can perform text pattern substitutions and deletions using regular expressions, like the ones used with the grep command; see Section 4.2.
The editing commands are similar to the ones used in the vi editor:
Command	Result
a\	Append text below current line.
c\	Change text in the current line with new text.
d	Delete text.
i\	Insert text above current line.
p	Print text.
r	Read a file.
s	Search and replace text.
w	Write to a file.

awk

The variables $1, $2, $3, ..., $N hold the values of the first, second, third until the last field of an input line. The variable $0 (zero) holds the value of the entire line. This is depicted in the image below,

\a	Bell character
\n	Newline character
\t	Tab

Quotes, dollar signs and other meta-characters should be escaped with a backslash.

2008-01-27 23:38$ ls -l | awk 'BEGIN { print "You are a loser" }{ print $8 "\t is \t" $5 " bytes" } END { print "But what do you care?" }'
You are a loser
12341    is     0 bytes
12342    is     0 bytes
12343    is     0 bytes
But what do you care?

BEGIN { FS=":" }
{ print $1 "\t" $5 }
$ awk -f printnames.awk /etc/passwd

awk 'BEGIN { OFS=";" ; ORS="\n-->\n" } { print $1,$2}' test
record1;data1
-->
record2;data2

cat processed.awk
BEGIN { OFS="-" ; ORS="\n--> done\n" }
{ print "Record number " NR ":\t" $1,$2 }
END { print "Number of records processed: " NR } 

awk -f processed.awk test
Record number 1:        record1-data1
--> done
Record number 2:        record2-data2
--> done
Number of records processed: 2
--> done

Count the size of file in current directory.

find . -maxdepth 1 -type f -ls | awk 'BEGIN { print "Hello\n" } { total=total + $7 } { print "counting " total } END { print "---------------------------------\nTotal revenue: " total }'

Print last word of sentance.

awk '{ print $NF }'

printf

define output format.

if

if TEST-COMMANDS; then CONSEQUENT-COMMANDS; fi The TEST-COMMAND list is executed, and if its return status is zero, the CONSEQUENT-COMMANDS list is executed. The return status is the exit status of the last command executed, or zero if no condition tested true.

Primary expressions used with if.

The table below contains an overview of the so-called "primaries" that make up the TEST-COMMAND command or list of commands. These primaries are put between square brackets to indicate the test of a conditional expression.

[ -a FILE ]		True if FILE exists.
[ -b FILE ]		True if FILE exists and is a block-special file.
[ -c FILE ]		True if FILE exists and is a character-special file.
[ -d FILE ]		True if FILE exists and is a directory.
[ -e FILE ]		True if FILE exists.
[ -f FILE ]		True if FILE exists and is a regular file.
[ -g FILE ]		True if FILE exists and its SGID bit is set.
[ -h FILE ]		True if FILE exists and is a symbolic link.
[ -k FILE ]		True if FILE exists and its sticky bit is set.
[ -o optname ]         True if shell option optname is enabled.
[ -p FILE ]		True if FILE exists and is a named pipe (FIFO).
[ -r FILE ]		True if FILE exists and is readable.
[ -s FILE ]		True if FILE exists and has a size greater than zero.
[ -t FD ]		True if file descriptor FD is open and refers to a terminal.
[ -u FILE ]		True if FILE exists and its SUID (set user ID) bit is set.
[ -w FILE ]		True if FILE True if FILE exists and is writable.
[ -x FILE ]		True if FILE exists and is executable.
[ -O FILE ]		True if FILE exists and is owned by the effective user ID.
[ -G FILE ]		True if FILE exists and is owned by the effective group ID.
[ -L FILE ]		True if FILE exists and is a symbolic link.
[ -N FILE ]		True if FILE exists and has been modified since it was last read.
[ -S FILE ]		True if FILE exists and is a socket.
[ FILE1 -nt FILE2 ]	True if FILE1 has been changed more recently than FILE2, or if FILE1 exists and FILE2 does not.
[ FILE1 -ot FILE2 ]	True if FILE1 is older than FILE2, or is FILE2 exists and FILE1 does not.
[ FILE1 -ef FILE2 ]	True if FILE1 and FILE2 refer to the same device and inode numbers.
[ -o OPTIONNAME ]	True if shell option "OPTIONNAME" is enabled.
[ -z STRING ]		True of the length of "STRING" is zero.
[ -n STRING ] or [ STRING ]	True of the length of "STRING" is non-zero.
[ STRING1 == STRING2 ] 	True if the strings are equal. "=" may be used instead of "==" for strict POSIX compliance.
[ STRING1 != STRING2 ] 	True if the strings are not equal.
[ STRING1 < STRING2 ] 	True if "STRING1" sorts before "STRING2" lexicographically in the current locale.
[ STRING1 > STRING2 ] 	True if "STRING1" sorts after "STRING2" lexicographically in the current locale.
[ ARG1 OP ARG2 ]	"OP" is one of -eq, -ne, -lt, -le, -gt or -ge. These arithmetic binary operators return true if "ARG1" is equal to, not equal to, less than, less than or equal to, greater than, or greater than or equal to "ARG2", respectively. "ARG1" and "ARG2" are integers.
-eq	equal
-ne	not equal
-lt	lesser than
-gt	greater than.
-ge	greater than or equal 

Expressions may be combined using the following operators, listed in decreasing order of precedence:

Operation		Effect
[ ! EXPR ]		True if EXPR is false.
[ ( EXPR ) ]		Returns the value of EXPR. This may be used to override the normal precedence of operators.
[ EXPR1 -a EXPR2 ]	True if both EXPR1 and EXPR2 are true.
[ EXPR1 -o EXPR2 ]	True if either EXPR1 or EXPR2 is true.

check both success and error

diff -q /tmp/1 /tmp/2 2>/dev/null && echo same || echo broken

elif

Elif statement.

#!/bin/bash
count=99
if [ $count -eq 100 ]
then
  echo "Count is 100"
elif [ $count -gt 100 ]
then
  echo "Count is greater than 100"
else
 echo "Count is less than 100"
fi

case statement

case EXPRESSION in 
CASE1) COMMAND-LIST;;
CASE2) COMMAND-LIST;;
CASEN) COMMAND-LIST;;
esac

Each case is an expression matching a pattern. The commands in the COMMAND-LIST for the first match are executed. The "|" symbol is used for separating multiple patterns, and the ")" operator terminates a pattern list. Each case plus its according commands are called a clause. Each clause must be terminated with ";;". Each case statement is ended with the esac statement.

read

read [options] NAME1 NAME2 ... NAMEN

One line is read from the standard input, or from the file descriptor supplied as an argument to the -u option. The first word of the line is assigned to the first name, NAME1, the second word to the second name, and so on, with leftover words and their intervening separators assigned to the last name, NAMEN. If there are fewer words read from the input stream than there are names, the remaining names are assigned empty values.

Option Meaning -a ANAME The words are assigned to sequential indexes of the array variable ANAME, starting at 0. All elements are removed from ANAME before the assignment. Other NAME arguments are ignored. -d DELIM The first character of DELIM is used to terminate the input line, rather than newline. -e readline is used to obtain the line. -n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input. -p PROMPT Display PROMPT, without a trailing newline, before attempting to read any input. The prompt is displayed only if input is coming from a terminal. -r If this option is given, backslash does not act as an escape character. The backslash is considered to be part of the line. In particular, a backslash-newline pair may not be used as a line continuation. -s Silent mode. If input is coming from a terminal, characters are not echoed. -t TIMEOUT Cause read to time out and return failure if a complete line of input is not read within TIMEOUT seconds. This option has no effect if read is not reading input from the terminal or from a pipe. -u FD Read input from file descriptor FD.

Do something on each line of output

#!/bin/bash

line
do
echo mv \'$line\' backupold
done < <(ls | grep -v backupold)

Do something on all lines of output.

#!/bin/bash
domassive()
{
  local item
  while read item
  do
    echo "I have been feed with the following lines" \"${item}\" &
  done
  wait
}

ls | domassive

Do something on each line of variable.

while read -r line
do
  echo "$line"
done <<< "$var"

file descriptors

File input and output are accomplished by integer handles that track all open files for a given process. These numeric values are known as file descriptors. The best known file descriptors are stdin, stdout and stderr, with file descriptor numbers 0, 1 and 2, respectively. These numbers and respective devices are reserved. Bash can take TCP or UDP ports on networked hosts as file descriptors as well.

The output below shows how the reserved file descriptors point to actual devices:

michel ~> ls -l /dev/std*
lrwxrwxrwx  1 root    root     17 Oct  2 07:46 /dev/stderr -> ../proc/self/fd/2
lrwxrwxrwx  1 root    root     17 Oct  2 07:46 /dev/stdin -> ../proc/self/fd/0
lrwxrwxrwx  1 root    root     17 Oct  2 07:46 /dev/stdout -> ../proc/self/fd/1

michel ~> ls -l /proc/self/fd/[0-2]
lrwx------  1 michel  michel   64 Jan 23 12:11 /proc/self/fd/0 -> /dev/pts/6
lrwx------  1 michel  michel   64 Jan 23 12:11 /proc/self/fd/1 -> /dev/pts/6
lrwx------  1 michel  michel   64 Jan 23 12:11 /proc/self/fd/2 -> /dev/pts/6

Bash allows for both standard output and standard error to be redirected

&> FILE

Echo test after header and before footer.

echo -e '\nTest\n' | cat header.txt /dev/fd/0 footer.txt

Since child processes inherit open file descriptors, it is good practice to close a file descriptor when it is no longer needed. This is done using the

exec fd<&- 

Here documents

Frequently, your script might call on another program or script that requires input. The here document provides a way of instructing the shell to read input from the current source until a line containing only the search string is found (no trailing blanks). All of the lines read up to that point are then used as the standard input for a command.

# Start here document
cat << BROWSERS
mozilla
konqueror
firefox
BROWSERS

# Remove prepended tabs.
<<-TAG
...
TAG

# Do not expand variables.
cat <<"EOF"
This won't be expanded: $PATH
EOF

here strings

<<< WORD

# The here-strings are a variation of the here-documents.

cat <<< "Hello world... $NAME is here..."

Modifications on variable

ANSWER=$(sed 's/abjorklund/king/g' <<< $USERNAME );echo $ANSWER

for loop

for NAME [in LIST ]; do COMMANDS; done

If [in LIST] is not present, it is replaced with in $@ and for executes the COMMANDS once for each positional parameter

SAVEIFS=$IFS ; IFS=$(echo -en "\n\b"); for FILM in $(cd /filmdir;ls -1);do echo ${FILM}; ln /temp/${FILM} /films/child/Pingu_${FILM}";done ;IFS=$SAVEIFS

for loop with space names in it.

while loop

The while construct allows for repetitive execution of a list of commands, as long as the command controlling the while loop executes successfully (exit status of zero). The syntax is:

while CONTROL-COMMAND; do CONSEQUENT-COMMANDS; done

#!/bin/bash
# echo 1 to 5.
i="1"
while [ $i -le 5 ]
do
echo $i
i=$[$i+1]
done

In this example, the break statement will skip the while loop when user enters -1, otherwise it will keep adding two numbers:

#!/bin/bash
while :
do
  read -p "Enter two numnbers ( - 1 to quit ) : " a b
  if [ $a -eq -1 ]
  then
    break
  fi
  ans=$(( a + b ))
  echo $ans
done

Early continuation with the continue statement

• To resume the next iteration of the enclosing WHILE loop use the continue statement as follows:

while [ condition ]
do
  statements1      #Executed as long as condition is true and/or, up to a disaster-condition if any.
  statements2
if (condition)
then
  continue   #Go to next iteration of I in the loop and skip statements3
fi
statements3
done

until loop

As long as this command fails, the loop continues. The syntax is the same as for the while loop:

until TEST-COMMAND; do CONSEQUENT-COMMANDS; done 

redirection

In the example below, output of the find command is used as input for the read command controlling a while loop:

#!/bin/bash
# This script creates a subdirectory in the current directory, to which old
# files are moved.
# Might be something for cron (if slightly adapted) to execute weekly or 
# monthly.

ARCHIVENR=`date +%Y%m%d`
DESTDIR="$PWD/archive-$ARCHIVENR"

mkdir $DESTDIR 

find $PWD -type f -a -mtime +5 | while read file
do
gzip "$file"; mv "$file".gz "$DESTDIR"
echo "$file archived"
done

break

Mind that break exits the loop, not the script.

continue

The continue statement resumes iteration of an enclosing for, while, until or select loop.

for name in "$LIST"; do
if [[ "$name" != *upper:* ]]; then
continue
fi
mv ...
done

select

The select construct allows easy menu generation. The syntax is quite similar to that of the for loop:
select WORD [in LIST]; do RESPECTIVE-COMMANDS; done
LIST is expanded, generating a list of items. The expansion is printed to standard error; each item is preceded by a number. If in LIST is not present, the positional parameters are printed, as if in $@ would have been specified. LIST is only printed once.
Upon printing all the items, the PS3 prompt is printed and one line from standard input is read. If this line consists of a number corresponding to one of the items, the value of WORD is set to the name of that item. If the line is empty, the items and the PS3 prompt are displayed again. If an EOF (End Of File) character is read, the loop exits. Since most users don't have a clue which key combination is used for the EOF sequence, it is more user-friendly to have a break command as one of the items. Any other value of the read line will set WORD to be a null string.
The read line is saved in the REPLY variable.

#!/bin/bash
echo "This script can make any of the files in this directory private."
echo "Enter the number of the file you want to protect:"
PS3="Your choice: "
QUIT="QUIT THIS PROGRAM - I feel safe now."
touch "$QUIT"
select FILENAME in *;
do
  case $FILENAME in
        "$QUIT")
          echo "Exiting."
          break
          ;;
        *)
          echo "You picked $FILENAME ($REPLY)"
          chmod go-rwx "$FILENAME"
          ;;
  esac
done
rm "$QUIT"

shift

The shift command is one of the Bourne shell built-ins that comes with Bash. This command takes one argument, a number. The positional parameters are shifted to the left by this number, N. The positional parameters from N+1 to $# are renamed to variable names from $1 to $# - N+1.

Say you have a command that takes 10 arguments, and N is 4, then $4 becomes $1, $5 becomes $2 and so on. $10 becomes $7 and the original $1, $2 and $3 are thrown away.

Types of variables

General assignment of values

VARIABLE=12

Using a declare statement, we can limit the value assignment to variables.

declare OPTION(s) VARIABLE=value 

Option	Meaning
-a	Variable is an array.
-f	Use function names only.
-i	The variable is to be treated as an integer; arithmetic evaluation is performed when the variable is assigned a value (see Section 3.4.6).
-p	Display the attributes and values of each variable. When -p is used, additional options are ignored.
-r	Make variables read-only. These variables cannot then be assigned values by subsequent assignment statements, nor can they be unset.
-t	Give each variable the trace attribute.
-x	Mark each variable for export to subsequent commands via the environment.
Using + instead of - turns off the attribute instead. When used in a function, declare creates local variables.

Constants.

readonly OPTION VARIABLE(s)

arrays

Create multi word array and loop over it.

while IFS= read -r line; do ARRAY+=( "$line" ); done < <( ls -al ) ; for i in "${ARRAY[@]}"; do echo '***' "$i"; done

Create multi word array and loop over it.

mapfile -t ARRAY < <( ls -la ) ; for i in "${ARRAY[@]}"; do echo '***' "$i"; done

declare -a ARRAYNAME
ARRAY=(value1 value2 ... valueN)
ARRAYNAME[indexnumber]=value

ARRAY=(one two three)
echo ${ARRAY[*]}
one two three

[bob in ~] echo $ARRAY[*]
one[*] 

[bob in ~] echo ${ARRAY[2]}
three 

[bob in ~] ARRAY[3]=four

[bob in ~] echo ${ARRAY[*]}
one two three four

Deleting array variables

The unset built-in is used to destroy arrays

Code example of Arrays.

#!/bin/bash
# Bash arrays have numbered indexes only, but they are sparse, ie you don't have to define all the indexes. An entire array can be assigned by enclosing thearray items in parenthesis:
arr=(Hello World)
# Individual items can be assigned with the familiar array syntax (unless you're used to Basic or Fortran):
arr2[0]=Hello2
arr2[1]=World2
# But it gets a bit ugly when you want to refer to an array item:
echo ${arr[0]} ${arr[1]}
echo ${arr2[0]} ${arr2[1]}
# In addition the following funky constructs are available:
echo ${arr[*]}         # All of the items in the array
echo ${!arr[*]}        # All of the indexes in the array
echo ${#arr[*]}        # Number of items in the array
echo ${#arr[0]}        # Length if item zero
echo ${arr2[*]}         # All of the items in the array
echo ${!arr2[*]}        # All of the indexes in the array
echo ${#arr2[*]}        # Number of items in the array
echo ${#arr2[0]}        # Length if item zero

add value to array

ping_reachable+=("${VAR}")

add to variable

Add content to variable

s=test1
s+=test2
echo $s
test1test2

Operations on variables

Using the ${#VAR} syntax will calculate the number of characters in a variable

Substitution

If VAR is not defined or null, the expansion of WORD is substituted; otherwise the value of VAR is substituted:

${VAR:-WORD}

This form is often used in conditional tests, for instance in this one:

[ -z "${COLUMNS:-}" ] && COLUMNS=80

It is a shorter notation for

if [ -z "${COLUMNS:-}" ]; then
         	COLUMNS=80
fi

the value is assigned to the parameter if it does not exist:

echo $TEST2

echo ${TEST2:=$TEST}
a_string
echo $TEST2
a_string

If variable does not exist set it to the one after :=

GOD="andreas"
echo ${GOOD:=${GOD}}
andreas

The following syntax tests the existence of a variable. If it is not set, the expansion of WORD is printed to standard out and non-interactive shells quit. A demonstration:

echo ${TESTVAR:?"There's so much I still wanted to do..."}
echo "TESTVAR is set, we can proceed."

Change variable if it exists. ${VARIABLE:+hej}

Removing substrings

To strip a number of characters, equal to OFFSET, from a variable, use this syntax:

${VAR:OFFSET:LENGTH}

The LENGTH parameter defines how many characters to keep, starting from the first character after the offset point. If LENGTH is omitted, the remainder of the variable content is taken:

export STRING="thisisaverylongname"
echo ${STRING:4}
isaverylongname
echo ${STRING:6:5}
avery

${VAR#WORD} and ${VAR##WORD}

These constructs are used for deleting the pattern matching the expansion of WORD in VAR. WORD is expanded to produce a pattern just as in file name expansion. If the pattern matches the beginning of the expanded value of VAR, then the result of the expansion is the expanded value of VAR with the shortest matching pattern ("#") or the longest matching pattern (indicated with "##"). If VAR is * or @, the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If VAR is an array variable subscribed with "*" or "@", the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list. This is shown in the examples below:

The opposite effect is obtained using "%" and "%%", as in this example below. WORD should match a trailing portion of string:

Modify variables content.

export TEST="beginning.andreas.ending" && export TEST="${TEST##beginning.}" && export TEST="${TEST%.ending}" && echo "${TEST}"

Replacing parts of variable names

${VAR/PATTERN/STRING} or ${VAR//PATTERN/STRING}

syntax. The first form replaces only the first match, the second replaces all matches of PATTERN with STRING:

variable expansion

parameter result

-----------   ------------------------------
${NAME}       polish.ostrich.racing.champion
${NAME#*.}           ostrich.racing.champion
${NAME##*.}                         champion
${NAME%%.*}   polish
${NAME%.*}    polish.ostrich.racing
${LINK/pending.htm/update.htm}              string pending.htm is replaced with update.htm in variable.
echo ${TEST} ; /temp/temp/test.txt =>  echo ${TEST//\//_} ; _temp_temp_test.txt = replace all occurrences of / with _.

convert minor to capital letters

${NAME^^}

Function syntax

Functions either use the syntax

function FUNCTION { COMMANDS; }
FUNCTION () { COMMANDS; }

Both define a shell function FUNCTION. The use of the built-in command function is optional; however, if it is not used, parentheses are needed. The commands listed between curly braces make up the body of the function. These commands are executed whenever FUNCTION is specified as the name of a command. The exit status is the exit status of the last command executed in the body.

Positional parameters in functions

Functions are like mini-scripts: they can accept parameters, they can use variables only known within the function (using the local shell built-in) and they can return values to the calling shell.
A function also has a system for interpreting positional parameters. However, the positional parameters passed to a function are not the same as the ones passed to a command or script.
When a function is executed, the arguments to the function become the positional parameters during its execution. The special parameter # that expands to the number of positional parameters is updated to reflect the change. Positional parameter 0 is unchanged. The Bash variable FUNCNAME is set to the name of the function, while it is executing.
If the return built-in is executed in a function, the function completes and execution resumes with the next command after the function call. When a function completes, the values of the positional parameters and the special parameter # are restored to the values they had prior to the function's execution. If a numeric argument is given to return, that status is returned. A simple example:

trap

trap [COMMANDS] [SIGNALS]
This instructs the trap command to catch the listed SIGNALS, which may be signal names with or without the SIG prefix, or signal numbers. If a signal is 0 or EXIT, the COMMANDS are executed when the shell exits. If one of the signals is DEBUG, the list of COMMANDS is executed after every simple command. A signal may also be specified as ERR; in that case COMMANDS are executed each time a simple command exits with a non-zero status. Note that these commands will not be executed when the non-zero exit status comes from part of an if statement, or from a while or until loop. Neither will they be executed if a logical AND (&&) or OR (||) result in a non-zero exit code, or when a command's return status is inverted using the ! operator.

(())

The (( )) construct expands and evaluates an arithmetic expression. If the expression evaluates as zero, it returns an exit status of 1, or "false". A non-zero expression returns an exit status of 0, or "true". This is in marked contrast to using the test and [ ] constructs previously discussed.

arithmetic operators

+    plus

-    minus

*    multiplication

/    division

**    exponentiation

%    modulo, or mod (returns the remainder of an integer division operation)

+=    plus-equal (increment variable by a constant)

-=    minus-equal (decrement variable by a constant)

*=    times-equal (multiply variable by a constant)

/=    slash-equal (divide variable by a constant)

%=    mod-equal (remainder of dividing variable by a constant)

id++ id--
variable post-increment and post-decrement

++id --id
variable pre-increment and pre-decrement

- +
unary minus and plus

! ~
logical and bitwise negation

**
exponentiation

* / %
multiplication, division, remainder

+ -
addition, subtraction

<< >>
left and right bitwise shifts

<= >= < >
comparison

== !=
equality and inequality

&
bitwise AND

^
bitwise exclusive OR

|
bitwise OR

&&
logical AND

||
logical OR

expr ? expr : expr
conditional evaluation

= *= /= %= += -= <<= >>= &= ^= |=
assignment

expr1 , expr2
comma

[[]]

Using the [[ ... ]] test construct, rather than [ ... ] can prevent many logic errors in scripts. For example, the &&, ||, <, and >  operators work within a [[ ]] test, despite giving an error within a [ ] construct.

{}

Block of code [curly brackets]. Also referred to as an inline group, this construct, in effect, creates an anonymous function (a function without a name). However, unlike in a "standard" function, the variables inside a code block remain visible to the remainder of the script.

Bash options

Abbreviation	Name	Effect
-C	noclobber	Prevent overwriting of files by redirection (may be overridden by >|)
-D	(none)		List double-quoted strings prefixed by $, but do not execute commands in script
-a	allexport	Export all defined variables
-b	notify		Notify when jobs running in background terminate (not of much use in a script)
-c	...		(none)	Read commands from ... 
-e	errexit		Abort script at first error, when a command exits with non-zero status (except in until or while loops, if-tests, list constructs)
-f	noglob		Filename expansion (globbing) disabled
-i	interactive	Script runs in interactive mode
-n	noexec		Read commands in script, but do not execute them (syntax check)
-o	Option-Name	(none)	Invoke the Option-Name option
-o	posix	POSIX	Change the behavior of Bash, or invoked script, to conform to POSIX standard.
-o	pipefail	pipe failure	Causes a pipeline to return the exit status of the last command in the pipe that returned a non-zero return value.
-p	privileged	Script runs as "suid" (caution!)
-r	restricted	Script runs in restricted mode (see Chapter 21).
-s	stdin		Read commands from stdin
-t	(none)		Exit after first command
-u	nounset		Attempt to use undefined variable outputs error message, and forces an exit
-v	verbose		Print each command to stdout before executing it
-x	xtrace		Similar to -v, but expands commands
-	(none)		End of options flag. All other arguments are positional parameters.
--	(none)		Unset positional parameters. If arguments given (-- arg1 arg2), positional parameters set to arguments.

bash prompt

^r	Search history.
!$	expands to the end word of the previous command.
!*	all args from last line in history
!$	last arg from last line in history
!^	first arg from last line in history   	:p	!xyz:p will print the command that would be executed rather than executing it. :p is also clever enough to add the printed command to your history list as the last command executed
!-2	Run command two commands away.
^a^b	Change all a to be in the former command.
h:	Remove a trailing file name component, leaving only the head.
t:	Remove all leading file name components, leaving the tail.
r:	Remove a trailing suffix of the form .xxx, leaving the basename.
e:	Remove all but the trailing suffix.
:s/a/b/	Substitute first a with b in command line.
:gs/a/b/	Substitute first a with b in command line.
cd !$:h	cd to directory mentioned last in your command.

Control Characters

Ctl-A	Beginning of line
Ctl-B  Backspace (nondestructive).
Ctl-C  Break. Terminate a foreground job.
Ctl-D  Log out from a shell (similar to exit).
Ctl-E	End of line.
Ctl-F	Move forward (nondestructive.
Ctl-G  BEL. On some old-time teletype terminals, this would actually ring a bell. In an xterm it might beep.
Ctl-H  Rubout (destructive backspace). Erases characters the cursor backs over while backspacing.
Ctl-I  Horizontal tab.
Ctl-J  Newline (line feed). In a script, may also be expressed in octal notation -- '\012' or in hexadecimal -- '\x0a'.
Ctl-K  Vertical tab.
Ctl-L  Formfeed (clear the terminal screen). In a terminal, this has the same effect as the clear command. When sent to a printer, a Ctl-L causes an advance to end of the paper sheet.
Ctl-M  Carriage return.
Ctl-Q  Resume (XON). This resumes stdin in a terminal.
Ctl-S  Suspend (XOFF). This freezes stdin in a terminal. (Use Ctl-Q to restore input.)
Ctl-U  Erase a line of input, from the cursor backward to beginning of line. In some settings, Ctl-U erases the entire line of input, regardless of cursor position.
Ctl-V  When inputting text, Ctl-V permits inserting control characters. For example, the following two are equivalent:
Ctl-V  is primarily useful from within a text editor.
Ctl-W  When typing text on the console or in an xterm window, Ctl-W erases from the character under the cursor backwards to the first instance of whitespace.
Ctl-Y	Pastes back text previously erased (with Ctl-K or Ctl-U).
Ctl-Z  Pauses a foreground job.

The default shell on most Linux operating systems is called Bash. There are a couple of important hotkeys that you should get familiar with if you plan to spend a lot of time at the command line. These shortcuts will save you a ton of time if you learn them.

Here is the List of Bash Shell Keyboard Shortcuts For Linux Users

Ctrl + A - Go to the beginning of the line you are currently typing on
Ctrl + E - Go to the end of the line you are currently typing on
Ctrl + L - Clears the Screen, similar to the clear command
Ctrl + U - Clears the line before the cursor position. If you are at the end of the line, clears the entire line.
Ctrl + H - Same as backspace
Ctrl + R - Let’s you search through previously used commands
Ctrl + C - Kill whatever you are running
Ctrl + D - Exit the current shell
Ctrl + Z - Puts whatever you are running into a suspended background process. fg restores it.
Ctrl + W - Delete the word before the cursor
Ctrl + K - Clear the line after the cursor
Ctrl + T - Swap the last two characters before the cursor
Esc + T - Swap the last two words before the cursor
Alt + F - Move cursor forward one word on the current line
Alt + B - Move cursor backward one word on the current line
Tab - Auto-complete files and folder names
Ctrl + W - erase word before cursor
Ctrl + Y - to paste it (as in delete and copy) all text in front of the cursor
Esc + . (or Esc + Underscore) - Insert Last Argument
Ctrl + b - Move back a character
Ctrl + f - Move forward a character
Ctrl + r - Search the history backwards
Ctrl + xx - Move between EOL and current cursor position
Ctrl + x @ - Show possible hostname completions
Alt + < - Move to the first line in the history
Alt + > - Move to the last line in the history
Alt + ? - Show current completion list
Alt + * - Insert all possible completions
Alt + / - Attempt to complete filename
Alt + . - Yank last argument to previous command
Alt + c - Capitalize the word
Alt + d - Delete word
Alt + l - Make word lowercase
Alt + n - Search the history forwards non-incremental
Alt + p - Search the history backwards non-incremental
Alt + r - Recall command
Alt + t - Move words around
Alt + u - Make word uppercase
Alt + back-space - Delete backward from cursor
Here "2T" means Press TAB twice
$ 2T - All available commands(common)
$ (string) 2T - All available commands starting with (string)
$ /2T - Entire directory structure including Hidden one
$ 2T - Only Sub Dirs inside including Hidden one
$ *2T - Only Sub Dirs inside without Hidden one
$ ~2T - All Present Users on system from "/etc/passwd"
$ $2T - All Sys variables
$ @2T - Entries from "/etc/hosts"
$ =2T - Output like ls or dir

Color text.

echo -e "\033[1mThis is bold text.\033[0m"
echo -e '\E[1;33;44m'"BOLD yellow text on blue background"; tput sgr0
Color	Foreground	Background
black	30	40
red	31	41
green	32	42
yellow	33	43
blue	34	44
magenta	35	45
cyan	36	46
white	37	47

graphical input output.

zenity

pipe

Character	Action
>		Redirect standard output
2>		Redirect standard error
2>&1		Redirect standard error to standard output
<		Redirect standard input
|		Pipe standard output to another command
>>		Append to standard output
2>&1|		Pipe standard output and stan
>/dev/null 2>&1	Send both standard output and error to /dev/null
some_cmd > some_file 2>&1 &  Run command send output to file with standard error and put in background.

Redirecting standard output temporary

#!/bin/bash
LOGFILE=~/script.log
exec 3>&1				#save FD1
exec 1>$LOGFILE			#stdout going to $LOGFILE
ls -alh /usr/share			#do a command
pwd					# and another command
who am i				# at least now I know ;-)
echo "Finished" >&3		# This now goes to stdout
echo "Now I'm writing to the log file again"
exec 1>&3				#Reset FD1
exec 3>&-				#Close FD3

Add logging to script

DATE_FILE=$(date '+%Y-%m-%d_%H-%M-%S')
exec >>/tmp/${0##*/}.${DATE_FILE}.$$.log
exec 2>>/tmp/${0##*/}.${DATE_FILE}.$$.log
set -x

echo to standard error

{ echo >&2 "This is an error" ; } >/dev/null
This is an error

1. Stack overflow gave this

>&2 echo "error"

where document

cat << EOF > /tmp/file
Hello my friend.
whats up?
EOF

get random number

echo $(( ( RANDOM % 20 )  + 1 ))

spawn processes

#!/bin/bash
JOB_LIMIT=20
for NUMBER in {1..40};do
   until [ $(wc -l <<< "${JOBS}") -lt 20 ] ;do
     echo "Working with ${NUMBER}. Job limit ${JOB_LIMIT}."
     JOBS=$(jobs)
     sleep 1
   done
   (sleep $(( ( RANDOM % 20 )  + 1 )); echo $NUMBER) &
   JOBS=$(jobs)
   sleep 0.2
done
wait
echo "All jobs are done."

Standard error in pipe

mvn clean install $@ | tee $logfile
echo ${PIPESTATUS[0]}

disable bell

bind 'set bell-style none'