Shell

The shell is the terminal of your operating system. This is the *nix shell.

Executing Complex Commands In The Shell

If you are writing out a more complex command, you have a few options to make it easier for you:

You can use <c-x> followed by <c-e> to open your default editor and run the contents in your shell on close
You can use fc in the same way as above
You can make an alias or function in your rc file (e.g. zsh's .zshrc, or bash's .bashrc), or shell script and execute that if you are going to need it multiple times or over a long period of time (see below)

Basic Shell Tools

cp [-r] ./source ./destination - Copy file or directory [-r] from source to destination
mv [-r] ./source ./destination - Move/rename file or directory [-r] from source to destination
rm [-r] ./file - Remove file or directory [-r] NOTE: THIS IS PERMANENT. There are no trash cans or recycle bins here.

Set Options[x]

Using set to set or unset options in your shell script can solve a lot of problems. It is used like this example: set -a.

Option	Effect
`-e` / `-o errexit`	Stop running the script once an error is encountered
`-u` / `-o nounset`	Stop running the script once an unset variable is encountered
`-o pipefail`	Stop executing the rest of the pipe if any command fails

These set args can be combined like this: set -euo pipefail.

Have an explicit and informed decision about: shopt -s nullglob, shopt -s extglob, and shopt -s globstar.[26,27]

Aliases, Functions, and Scripts[28]

Within your ~/.bashrc file, you can use the keyword alias or a function to map a command to any other set of shell commands. Aliases and functions have different quirks[28], so be aware of them before choosing which to use.

Your RC file

alias hello="say 'Hello world!'"

goodbye() {
  say 'Goodbye world!'
}

Your shell

$ hello   # says hello world
$ goodbye # says goodbye world

When complete, run source ~/.bashrc or source ~/.zshrc and restart bash to have them take effect.

Shell Script

The first line of your shell script should be a shebang (#!) followed by the path of the shell you want to use (/bin/sh). Then all shell commands should follow. Once this is created, you need to change the mode (chmod) of your shell script to executable (e.g. if you have script.sh, you might use chmod u+x script.sh, with u+x meaning make this file executable for users). To run your new shell script, you need to preface the script's filename with ./.

echo "#!/bin/bash\n\necho 'Hello world!'" > script.sh
chmod u+x script.sh
./script.sh

Background Tasks[21]

You can run tasks in the background within a terminal window by placing an & at the end of the command you want running. You can see these background jobs with jobs and kill the job that you want with kill % followed by the index, or just kill % to kill all jobs.

You can also bring these background tasks to the foreground by typing fg % followed by the index. Or send a suspended job to the background by typing bg % followed by the index.

To suspend a process, you can use <c-z> (for instance, in suspending a Vim process).

Variables

Variables are all defined by a non-spaced variable name followed by an equals sign. Variables are recalled/invoked using the dollar sign followed by the variable name.

Variables that are within a string and directly next to another character that is not a space need to be enclosed within a dollar sign and curly braces, e.g. ${var_name}.

var_name="Bob"
number_var=123

echo "$var_name"
echo "$number_var"
echo "There are $number_var cans in ${var_name}'s closet."

Single quotes will preserve the literal value of all characters within it, while double quotes will allow the expansion/interpolation of the variables.[10] Always use double quotes when expanding your variables. Using double quotes around a variable will ensure that it will not be split up like a series of arguments[11].

var_name="Bob"
number_var=123

literal_values='$var_name $number_var'
expanded_values="$var_name $number_var"

echo "$literal_values"  # $var_name $number_var
echo "$expanded_values" # Bob 123

Parameter Expansions

Variables in the shell also have a variety of parameter expansions[12] that allow pattern matching, replacement, string slicing, and more.

Symbol	Description
`${var:-rep}`	If `var` is null or unset, replace it with literal `rep` or variable named `$rep` if preceded by a `$` (e.g. `${var:-$rep}`)[13]
`${#var}`	Length of `var` in bytes
`${var#pattern}`	Remove shortest match of `pattern` if at start of `var`.
`${var##pattern}`	Remove longest match of `pattern` if at start of `var`.
`${var%pattern}`	Remove shortest match of `pattern` if at end of `var`.
`${var%%pattern}`	Remove longest match of `pattern` if at end of `var`.
`${var/pattern/replacement}`	Replace first `pattern` with `replacement`
`${var//pattern/replacement}`	Replace all `pattern`'s with `replacement`

Conditionals

There are a zillion operators to use in conditionals that are all slightly different than the usual <, >, ==, etc.[14] But here is the general construction of if/then conditionals in shell.

The big footgun is that -eq is used for integers and =/== is used for string comparison.

# If var_name is equal to other_var
if [[ "$var_name" -eq "$other_var" ]]; then
  # stuff happens
fi

# If var_name is NOT equal to other_var
if [[ ! "$var_name" -eq "$other_var" ]]; then
  # stuff happens
elif [[ "$var_name" -eq "$another_different_var" ]]; then
  # different stuff happens
else
  # other stuff happens
fi

There is also while and until, which executes until the exit status of the condition is non-zero or zero, respectively.

i=3
j=3

while test $i -ne 0; do
    i=$((i - 1))
done
echo "i is '$i'" # i is '0'

until test $j -ne 0; do
    j=$((j - 1))
done
echo "j is '$j'" # j is '3'

Regular Expressions[15]

Regular expressions in shell use POSIX style patterns. A check can be made by using =~ between the variable and the pattern.

For instance, if we wanted to check if the date in a variable was between April 1 through April 3, we would use this:

if [[ "$var_name" =~ 'April [1-3]' ]]; then
  #stuff happens
fi

If you want to put the regex pattern into a variable, be sure to use single quotes in it's initialization and be sure to not use quotes around the variable when it's used. Double quotes disable the shell from recognizing it is a regex.[16]

re='April [1-3]' # use single quotes

# no quotes on $re
if [[ "$var_name" =~ $re ]]; then
  #stuff happens
fi

Arrays[22]

Arrays can be made in a shell script by using parentheses with spaces separating the elements.

arr=("A" "B" "Cat" "Delaware" 3)

# Subscripts ([index])
echo "${arr[1]}" # B

# Slices ([@]:start:length)
echo "${arr[@]:2:1}" # Cat Delaware

# Iterate
for element in "${arr[@]}"; do
  echo "$element"
done

# Iterate through script arguments
for arg in "$@"; do
  echo "$arg"
done

# Test if item is in array
value="A"
if [[ " ${arr[*]} " =~ [[:space:]]${value}[[:space:]] ]]; then
    echo "Found '$value'."
fi

This is also a useful way of saving and passing command line arguments to a function. The following would add the date as a markdown header to the diary file.

args=(-c "let @a=\"\n## $(date +%Y%m%d)\n\n\n\"" -c "silent put a")
$EDITOR diary.md -c "$" "${args[@]}"

Iteration

To iterate over a series of files, you can use a for loop:

# Files in directory include: a.txt, b.jpg, c.exe, d.txt
# This will only iterate through a.txt and d.txt .

for file in *.txt; do
  echo -e "${file}:\n$(cat $file)"
done

To iterate over each line in a file:

while read -r line; do
  do echo $line
done < path/to/file

To iterate over numbers:

for i in {0..10}; do
  echo -e "The number ${i}!"
done

# or

for ((i=0; i<=10; i+=1)); do
  echo -e "The number ${i}!"
done

To iterate over an array:

for name in John Jane Bailer Arin; do
    echo "$name"
done

arr=(John Jane Bailey Arin)

for name in "${arr[@]}"; do
    echo -e "The name ${name}!"
done

Event Designators

An event designator is a reference to a command line entry in the history list. Unless the reference is absolute, events are relative to the current position in the history list.

$_ Repeat the last argument used, e.g. mkdir folder-name && cd "$_"
!! Repeats the previous command
!10 Repeat the 10th command from the history
!-2 Repeat the 2nd command (from the last) from the history
!string Repeat the command that starts with “string” from the history
!?string Repeat the command that contains the word “string” from the history
^str1^str2^ Substitute str1 in the previous command with str2 and execute it
!!:$ Gets the last argument from the previous command.
!string:n Gets the nth argument from the command that starts with “string” from the history.
!^ first argument of the previous command
!$ last argument of the previous command
!* all arguments of the previous command
!:2 second argument of the previous command
!:2-3 second to third arguments of the previous command
!:2-$ second to last arguments of the previous command
!:2* second to last arguments of the previous command
!:2- second to next to last arguments of the previous command
!:0 the command itself

Redirection and Pipe

To send the STDOUT of one command to a file, use the redirection operator >. To append the STDOUT to a file, use two redirection operators >>.

To read a file into a command as STDIN, use the reverse redirection operator or "less than" operator, <.

To connect the STDOUT of one command to the STDIN of another use the | symbol, commonly known as a pipe.

# thing1 outputs data to STDOUT and thing2 takes in input from STDIN
# long way
$ thing1 > tempfile
$ thing2 < tempfile

# shorter
$ thing1 > tempfile && thing2 < tempfile

# shortest
$ thing1 | thing2

Routing Different Outputs

To route STDOUT to different areas, you can precede the redirection operator with a 1. To route STDERR, use 2. To route both, use &.

$ cat file.txt 1>output.txt # STDOUT
$ cat file.txt 2>output.txt # STDERR
$ cat file.txt &>output.txt # Both

Heredocs[30]

You can use heredocs for multiline strings. It is signified using << followed by the delimiter, often EOF; then the multiline string must be ended with a single line containing your delimiter. If you want to not allow parameter substitution, at the start, surround the delimiter with double quotes.

cat <<EOF
This is a long string. It allows variables, like $HOME.

It can contain any characters, as long as it isn't a line containing only the delimiter. Even special characters, like these:   
EOF

cat <<"EOF"
This doesn't allow variables, like $HOME.
EOF

You can write a heredoc to a file using tee[31]:

tee newfile <<EOF
line 1
line 2
line 3
EOF

Command Substitution

Placing an argument within backticks or $(...) will execute the command first and insert the result. e.g.

$ vim `find "start/path" -name "filename.txt"`
$ # the same as
$ vim $(find "start/path" -name "filename.txt")
$ # resolves to
$ vim "start/path/filename.txt"

Using TDD with Shell Scripts

DIY

Luckily, shell scripts are so simple and rely pretty much entirely on globals, so things like mocks, before each/all, and test runners are all pretty straightforward. If your script is simple, I think rolling your own tiny framework is a good solution. I made one sufficient for a project in a couple hours, but now I can use it for anything going forward, assuming the project and requirements are sufficiently simple.

Third-Party

There are many libraries that can help ensure your app is well tested and make development akin to other paradigms. The most promising I have seen is ShellSpec[17], but there are lots[18-19].

Troubleshooting

Errors from Windows

If you got a script that looks totally fine but is throwing errors that make very little to no sense, like failing cd and cp, it's probably containing \r from a Windows computer.

tr -d "\r" < oldname.sh > newname.sh

References

Incoming Links

Last modified: 202412060412