Shell

The shell is the terminal of your operating system. This is the *nix shell.

Executing Complex Commands In The Shell

If you are writing out a more complex command, you have a few options to make it easier for you:

  1. You can use <c-x> followed by <c-e> to open your default editor and run the contents in your shell on close
  2. You can use fc in the same way as above
  3. You can make an alias or function in your rc file (e.g. zsh's .zshrc, or bash's .bashrc), or shell script and execute that if you are going to need it multiple times or over a long period of time (see below)

Basic Shell Tools

Set Options[x]

Using set to set or unset options in your shell script can solve a lot of problems. It is used like this example: set -a.

Option Effect
-e / -o errexit Stop running the script once an error is encountered
-u / -o nounset Stop running the script once an unset variable is encountered
-o pipefail Stop executing the rest of the pipe if any command fails

These set args can be combined like this: set -euo pipefail.

Have an explicit and informed decision about: shopt -s nullglob, shopt -s extglob, and shopt -s globstar.[26,27]

Aliases, Functions, and Scripts[28]

Within your ~/.bashrc file, you can use the keyword alias or a function to map a command to any other set of shell commands. Aliases and functions have different quirks[28], so be aware of them before choosing which to use.

Your RC file

alias hello="say 'Hello world!'"

goodbye() {
  say 'Goodbye world!'
}

Your shell

$ hello   # says hello world
$ goodbye # says goodbye world

When complete, run source ~/.bashrc or source ~/.zshrc and restart bash to have them take effect.

Shell Script

The first line of your shell script should be a shebang (#!) followed by the path of the shell you want to use (/bin/sh). Then all shell commands should follow. Once this is created, you need to change the mode (chmod) of your shell script to executable (e.g. if you have script.sh, you might use chmod u+x script.sh, with u+x meaning make this file executable for users). To run your new shell script, you need to preface the script's filename with ./.

echo "#!/bin/bash\n\necho 'Hello world!'" > script.sh
chmod u+x script.sh
./script.sh

Background Tasks[21]

You can run tasks in the background within a terminal window by placing an & at the end of the command you want running. You can see these background jobs with jobs and kill the job that you want with kill % followed by the index, or just kill % to kill all jobs.

You can also bring these background tasks to the foreground by typing fg % followed by the index. Or send a suspended job to the background by typing bg % followed by the index.

To suspend a process, you can use <c-z> (for instance, in suspending a Vim process).

Variables

Variables are all defined by a non-spaced variable name followed by an equals sign. Variables are recalled/invoked using the dollar sign followed by the variable name.

Variables that are within a string and directly next to another character that is not a space need to be enclosed within a dollar sign and curly braces, e.g. ${var_name}.

var_name="Bob"
number_var=123

echo "$var_name"
echo "$number_var"
echo "There are $number_var cans in ${var_name}'s closet."

Single quotes will preserve the literal value of all characters within it, while double quotes will allow the expansion/interpolation of the variables.[10] Always use double quotes when expanding your variables. Using double quotes around a variable will ensure that it will not be split up like a series of arguments[11].

var_name="Bob"
number_var=123

literal_values='$var_name $number_var'
expanded_values="$var_name $number_var"

echo "$literal_values"  # $var_name $number_var
echo "$expanded_values" # Bob 123

Parameter Expansions

Variables in the shell also have a variety of parameter expansions[12] that allow pattern matching, replacement, string slicing, and more.

Symbol Description
${var:-rep} If var is null or unset, replace it with literal rep or variable named $rep if preceded by a $ (e.g. ${var:-$rep})[13]
${#var} Length of var in bytes
${var#pattern} Remove shortest match of pattern if at start of var.
${var##pattern} Remove longest match of pattern if at start of var.
${var%pattern} Remove shortest match of pattern if at end of var.
${var%%pattern} Remove longest match of pattern if at end of var.
${var/pattern/replacement} Replace first pattern with replacement
${var//pattern/replacement} Replace all pattern's with replacement

Conditionals

There are a zillion operators to use in conditionals that are all slightly different than the usual <, >, ==, etc.[14] But here is the general construction of if/then conditionals in shell.

The big footgun is that -eq is used for integers and =/== is used for string comparison.

# If var_name is equal to other_var
if [[ "$var_name" -eq "$other_var" ]]; then
  # stuff happens
fi

# If var_name is NOT equal to other_var
if [[ ! "$var_name" -eq "$other_var" ]]; then
  # stuff happens
elif [[ "$var_name" -eq "$another_different_var" ]]; then
  # different stuff happens
else
  # other stuff happens
fi

Regular Expressions[15]

Regular expressions in shell use POSIX style patterns. A check can be made by using =~ between the variable and the pattern.

For instance, if we wanted to check if the date in a variable was between April 1 through April 3, we would use this:

if [[ "$var_name" =~ 'April [1-3]' ]]; then
  #stuff happens
fi

If you want to put the regex pattern into a variable, be sure to use single quotes in it's initialization and be sure to not use quotes around the variable when it's used. Double quotes disable the shell from recognizing it is a regex.[16]

re='April [1-3]' # use single quotes

# no quotes on $re
if [[ "$var_name" =~ $re ]]; then
  #stuff happens
fi

Arrays[22]

Arrays can be made in a shell script by using parentheses with spaces separating the elements.

arr=("A" "B" "Cat" "Delaware" 3)

for element in "${arr[@]}"
do
  echo "$element"
done

This is also a useful way of saving and passing command line arguments to a function. The following would add the date as a markdown header to the diary file.

args=(-c "let @a=\"\n## $(date +%Y%m%d)\n\n\n\"" -c "silent put a")
$EDITOR diary.md -c "$" ${args[@]}

Iteration

To iterate over a series of files, you can use a for loop:

# Files in directory include: a.txt, b.jpg, c.exe, d.txt
# This will only iterate through a.txt and d.txt .

for file in *.txt
do
  echo -e "${file}:\n$(cat $file)"
done

To iterate over each line in a file:

while read line; do
  do echo $line
done < path/to/file

To iterate over numbers:

for i in {0..10}
do
  echo -e "The number ${i}!"
done

# or

for ((i=0; i<=10; i+=1))
do
  echo -e "The number ${i}!"
done

To iterate over an array:

arr=(John Jane Bailey Arin)

for name in "${arr[@]}"; do
    echo -e "The name ${name}!"
done

Event Designators

An event designator is a reference to a command line entry in the history list. Unless the reference is absolute, events are relative to the current position in the history list.

Redirection and Pipe

To send the STDOUT of one command to a file, use the redirection operator >. To append the STDOUT to a file, use two redirection operators >>.

To read a file into a command as STDIN, use the reverse redirection operator or "less than" operator, <.

To connect the STDOUT of one command to the STDIN of another use the | symbol, commonly known as a pipe.

# thing1 outputs data to STDOUT and thing2 takes in input from STDIN
# long way
$ thing1 > tempfile
$ thing2 < tempfile

# shorter
$ thing1 > tempfile && thing2 < tempfile

# shortest
$ thing1 | thing2

Routing Different Outputs

To route STDOUT to different areas, you can precede the redirection operator with a 1. To route STDERR, use 2. To route both, use &.

$ cat file.txt 1>output.txt # STDOUT
$ cat file.txt 2>output.txt # STDERR
$ cat file.txt &>output.txt # Both

Command Substitution

Placing an argument within backticks or $(...) will execute the command first and insert the result. e.g.

$ vim `find "start/path" -name "filename.txt"`
$ # the same as
$ vim $(find "start/path" -name "filename.txt")
$ # resolves to
$ vim "start/path/filename.txt"

Using TDD with Shell Scripts

DIY

Luckily, shell scripts are so simple and rely pretty much entirely on globals, so things like mocks, before each/all, and test runners are all pretty straightforward. If your script is simple, I think rolling your own tiny framework is a good solution. I made one sufficient for a project in a couple hours, but now I can use it for anything going forward, assuming the project and requirements are sufficiently simple.

Third-Party

There are many libraries that can help ensure your app is well tested and make development akin to other paradigms. The most promising I have seen is ShellSpec[17], but there are lots[18-19].

Troubleshooting

Errors from Windows

If you got a script that looks totally fine but is throwing errors that make very little to no sense, like failing cd and cp, it's probably containing \r from a Windows computer.

tr -d "\r" < oldname.sh > newname.sh

References

  1. https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-base64-encode-and-decode-from-command-line/
  2. https://linuxize.com/post/how-to-create-bash-aliases/
  3. https://askubuntu.com/questions/172982/what-is-the-difference-between-redirection-and-pipe/172989#172989?newreg=cfc8024a2d4b40daa24578e47df2b7cf
  4. https://stackoverflow.com/a/11428439
  5. https://unix.stackexchange.com/questions/19654/how-do-i-change-the-extension-of-multiple-files
  6. https://mywiki.wooledge.org/BashFAQ/030
  7. https://devhints.io/bash
  8. Pure Bash Bible
  9. https://linoxide.com/make-bash-script-executable-using-chmod/
  10. https://stackoverflow.com/a/6697781/14857724
  11. https://guide.bash.academy/expansions/?=Command_Substitution#a1.3.0_2
  12. https://guide.bash.academy/expansions/?=Command_Substitution#p2.2.2_5
  13. https://stackoverflow.com/questions/27445455/what-does-the-colon-dash-mean-in-bash
  14. https://tldp.org/LDP/abs/html/comparison-ops.html#ICOMPARISON1
  15. https://stackoverflow.com/questions/18709962/regex-matching-in-a-bash-if-statement
  16. https://stackoverflow.com/questions/18709962/regex-matching-in-a-bash-if-statement#comment27568516_18709962
  17. https://github.com/shellspec/shellspec
  18. https://github.com/timurb/shell-test-frameworks
  19. https://thomaslevine.com/computing/shell-testing/
  20. https://rhodesmill.org/brandon/2009/commands-with-comma/
  21. https://www.maketecheasier.com/run-bash-commands-background-linux/
  22. https://stackoverflow.com/a/18898718
  23. https://www.shellcheck.net/
  24. https://clig.dev/
  25. https://wizardzines.com/comics/bash-errors/
  26. https://mastodon.social/@gnomon/108673882215603396
  27. https://www.gnu.org/software/bash/manual/html_node/The-Shopt-Builtin.html
  28. https://www.baeldung.com/linux/bash-alias-vs-script-vs-new-function
  29. Pure sh Bible
Incoming Links

Last modified: 202301072000