Shell & Bash

Almost every surprising shell behaviour comes from not knowing this order. The shell performs, strictly in sequence:

1. Brace expansion — {a,b}
2. Tilde expansion — ~
3. Parameter and variable expansion — $VAR
4. Command substitution — $(…)
5. Arithmetic expansion — $((…))
6. Word splitting on $IFS
7. Pathname expansion (globbing)
8. Quote removal

Steps 6 and 7 happen after your variable has been substituted. That is the entire explanation for the filename-with-spaces bug: the shell substitutes my report.pdf, then splits it into two words, then hands rm two arguments that don't exist.

Double quotes suppress steps 6 and 7. That is what they are for. Quote every expansion unless you have a specific reason not to.

$? holds the exit status of the last command: 0 for success, 1–255 otherwise. By convention 1 is a general error, 2 is a usage error, 126 means found but not executable, 127 means not found, and 128+N means killed by signal N — so 137 is SIGKILL (128+9), which is what an OOM kill looks like from the outside.

The pipeline trap

In a | b | c, $? is c's exit status. If a fails and c succeeds, the pipeline reports success. A backup script of the form pg_dump … | gzip > backup.gz will happily report success while writing a perfectly valid gzip of an error message.

set -o pipefail makes the pipeline return the rightmost non-zero status. ${PIPESTATUS[@]} gives you every stage's status individually.

Every process starts with three descriptors: 0 stdin, 1 stdout, 2 stderr. Redirection rewires them before the command runs.

Order matters, and it is the reverse of how people read it. >file 2>&1 first points 1 at the file, then points 2 at wherever 1 is now — the file. Both go to the file. 2>&1 >file first points 2 at wherever 1 currently is — the terminal — and then moves 1 to the file. stdout goes to the file, stderr still goes to the terminal. This is the single most common shell redirection bug.

[ is a command — historically /usr/bin/[. Its arguments go through word splitting and globbing like any other command's, which is why an unquoted empty variable turns [ $x = y ] into [ = y ] and a syntax error.

[[ ]] is a bash keyword. The shell parses it specially: no word splitting, no globbing on the left side, and it adds =~ for regex and &&/|| inside. Use it in bash, always.

(( )) is arithmetic evaluation: bare variable names, C-style operators, and an exit status that is 0 when the expression is non-zero — the opposite of every other exit code convention, which trips people up exactly once.

set -e exits on an unhandled non-zero status. set -u errors on an unset variable. set -o pipefail propagates pipeline failures. Together they turn a script that limps on after an error into one that stops. Every ops script should start with them.

Three places -e does not fire

• Inside a condition. if cmd; then, cmd && …, ! cmd — the failure is being tested, so it is not an error. Correct, but it means a function called from an if loses errexit throughout its body.
• Command substitution in an assignment. local x=$(failing) succeeds, because local succeeded. Split the declaration from the assignment.
• Anything but the last command in a pipeline — unless pipefail is set.

trap 'handler' EXIT runs the handler however the script ends — normal exit, error under set -e, or a caught signal. It is the shell's finally, and it is the right place for every temp file, lock and mount you created.

Trapping INT TERM as well lets you clean up when someone hits Ctrl-C or the orchestrator sends SIGTERM. In a container, a script that traps TERM and forwards it to its child is the difference between a two-second shutdown and the full 30-second grace period.

<(cmd) runs cmd and substitutes the path of a file descriptor (/dev/fd/63) that reads its output. Anything expecting a filename now accepts a command. No temp file, no cleanup, no race.

The classic use is comparing two things that aren't files — the output of a command on two hosts, a sorted list against another sorted list, a config as deployed versus as intended.

A for loop over 400 hosts running one SSH command each, serially at two seconds apiece, is thirteen minutes. xargs -P 20 makes it forty seconds, with a bounded concurrency you control.

-P N sets parallelism, -n 1 passes one argument per invocation, and -0 with find -print0 handles filenames with spaces and newlines safely. For anything more complex, GNU parallel adds per-job output grouping — which matters, because interleaved output from 20 concurrent jobs is unreadable.