What this section covers
- Standard Unix streams, redirection and piping
- How and why to direct script diagnostic messages to standard error, but function results to standard output
- Argument defaulting
- Automatically creating a log file for a script
- Capturing standard output in a variable using backtick quoting
Standard streams
As described in a section of Linux Fundamentals page, there are three built-in, Standard Streams, each with a well-defined stream number:
- 0 - standard input
- 1 - standard output
- 2 - standard error
Redirection characters allow you to control the stream input or output
- "<" read from
- ">" write to (and it's cousin ">>" append to, when the destination is a file)
Examples:
- redirect standard output to a file, overwriting any exiting contents:
echo "Output text" > out.txt
echo "Output
text" 1> out.txt - redirect standard output to a file, appending to any exiting contents:
echo "More text" >> out.txt
echo "More text" 1>> out.txt - redirect standard error output to a file, overwriting any exiting contents:
echo "Error
text" 2> err.txt - redirect standard error to standard output:
ls xxxx > ls.log 2>&1
- redirect standard output to standard error:
echo "
Output that will go to standard error
" 1>&2
There's also the tee program, which takes its standard input and writes it to the specified file as well as to its standard output.
For example, here's how to call the step_01.sh script so that its output goes to both the terminal and to a file.
~/workshop/step_01.sh helloWorld My name is Anna | tee step_01.log
See Piping in the Linux Fundamentals page.
When standard output and standard error streams are used
Let's look at a command that shows the difference between standard error and standard output:
# list 2 files, one that exists and one that does not ls ~/.profile ~/xxx
Produces this output:
ls: cannot access 'xxx': No such file or directory .profile
What is not obvious here, since both streams are displayed on the terminal, is that the diagnostic text "ls: cannot access 'xxx': No such file or directory" is being written to standard error, while the listing of the existing file (text ".profile") is being written to standard output.
To see this, redirect output to different files and look at their contents:
ls ~/.profile ~/xxx 1>stdout.txt 2>stderr.txt cat stdout.txt cat stderr.txt
The step_02.sh Script
Here's a step_02.sh script that builds on our step_01.sh work. It is located at ~/workshop/step_02.sh. Make sure it is executable (chmod +x ~/step_02.txt
).
#!/bin/bash # Script version global variable. Edit this whenever changes are made. __ADVANCED_BASH_VERSION__="step_02" # ======================================================================= # Helper functions # ======================================================================= # Echo's its arguments to std error echo_se() { echo "$@" 1>&2; } # Sets up auto-logging to a log file in the current directory # using the specified logFileTag (arg 1) in the log file name. auto_log() { local logFileTag="$1" if [[ "$logFileTag" != "" ]]; then local logFilePath="./autoLog_${logFileTag}.log" echo_se ".. logging to $logFilePath" exec 1> >(tee "$logFilePath") 2>&1 else echo_se "** ERROR in autoLog: no logFile argument provided" exit 255 fi } # ======================================================================= # Command processing functions # ======================================================================= # function that says "Hello World!" and displays user-specified text. function helloWorld() { local txt1=$1 local txt2=$2 shift; shift local rest=$@ echo "Hello World!" echo " text 1: '$txt1'" echo " text 2: '$txt2'" echo " rest: '$rest'" } # function that displays its 1st argument on standard output and # its 2nd argument on standard error function stdStreams() { local outTxt=${1:-"text for standard output"} local errTxt=${2:-"text for standard error"} echo "to standard output: '$outTxt'" echo_se "to standard error: '$errTxt'" } # function that illustrates auto-logging and capturing function output # arg 1 - (required) tag to identify the logfile # arg 2 - (optional) text for standard output # arg 3 - (optional) text for standard error function testAutolog() { local logFileTag="$1" local outTxt=${2:-"text for standard output"} local errTxt=${3:-"text for standard error"} auto_log "$logFileTag" echo -e "\n1) Call stdStreams with output and error text:" stdStreams "$outTxt" "$errTxt" echo -e "\n2) Capture echo output in a variable and display it:" local output=`echo $outTxt` echo -e " echo output was:\n$output" echo -e "\n3) Call echo_se with error text:" echo_se "$errTxt" echo -e "\n4)Capture echo_se function output in a variable and display it:" output=`echo_se "$errTxt"` echo -e "echo_se output was: '$output'" } # ======================================================================= # Main script command-line processing # ======================================================================= function usage() { echo " advanced_bash.sh, version $__ADVANCED_BASH_VERSION__ Usage: advanced_bash.sh <command> [arg1 arg2...] Commands: helloWorld [text to display] stdStreams [text for stdout] [text for stderr] testAutolog <logFileTag> [text for stdout] [text for stderr] " exit 255 } CMD=$1 # initially $1 will be the command shift # after "shift", $1 will be the 2nd command-line argument; $2 the 3rd, etc. case "$CMD" in helloWorld) helloWorld "$@" ;; stdStreams) stdStreams "$1" "$2" ;; testAutolog) testAutolog "$1" "$2" "$3" ;; *) usage ;; esac
Some Parts
echo_se function
In our new Helper functions section we've defined a function that redirects its arguments to standard error. This allows the script to separate diagnostic messages about its execution, from function "results" written to standard output.
# Echo's its arguments to std error echo_se() { echo "$@" 1>&2; }
stdStreams function & command processing
In our new Command processing functions section we've defined a function that writes its first argument to standard output and its second argument to standard error:
# function that displays its 1st argument on standard output and # its 2nd argument on standard error function stdStreams() { local outTxt=${1:-"text for standard output"} local errTxt=${2:-"text for standard error"} echo "to standard output: '$outTxt'" echo_se "to standard error: '$errTxt'" }
We've also added command-line processing and usage support for new functions.
argument defaulting
Often you will want to provide a default for arguments not explicitly provided by the user. This can be done using this rather odd syntax, with ":-" characters separating the positional argument number from the desired default.
textArg=${1:-"text default for 1st argument"} integerArg=${2:-54321}
simpler usage function
We've replaced the multiple echo lines in usage with a single echo'd string, illustrating that the invisible line feeds in the echo'd text are faithfully preserved.
function usage() { echo " advanced_bash.sh, version $__ADVANCED_BASH_VERSION__ Usage: advanced_bash.sh <command> [arg1 arg2...] Commands: helloWorld [text to display] stdStreams [text for stdout] [text for stderr] testAutolog <logFileTag> [text for stdout] [text for stderr] " exit 1 }
Here's how to call the stdStreams command specifying the text "hello world!" for standard output and "goodbye world!" for standard error.
~/workshop/step_02.sh stdStreams 'hello world!' "goodbye world!"
But how can you tell which stream is which? They're both written to the terminal by default!
exercise 1
Call the stdStreams command in some manner that can distinguish between standard output and standard error.
exercise 2
Call the stdStreams command so that all output goes to standard output.
More Parts - automatic logging
So you have written a user-callable script like step_02.sh, and you want to make sure all its output is logged to a log file. This would save the user from having to do the needed redirection on their command line – it would be done automatically by the script itself. Is this possible?
Yes! But the syntax is really weird, and I'm not sure even I can completely explain it:
exec 1> >(tee my_logfile.log) 2>&1
As far as I can tell:
- exec 1> causes redirection of all standard output for the duration of the current shell environment (here the script).
- the 2>&1 at the end is our normal "redirect standard error to standard output" idiom
- >(tee "my_logfile.log") is the magic that says to write standard output (which now includes standard error text) to a file via tee.
- the >( ) syntax also sends standard output to a sub-shell (more on this later), which appears to be necessary because otherwise tee would just act on its piped-in standard input.
auto_log function
The new auto_log helper function sets up automatic logging to a log file in the current directory, named using a tag string specified as its 1st argument.
If you arrange for such a tag to be uniquely associated with a specific invocation of a script (e.g. the output file prefix used for all FASTQ alignment files), this will create a uniquely named log file for each different script invocation. This is useful, for example, when calling the same script with different arguments in a file of batch commands at TACC.
Importantly, auto_log reports an error and terminates script execution if no tag string is specified (we'll see much more on error handling shortly).
The test is made using a bash if/else/fi block, where double brackets ('"[[ ]]") enclose the test, always followed by a semicolon (";"). Here the test is for string equality (see https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html for a complete list of bash comparison operators).
# Sets up auto-logging to a log file in the current directory # using the specified logFileTag (arg 1) in the log file name. auto_log() { local logFileTag="$1" if [[ "$logFileTag" != "" ]]; then local logFilePath="./autoLog_${logFileTag}.log" echo_se ".. logging to $logFilePath" exec 1> >(tee "$logFilePath") 2>&1 else echo_se "** ERROR in autoLog: no logFile argument provided" exit 255 fi }
testAutolog function and command processing
The new testAutolog command processing function has a lot going on. It:
- Starts automatic logging to a log file named using its 1st logFileTag argument.
- Calls stdStreams with its 2nd and 3rd arguments.
- Calls stdStreams again, but this time capturing its output in a local variable using backtick execution syntax, then displays the captured text.
- Calls the echo_se function with some text.
- Calls the echo_se function again, capturing its output in a local variable, then displays the captured text.
# function that illustrates auto-logging and capturing function output # arg 1 - (required) tag to identify the logfile # arg 2 - (optional) text for standard output # arg 3 - (optional) text for standard error function testAutolog() { local logFileTag="$1" local outTxt=${2:-"text for standard output"} local errTxt=${3:-"text for standard error"} auto_log "$logFileTag" echo -e "\n1) Call stdStreams with output and error text:" stdStreams "$outTxt" "$errTxt" echo -e "\n2) Capture echo output in a variable and display it:" local output=`echo $outTxt` echo -e " echo output was:\n$output" echo -e "\n3) Call echo_se with error text:" echo_se "$errTxt" echo -e "\n4)Capture echo_se function output in a variable and display it:" output=`echo_se "$errTxt"` echo -e "echo_se output was: '$output'" }
So why is echo -e used in the testAutoLog function? Because the -e argument to echo enables interpretation of backslash escapes. For example, "\n" will be interpreted as a newline, and "\t" as a tab character.
exercise 3
Call the testAutoLog command with no further command line arguments. What happens, and why?
exercise 4
What output is produced when you call the testAutoLog command with a tag string of "test1".
Why are both standard error and standard output text strings displayed in segment 1) when stdStreams is called?
Why does the $output variable contain the outTxt in 2) when echo is called with output being captured?
Why is the $output variable empty in segment 4) when echo_se is called with output being captured?
What log file is produced, what are its contents, and why?