Stream handling

What this section covers

Standard Unix streams, redirection and piping
How and why to direct script diagnostic messages to standard error, but function results to standard output
Argument defaulting
Automatically creating a log file for a script
Capturing standard output in a variable using backtick quoting

Standard streams

As described Intro Unix: Standard streams and redirection each with a well-defined stream number:

0 - standard input
1 - standard output
2 - standard error

Redirection characters allow you to control the stream input or output

"<" read from
">" write to (and it's cousin ">>" append to, when the destination is a file)

Examples:

redirect standard output to a file, overwriting any exiting contents:
echo "Output text" > out.txt echo "Some output" 1> out.txt
redirect standard output to a file, appending to any exiting contents:
echo "Some text" >> out.txt echo "More text" 1>> out.txt
redirect standard error output to a file, overwriting any exiting contents:ls xxxx 2> err.txt
redirect standard error to standard output, and redirect standard output to a file:
ls xxxx > ls.log 2>&1
redirect standard output to standard error:
echo "Output that will be redirected to standard error" 1>&2 echo "Output that will be redirected standard error" 2> err.txt 1>&2

There's also the tee program, which takes its standard input and writes it to the specified file as well as to its standard output.

For example, here's how to call the step_01.sh script so that its standard output goes to both the Terminal and to a file.

~/workshop/step_01.sh helloWorld My name is Anna | tee step_01.log

See Standard streams and piping in the Intro Unix wiki.

When standard output and standard error streams are used

Let's look at a command that shows the difference between standard error and standard output:

# list 2 files, one that exists and one that does not
ls ~/.profile xxx

Produces this output in your terminal:

ls: cannot access 'xxx': No such file or directory
.profile

What is not obvious here, since both streams are displayed on the terminal, is that the diagnostic text "ls: cannot access 'xxx': No such file or directory" is being written to standard error, while the listing of the existing file (text ".profile") is being written to standard output.

To see this, redirect output to different files and look at their contents:

ls ~/.profile ~/xxx 1>out.txt 2>err.txt
cat out.txt
cat err.txt

Finally redirect both standard output and standard error to a single file:

ls ~/.profile ~/xxx 1> ls.log 2>&1
# Note that the 2>&1 syntax has to come **after** the redirection of standard output

# So this does not redirect stdout
ls ~/.profile ~/xxx 2>&1 1> ls.log

The step_02.sh Script

Here's a step_02.sh script that builds on our step_01.sh work. It is located at ~/workshop/step_02.sh. Make sure it is executable (chmod +x ~/workshop/step_02.txt).

#!/bin/bash

# Script version global variable. Edit this whenever changes are made.
__ADVANCED_BASH_VERSION__="step_02"

# =======================================================================
#  Helper functions
# =======================================================================

# Echo's its arguments to std error
echo_se() { echo "$@" 1>&2; }

# Sets up auto-logging to a log file in the current directory
# using the specified logFileTag (arg 1) in the log file name.
auto_log() {
  local logFileTag="$1"
  if [[ "$logFileTag" != "" ]]; then
    local logFilePath="./autoLog_${logFileTag}.log"
    echo_se ".. logging to $logFilePath"
    exec 1> >(tee "$logFilePath") 2>&1
  else
    echo_se "** ERROR in auto_log: no logFile argument provided"
    exit 255
  fi
}

# =======================================================================
#  Command processing functions
# =======================================================================

# function that says "Hello World!" and displays user-specified text.
function helloWorld() {
  local txt1=$1
  local txt2=$2
  shift; shift
  local rest=$@
  echo "Hello World!"
  echo "  text 1: '$txt1'"
  echo "  text 2: '$txt2'"
  echo "  rest:   '$rest'"
}

# function that displays its 1st argument on standard output and
# its 2nd argument on standard error
function stdStreams() {
  local outTxt=${1:-"text for standard output"}
  local errTxt=${2:-"text for standard error"}
  echo    "to standard output: '$outTxt'"
  echo_se "to standard error:  '$errTxt'"
}

# function that illustrates auto-logging and capturing function output
#  arg 1 - (required) tag to identify the logfile
#  arg 2 - (optional) text for standard output
#  arg 3 - (optional) text for standard error
function testAutolog() {
  local logFileTag="$1"
  local outTxt=${2:-"text for standard output"}
  local errTxt=${3:-"text for standard error"}

  auto_log "$logFileTag"

  echo -e "\n1) Call stdStreams with output and error text:"
  stdStreams "$outTxt" "$errTxt"

  echo -e "\n2) Capture echo output in a variable and display it:"
  local output=`echo $outTxt`
  echo -e "   echo output was:\n$output"

  echo -e "\n3) Call echo_se with error text:"
  echo_se "$errTxt"

  echo -e "\n4)Capture echo_se function output in a variable and display it:"
  output=`echo_se "$errTxt"`
  echo -e "echo_se output was: '$output'"
}

# =======================================================================
#  Main script command-line processing
# =======================================================================

function usage() {
  echo "
advanced_bash.sh, version $__ADVANCED_BASH_VERSION__

Usage: advanced_bash.sh <command> [arg1 arg2...]

Commands:
  helloWorld [text to display]
  stdStreams [text for stdout] [text for stderr]
  testAutolog <logFileTag> [text for stdout] [text for stderr]
"
  exit 255
}

CMD=$1    # initially $1 will be the command
shift     # after "shift", $1 will be the 2nd command-line argument; $2 the 3rd, etc.
          # and $@ will be arguments 2, 3, etc.
case "$CMD" in
  helloWorld) helloWorld "$@"
    ;;
  stdStreams) stdStreams "$1" "$2"
    ;;
  testAutolog) testAutolog "$1" "$2" "$3"
    ;;
  *) usage
    ;;
esac

Some Parts

echo_se function

In our new Helper functions section we've defined a function that redirects its arguments to standard error. This allows the script to separate diagnostic messages about its execution, from function "results" written to standard output.

# Echo's its arguments to std error
echo_se() { echo "$@" 1>&2; }

stdStreams function & command processing

In our new Command processing functions section we've defined a function that writes its first argument to standard output and its second argument to standard error:

# function that displays its 1st argument on standard output and
# its 2nd argument on standard error
function stdStreams() {
  local outTxt=${1:-"text for standard output"}
  local errTxt=${2:-"text for standard error"}
  echo    "to standard output: '$outTxt'"
  echo_se "to standard error:  '$errTxt'"
}

argument defaulting

Often you will want to provide a default for arguments not explicitly provided by the user. This can be done using this rather odd syntax, with ":-" characters separating the positional argument number from the desired default. The entire defaulting construct is enclosed in "${ }".

textArg=${1:-"text default for 1st argument"}
integerArg=${2:-54321}

simpler usage function

We've also added command-line processing and usage support for the new functions.

In the usage function, we've replaced the multiple echo lines with a single echo'd string, illustrating that the invisible line feeds in the echo'd text are faithfully preserved. The function also exits with a non-0 ("failure") return code, since no valid command processing was performed.

function usage() {
  echo "
advanced_bash.sh, version $__ADVANCED_BASH_VERSION__

Usage: advanced_bash.sh <command> [arg1 arg2...]

Commands:
  helloWorld [text to display]
  stdStreams [text for stdout] [text for stderr]
  testAutolog <logFileTag> [text for stdout] [text for stderr]
"
  exit 255
}

Here's how to call the stdStreams command specifying the text "hello world" for standard output and "goodbye world!" for standard error.

~/workshop/step_02.sh stdStreams "hello world' 'goodbye world!'

But how can you tell which stream is which? They're both written to the Terminal by default!

exercise 1

Call the stdStreams command in some manner that can distinguish between standard output and standard error.

Hints

1> redirects standard output to a file
2> redirects standard error to a file
Redirecting to /dev/null discards data

Solutions

# discard standard error and just display standard output
~/workshop/step_02.sh stdStreams 'hello world!' "goodbye world!" 2>/dev/null

# discard standard output and just display standard error
~/workshop/step_02.sh stdStreams 'hello world!' "goodbye world!" 1>/dev/null 

# or redirect the two streams to different files
~/workshop/step_02.sh stdStreams 'hello world' "goodbye world!" 1>stdout.txt 2>stderr.txt
cat stdout.txt
cat stderr.txt

exercise 2

Call the stdStreams command so that all output goes to standard output.

Solution

~/workshop/step_02.sh stdStreams 'hello world!' "goodbye world!" 2>&1

exercise 3

What is written to the ssout.txt file when the following is executed, and why?

~/workshop/step_02.sh stdStreams 'hello world!' "goodbye world!" | tee ssout.txt

Solution

Only the standard output text is written:

to standard output: 'hello world'

Remember the pipe ("|") connects one program's standard output (here from the step_02.sh script) to the next program's standard input (here to the tee program); standard error is ignored by the pipe (unless redirected).

More Parts - automatic logging

So you have written a user-callable script like step_02.sh, and you want to make sure all its output is logged to a log file. This would save the user from having to do the needed redirection on their command line – it would be done automatically by the script itself. Is this possible?

Yes! But the syntax is really weird, and I'm not sure even I can completely explain it:

exec 1> >(tee my_logfile.log) 2>&1

As far as I can tell:

exec 1> causes redirection of all standard output for the duration of the current shell environment (here the script).
the 2>&1 at the end is our normal "redirect standard error to standard output" idiom
>(tee "my_logfile.log") is the magic that says to write standard output (which now includes standard error text) to a file via tee.
- the >( ) syntax also sends all the current execution environment's standard output to a sub-shell (more on this later)
- appears to be necessary because otherwise tee would just act on its piped-in standard input

auto_log function

The new auto_log helper function sets up automatic logging to a log file in the current directory, named using a tag string specified as its 1st argument.

If you arrange for such a tag to be uniquely associated with a specific invocation of a script (e.g. the output file prefix used for all FASTQ alignment files), this will create a uniquely named log file for each different script invocation. This is useful, for example, when calling the same script with different arguments in a file of batch commands at TACC.

Importantly, auto_log reports an error and terminates script execution if no tag string is specified, via exit 255 (we'll see much more on error handling shortly).

The conditional test is made using a bash if/else/fi block, where double brackets ('"[[ ]]") enclose the test, always followed by a semicolon (";") or a new line. Note that there must always be a space after the open brackets, and one before the close brackets. Here the test is for string equality (see https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html for a complete list of bash comparison operators).

# Sets up auto-logging to a log file in the current directory
# using the specified logFileTag (arg 1) in the log file name.
auto_log() {
  local logFileTag="$1"
  if [[ "$logFileTag" != "" ]]; then
    local logFilePath="./autoLog_${logFileTag}.log"
    echo_se ".. logging to $logFilePath"
    exec 1> >(tee "$logFilePath") 2>&1
  else
    echo_se "** ERROR in auto_log: no logFile argument provided"
    exit 255
  fi
}

The general form of an if/then/else/fi statement is:

if [[ some_test ]]
then
what_to_do_when_some_test_is_true (0)
else
what_to_do_when_some_test_is_false (not 0)
fi

As always in bash, clauses (technically commands themselves) can be put on one line if separated by a semicolon ( ; ).

if [[ some_test ]]; then echo "Test was true"; else echo "Test was false"; fi

testAutolog function and command processing

The new testAutolog command processing function has a lot going on. It:

Starts automatic logging to a log file named using its 1st logFileTag argument, by calling the auto_log function.
Uses echo -e where the -e argument to echo enables interpretation of backslash escapes.
- e.g. "\t" as a tab character and "\n" will be interpreted as a newline
Calls stdStreams with its 2nd and 3rd arguments.
Calls echo capturing its output in a local variable using backtick execution syntax, then displays the captured text.
- see Quoting in the shell
Calls the echo_se function with some text.
Calls the echo_se function again, capturing its output in a local variable, then displays the captured text.

# function that illustrates auto-logging and capturing function output
#  arg 1 - (required) tag to identify the logfile
#  arg 2 - (optional) text for standard output
#  arg 3 - (optional) text for standard error
function testAutolog() {
  local logFileTag="$1"
  local outTxt=${2:-"text for standard output"}
  local errTxt=${3:-"text for standard error"}

  auto_log "$logFileTag"

  echo -e "\n1) Call stdStreams with output and error text:"
  stdStreams "$outTxt" "$errTxt"

  echo -e "\n2) Capture echo output in a variable and display it:"
  local output=`echo $outTxt`
  echo -e "   echo output was:\n$output"

  echo -e "\n3) Call echo_se with error text:"
  echo_se "$errTxt"

  echo -e "\n4)Capture echo_se function output in a variable and display it:"
  output=`echo_se "$errTxt"`
  echo -e "echo_se output was: '$output'"
}

exercise 4

Call the testAutolog command with no further command line arguments. What happens, and why?

Solution

Executing:

~/workshop/step_02.sh testAutolog

produces this output:

** ERROR in auto_log: no logFile argument provided

No further code is executed after the auto_log helper function detects that no tag string has been provided, because it calls exit.

exercise 5

What output is produced when you call the testAutolog command with a tag string of "test1".

Solution

Executing:

~/workshop/step_02.sh testAutolog test1

produces this output:

.. logging to ./autoLog_test1.log

1) Call stdStreams with output and error text:
to standard output: 'text for standard output'
to standard error:  'text for standard error'

2) Capture echo output in a variable and display it:
   echo output was:
text for standard output

3) Call echo_se with error text:
text for standard error

4)Capture echo_se function output in a variable and display it:
text for standard error
echo_se output was: ''

Why does the $output variable contain the output text in 2) ?

Solution

Because backtick evaluation replaces the command in backticks ( `...` ) with the echo command's standard output.

Why is the $output variable empty in segment 4) when echo_se is called with output being captured?

Solution

Because only a command's standard output is returned by backtick evaluation, not standard error.

What log file is produced? What are its contents, and why?

Solution

The log file produced is autoLog_test1.log, written to the current directory in force when step_02.sh was called. This name is based on the logFileTag we specified as "test1".

Its contents (below) are nearly the same as when ~/workshop/step_02.sh testAutoLog test1 is called, except that the ".. logging to ./autoLog_test1.log" line is not reported, because that was written before automatic logging was started.

1) Call stdStreams with output and error text:
to standard output: 'text for standard output'
to standard error:  'text for standard error'

2) Capture echo output in a variable and display it:
   echo output was:
text for standard output

3) Call echo_se with error text:
text for standard error

4)Capture echo_se function output in a variable and display it:
text for standard error
echo_se output was: ''