Versions Compared

Old Version 31

changes.mady.by.user Anna Battenhouse

Saved on

compared with

New Version Current

changes.mady.by.user Anna Battenhouse

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Commands, options, arguments & command line input

The Intro Unix: The Bash shell and commands section introduced the bash  REPL – a Read, Eval, Print Loop that processes lines of command input. To review, a command consists of:

...

  • Short (1-character) options which can be provided separately, prefixed by a single dash ( - )
    • or can be combined with the combination prefixed by a single dash (e.g. ls -lah)
  • Long (multi-character/"word") options are prefixed with a double dash ( -- ) and must be supplied separately.
  • Many utilities have equivalent long and short options, both of which can have values.
    • The short option and its value are usually separated by a space, but can also be run together (e.g. -f 2 or -f2)
    • Strictly speaking, the long option and its value should be separated by an equal sign (=) according to the POSIX standard (see https://en.wikipedia.org/wiki/POSIX). But many programs let you use a space as separator also.
  • Options usually come before arguments, but may also be allowed after the arguments depending on the tool.

More at: Intro Unix: The Bash shell and commands: Command options

Getting help

To learn what options and arguments a command has:

  1. In the Terminal, type in the command name then the --help long option (e.g. ls --help)
    • Works for most Linux commands;
      • 3rd party tools may use -h or -? or even /? instead
    • May produce a lot of output, so you may need to scroll up quite a bit or pipe the output to a pager (
      • e.g. ls --help | more
      )
  2. Use the built-in manual system (e.g. type man ls)
    • This system uses the less pager (space advances the output by one screen/"page"; typing q exits the display)
  3. Ask the Google, e.g. search for ls man page
    • Can be easier to read
  4. Consult our Intro Unix: Some Linux commands wiki page
    • It lists many useful Linux commands along with some of their commonly used options

More at:

Literal characters and metacharacters

...

We'll be emphasizing the different metacharacters and their usages – which can depend on the context where they're used – both in the bash command line and in commands/programs called from bash.

More at:

Command line history and editing

...

For how to edit text on the command line, see: Intro Unix: The Bash shell and commands: Command line history and editing

Tab key completion

Hitting Tab when entering command line text invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. It's almost magic!

...

Code Block
languagebash
more jabberwocky.txt
cat jabberwocky.txt | more

Let's dissect the difference in detailThe main differences between these two commands:

  • In

...

  • more jaberwocky.txt, it is the more command that reads

...

  • the

...

  • then writes some output to standard output, which is displayed on your Terminal
  • it pauses at page boundaries (--More--) waiting for input on standard input
  • when it receives a space on on standard input it reads more input from jabberwocky.txt
  • then writes the output to standard output, which is displayed on your Terminal

...

  • file and performs its operations
    • displays some data on standard output, waits for a space on standard input, repeat until no more file data
    • since more is reading the file, it can display progress information about how much data has been read
  • In cat jabberwocky.txt | more, the cat program reads the file data and writes it to its standard output.
    • the pipe operator ( | ) then connects the standard output from cat to standard input of the more command
    • the more command then reads its input from standard input, instead of from a file

      ...

      • this "write until block" / "read when input wanted & available" behavior makes streams a very efficient means of inter-process communication.
          • and causes the cat program to block – stop writing data to its standard output until requested
        • more displays some data on standard output, waits for a space on standard input, then requests more input
          • repeat until no more data from cat.
        • since the data coming in to more is from an anonymous pipe, more cannot display progress information

      More at: Intro Unix: Viewing text in files: Standard streams and piping

      echo, head, tail, cat -n, wc

      ...

      • With no options, head shows the first 10 lines of its input and tail shows the last 10 lines.
        • You can use the -n option to specify how many lines to view, or just put the number you want after a dash (e.g. head -5 for 5 lines or head -1 for 1 line).
      • To start viewing lines at a particular line number, use tail and put a plus sign (+) in front of the number (with or without the -n option).
      • The cat -n option adds line numbers to the text it displays, which can help orient you when dealing with large files

      Use the wc (word count) command to count text lines (wc -l) or characters (wc -c).

      echo is the bash command to output text.

      • echo -e says to enable interpretation of backslash escapes
        • so, for example, \n is interpreted as a linefeed, and \t as a tab character
      • echo -n says don't output the trailing newline (linefeed) character

      ...

      Code Block
      languagebash
      head -n 5 haiku.txt                      # display the 1st 5 lines of "haiku.txt"  
cat -n haiku.txt                         # display "haiku.txt" contents with line numbers        
cat -n haiku.txt | tail -n 7             # display the last 7 lines of "haiku.txt"
cat -n haiku.txt | tail -n +7            # display text in "haiku.txt" starting at line 7
cat -n haiku.txt | tail -n +5 | head -3  # display the middle stanza of "haiku.txt"

wc -l haiku.txt                          # count lines in "haiku.txt" file
cat haiku.txt | wc -l                    # use wc -l to count lines of piped-in text

echo 'Hello world!' | wc -c              # count characters output by echo, including the trailing newline
echo -n 'Hello world' ! wc -c            # count characters output by echo, without the trailing newline

echo -e "aa\nbb\ncc"                     # output 3 lines of text using \n to represent newlines

      More at:

      Other shell concepts

      Environment variables

      Environment variables are just like variables in a programming language (in fact bash is a complete programming language): they are "names" that hold a value assigned to them. As with all programming language variables, they have two operations:

      ...

      In bash, you define (set/assign)an environment variable like this:

      Code Block
      languagebash
      varname='hello world!'  # Assign the environment variable named "varname" the value "hello world!"
varname='hello world!'


      Tip
      • Do not put spaces around the equals sign when assigning environment variable values! The shell is very picky about this.
      • Always enclose environment variable values that contain spaces in single or double quotes (see below)
      • Variable names can only contain alphnumeric (A-Z, a-z, 0-9) and underscore ( _ ) characters, and must begin with a letter.

      ...

      Code Block
      languagebash
      foo="My USER name is $USER";  echo $foo   # The textvariable "$USER" is evaluated and its value substituted
foo="My USER name is ${USER}; echo $foo   # Same as above using longer evaluation syntax
foo='My
USER# nameUndefined is $USER';  echo $foo   # The text "$USER" is left as-is

      Your built-in environment variables (e.g. $USER, $MY_GROUP, $PATH) and their values can be viewed with the env command.

      More at: Intro Unix: Writing text: Environment variables

      Quoting in the shell

      When the shell processes a command line, it first parses the text into tokens ("words"), which are groups of characters separated by whitespace (one or more space characters). Quoting affects how this parsing happens, including how metacharacters are treated and how text is grouped.

      There are three types of quoting in the shell:

      ...

      • it groups together all text inside the quotes into a single token
      • it tells the shell not to "look inside" the quotes to perform any evaluation
        • all metacharacters inside the single quotes are ignored
        • in particular, any environment variables in single-quoted text are not evaluated

      ...

      environment variables just appear as empty text
bar='chess'; echo "Today's game is: $bar"
unset bar;   echo "Today's game is: $bar"

# Evaluating an environment variable that contains an underscore 
# may need to use the longer evaluation syntax, if the literal text
# before or after it is an underscore.
my_var="middle"
echo "File name is: foo_${my_var}_bar.txt"

      Your built-in environment variables (e.g. $USER, $MY_GROUP, $PATH) and their values can be viewed with the env command.

      More at: Intro Unix: Writing text: Environment variables

      Quoting in the shell

      When the shell processes a command line, it first parses the text into tokens ("words"), which are groups of characters separated by whitespace (one or more space characters). Quoting affects how this parsing happens, including how metacharacters are treated and how text is grouped.

      There are three types of quoting in the shell:

      1. single quoting (e.g. 'some text') – serves two purposes
        • it groups together all text inside the quotes into a single token
        • it allows environment variable evaluation, but inhibits some metacharcters
          • e.g. asterisk ( * ) pathname globbing (more on globbing later...) and some other metacharacters
          double quoting also preserves any special characters present in the text e.g. tells the shell not to "look inside" the quotes to perform any evaluation
          • all metacharacters inside the single quotes are ignored
          • in particular, any environment variables in single-quoted text are not evaluated
      2. double quoting (e.g. "some text") – also serves two purposes
        • it groups together all text inside the quotes into a single token
        • it allows environment variable evaluation, but inhibits some metacharcters
          • e.g. asterisk ( * ) pathname globbing and some other metacharacters
        • double quoting also preserves any special characters present in the text
          • e.g. newlines (\n) or Tabs (\t)
      3. backtick quoting (e.g. `date`)
        • evaluates the expression inside the backtick marks ( ` )
        • the standard output of the expression replaces the text inside the backtick marks ( ` )
        • note that the syntax $(<expression>) is equivalent to `<expression>`

      Note that the quote characters themselves ( '  "  ` ) are metacharacters that tell the shell to "start a quoting process" then "end a quoting process" when the matching quote is found. Since they are part of the processing, the enclosing quotes are not included in the output.Single

      Tip

      Always use single ( ' ) or double ( " ) quotes when you define an environment variable whose value contains spaces so that the shell sees the quoted text as one item.

      Single vs double quotes examples:

      Code Block
      languagebash
      foo=echo "My USERUnix namegroup is $USER"; echo $foo$MY_GROUP"         # The text "$USER$MY_GROUP" is evaluated and its value substituted foo='My USER
nameecho is $USER';My echoUnix $foogroup    is $MY_GROUP'     # The text "$USER$MY_GROUP" is left as-is

FOOfoo="Hello world!"My USER name is $USER"; echo "The$foo  value of variable 'FOO' is \"$FOO\""  # EscapeThe thetext double"$USER" quotesis insideevaluated doubleand quotesits echovalue "The value of variable 'FOO' is"' "$FOO"'  # Single-quoted text after double-quoted text

      Backtick evaluation quoting examples:

      Code Block
      languagebash
      datesubstituted
foo='My USER name is $USER'; echo $foo         # The text "$USER" is left #as-is
Calling
the date command just displays date/time information
echo date     # Here "date" is treated as a literal word, and written to output
echo `date`   # The date command is evaluated and its output replaces the command

today=$( date );          echo $today  # environment variable "today" is assigned today's date
today="Today is: `date`"; echo $today  # "today" is assigned a string including today's date

      Redirection

      So far text we've been working with output to standard output, which I keep reminding you is mapped to your Terminal. But you can redirect text elsewhere.

      Recall the three standard Unix streams: they each have a number, a name and redirection syntax:

      Image Removed

      • standard output is stream 1
        • redirect standard output to a file with a the > or 1> operator
          • a single > or 1> overwrites any existing data in the target file
          • a double >> or 1>> appends to any existing data in the target file
      • standard error is stream 2
        • redirect standard error to a file with a the 2> operator
          • a single 2> overwrites any existing data in the target file
          • a double 2>> appends to any existing data in the target file

      If you want output to go to both the Terminal and a file, you can use the tee command (or tee -a to append)

      More at: Intro Unix: Writing text: Redirection

      Errors, output and their streams

      Any time a bash command encounters an error, diagnostic error information is written to standard error, not to standard output!

      When executing commands you will want to manipulate standard output and standard error appropriately – especially for 3rd party programs.

      You can see that error and output streams are different by directing one or the other to the /dev/null "global trash can"

      Code Block
      languagebash
      ls haiku.txt xxx.txt                # displays both output and error text on the Terminal
ls haiku.txt xxx.txt 2>/dev/null    # displays only output text on the Terminal
ls haiku.txt xxx.txt 1>/dev/null    # displays only error text on the Terminal

# And this syntax sends standard output to outerr.log
# and standard error to the same place as standard out (2>&1)
# So data from both standard output and standard error
# will be written to outerr.log
ls haiku.txt xxx.txt 1>outerr.log 2>&1

      More at:

      ...

      FOO="Hello world!"
echo "The value of variable 'FOO' is \"$FOO\""  # Escape the double quotes inside double quotes


      Tip

      If you see the greater than ( > ) character after pressing Enter, it can mean that your quotes are not paired, and the shell is waiting for more input to contain the missing quote of the pair (either single or double). Just use Ctrl-c to get back to the command prompt.

      Backtick evaluation quoting examples:

      Code Block
      languagebash
      date          # Calling the date command just displays date/time information
echo date     # Here "date" is treated as a literal word, and written to output
echo `date`   # The date command is evaluated and its output replaces the command

today=$( date );          echo $today  # environment variable "today" is assigned today's date
today="Today is: `date`"; echo $today  # "today" is assigned a string including today's date

      More at: Intro Unix: Writing text: Quoting in the shell

      Redirection

      So far text we've been working with output to standard output, which I keep reminding you is mapped to your Terminal. But you can redirect text elsewhere.

      Recall the three standard Unix streams: they each have a number, a name and redirection syntax:

      Image Added

      • standard output is stream 1
        • redirect standard output to a file with a the > or 1> operator
          • a single > or 1> overwrites any existing data in the target file
          • a double >> or 1>> appends to any existing data in the target file
      • standard error is stream 2
        • redirect standard error to a file with a the 2> operator
          • a single 2> overwrites any existing data in the target file
          • a double 2>> appends to any existing data in the target file

      If you want output to go to both the Terminal and a file, you can use the tee command (or tee -a to append).

      Note that the > redirection metacharacter sends its output to a file, not to another program's standard input stream as with the | pipe metacharacter. (There are some cases where redirection involves something other than a file, but that's a topic for the Advanced Bash scripting class.)

      More at: Intro Unix: Writing text: Redirection

      Errors, output and their streams

      Any time a bash command encounters an error, diagnostic error information is written to standard error, not to standard output!

      When executing commands you will want to manipulate standard output and standard error appropriately – especially for 3rd party programs.

      You can see that error and output streams are different by directing one or the other to the /dev/null "global trash can"

      Code Block
      languagebash
      ls haiku.txt xxx.txt                # displays both output and error text on the Terminal
ls haiku.txt xxx.txt 2>/dev/null    # displays only output text on the Terminal
ls haiku.txt xxx.txt 1>/dev/null    # displays only error text on the Terminal

# And this syntax (2>&1) sends standard output to outerr.log and standard error to the
# same place as standard out. So data from both standard output and standard error
# will be written to outerr.log
ls haiku.txt xxx.txt 1>outerr.log 2>&1

      More at:

      File systems, files, and file manipulation

      Let's review Intro Unix: Files and File Systems. The most important takeaways are: