Page Comparison

About File Systems 

So far all the files we've been dealing with have been in your Home directory, which is your personal directory. All Unix systems will provide you with a Home directory of your own.

Files and directories in your Home directory are part of the overall file system hierarchy – a tree of directories and their files/sub-directories.

In the file system tree, directories are the branches and files are the leaves.

To see the partial file system hierarchy of your Home directory, you can use the tree command. (Note that tree is not always available on Linux systems; it is an add-on tool that must be installed separately).

Just calling tree will produce output similar to something like this:

Image RemovedImage Added

This display shows the hierarchical structure of files and sub-directories. At top-level, there are a number of files that we pre-positioned for you (haiku.txt, mobydick.txt, etc.) along with some we created (out.txt, newfile.txt, etc.). There are also two sub-directories: bedfiles data and docs, each with several files in them. And notice how tree colors the directories differently to make them stand out.

Of course there are other areas of the file system hierarchy a tree of. For example, the /stor/work/CBRS_unix directory.

...

Produces output similar to this:

Image RemovedImage Added

There are three sub-directories under the /stor/work/CBRS_unix directory: fastq, test and unix, each with their own sub-directories and files. Again, notice how tree colors file with the extension .gz differently. These are FASTQ files, compressed using the gzip protocol, produced by a Next Generation Sequencing (NGS) run in our Genome Sequencing and Analysis (GSAF) core facility.

Note that in Unix, and on Macs, directories are separated by forward slash ( / ) characters, unlike Windows where the backslash ( \ ) is used. And the root of all the file systems is that 1st forward slash ( / ).

...

Navigating the file

...

So if you're on a new system, how do you know what file systems are available to you? The df (disk free) command will list all the available file systems. As its name suggests, it also shows file system sizes, so it's always good to use df -h (human readable) to see sizes in more readable form. Also, there can be many many file systems available on any given system, so always pipe the output to more.

df -h | more

This produces a rather busy display like this one:

Image Removed

Fortunately, you can ignore most of it. Focus on the Mounted on and Size columns.

• / (forward slash), under the Mounted on column:
  • / (forward slash) is the root of the file system where the operating system is installed
  • Note its Size is 98G with 47G used
• Look for Mounted on entries with large Size numbers:
  • Gigabytes (10^{^9} bytes), Terabytes (10^{^12} bytes), Petabytes (10^{^15} bytes)
• Ignore file systems with names like /run, /dev, /snap, /sys, /boot, /tmp, /var – these are system related
• Here we see a number starting with /stor (/stor, /stor/home, /stor/work, etc.)
  • Note its large Size: 39T - a sign that it's a file system you want to know about.

Here's a similar listing from the Lonestar5 compute cluster at TACC:

Image Removed

Here the big important file systems are /home1 (7.0T), /scratch (8.1P) and /work (6.8P). There's also /admin (3.5T) but its name suggests that normal users won't be able to access it.

Navigating the file system

Now that we know there are other places, how do we get there? Enter the cd (change directory) command:

• cd <optional directory_name>
  • with no argument, always changes to your Home directory.

There are also some "special" built-in directory names:

• ~ (tilde) means my Home directory
• . (single period) means the current directory
• .. (two periods) means the parent of the current directory (directory above it)
  
  • So ls .. means "list contents of the parent directory"
• - (dash) means whatever directory you were in before this one

So these two expressions do the same thing – take you to your Home directory from wherever you are in the file system.

cd
cd ~

When you've changed into a directory, how do you know where you are in the file system?

• On our system, we've arranged that the command prompt "follows you" around the file system.
• On other systems you can use the pwd (present working directory) command to see where you are

Exercise 4-1

Change into the /stor/work/CBRS_unix directory then back to your Home directory. Then use the - (dash) to go back to /stor/work/CBRS_unix without typing it in, then Home again using ~ (tilde). How does the command prompt change?

...

...

...

system

Now that we know there are other places, how do we get there? Enter the cd (change directory) command:

• cd <optional directory_name>
  • with no argument, always changes to your Home directory.
  • the special character ~ (tilde) means your Home directory.
    

cd       # make sure you're in your Home directory
cd data  # change into the data sub-directory
cd ~     # return to your Home directory using ~
cd data  # change into the data sub-directory again
cd       # return to Home

When you've changed into a directory, how do you know where you are in the file system?

• On our system, we've arranged that the command prompt "follows you" around the file system.
• On other systems you can use the pwd (present working directory) command to see the full pathname of the directory you're in

There are other "special" built-in directory names:

• . (single period) means the current directory
• .. (two periods) means the parent of the current directory (directory above it)
  
  • So ls .. means "list contents of the parent directory"
• - (dash) means whatever directory you were in before this one

Exercise 4-1

What is the full pathname of your Home directory?

Change into the /stor/work/CBRS_unix directory then back to your Home directory. Then use the - (dash) to go back to /stor/work/CBRS_unix without typing it in, then Home again using ~ (tilde). How does the command prompt change?

cd /stor/work/CBRS_unix
cd                          # Back home
cd -                        # Last directory (here /stor/work/CBRS_unix)
cd ~                        # Back home
cd -                        # Last directory (here And home again

Tab key completion

The Tab key is your best friend! Hit the Tab key once or twice - it's almost always magic!

Hitting Tab invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. On most modern Linux shells you use Tab completion by pressing:

• single Tab – completes file or directory name up to any ambiguous part
  • if nothing shows up, there is no unambiguous match
• Tab twice – display all possible completions
  
  • you then decide where to go next

Let's have some fun with the Tab key. Follow along if you can:

Relative pathname syntax

An absolute pathname lists all components of the full file system hierarchy that describes a file.

Absolute paths always start with  / (the forward slash), which is the root of the file system hierarchy. For example /stor/work/CBRS_unix/unix/haiku.txt is an absolute path.

Rather than always typing an absolute pathname of where you want to go, you can specify a directory relative to where you are. This is where the special directory names . (single period =  current directory) and .. (double period =  parent of current directory) come in handy.

Here are some examples. Follow along if you can – and use the Tab key to help out! (See Tab key completion)

cd                      # hitmake Tabsure keyyou're -in expandsyour toHome /stor/directory
whichpwd is the only match cd /stor/w               # hitwhere Tabam keyI again - expands tonow, really? /stor/workhome/,student01
againcd the.. only match cd /stor/work/C          # hit Tab once - nothing# happensup becauseone therelevel are- multiplenow matchesI'm cdin /stor/home
cd ../work/CCBRS_unix/   # change into /stor/work/CBRS_unix using relative  # hit Tab a 2nd time - all matching directories listed
cd /stor/work/CB  syntax
cd ./unix/docs/         # hitchange into Tabthe keyunix/docs sub-directory expandsrelative to /stor/work/CBRS_unix
cd ../stor../work/CBRS_unixfastq/         # go pressup Entertwo tolevels, changethen tointo thatfastq directory
cd ../../../home/       # change to the /stor/home directory using relative syntax
cd          # go Home

Relative pathname syntax

Rather than always typing a full pathname (that is, a pathname starting with the / root directory) of where you want to go, you can specify a directory relative to where you are. This is where the special directory names . (single period =  current directory) and .. (double period =  parent of current directory) come in handy.

Here are some examples. Follow along if you can – and use the Tab key to help out!

Expand
title Structure of the /stor/work/CBRS_unix directory
Image Removed # go Home

Pathname wildcards ("globbing")

Since we've seen that a goal of Unix is to type as little as possible, there are several metacharacters that serve as wildcards to represent sets of characters when typing file names. Using these metacharacters is called globbing (don't ask me why ) and the pattern is called a glob.

• asterisk ( * ) is the most common filename wildcard. It matches any length of any characters.
• brackets ( [ ] ) match any character between the brackets.
  • and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])
• braces ( {  } ) enclose a list of comma-separated strings to match (e.g. {dog,pony})

And wildcards can be combined. Some examples:

pwdls *.txt        # lists all files with names ending in ".txt"
ls [a-z]*.txt   # wheredoes amthe Isame now,but really? /stor/home/student01
cd ..      only lists files starting with a lowercase letter
ls [ABhi]*      # lists all filenames whose 1st letter is #A, upB, oneh, levelor -i
now I'm in /stor/home
cd ../work/CBRS_unix/   # change into /stor/work/CBRS_unix using relative syntax
cd ./unix/docs/         # change into the unix/docs sub-directory relative to /stor/work/CBRS_unix
cd ../../fastq/         # go up two levels, then into fastq directory
cd ../../../home/       # change to the /stor/home directory using relative syntax
cd                      # go Home

Pathname wildcards ("globbing")

Since we've seen that a goal of Unix is to type as little as possible, there are several metacharacters that serve as wildcards to represent sets of characters when typing file names. Using these metacharacters is called globbing (don't ask me why ) and the pattern is called a glob.

• asterisk ( * ) is the most common filename wildcard. It matches any length of any characters.
• brackets ( [ ] ) match any character between the brackets.
  • and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])
• braces ( {  } ) enclose a list of comma-separated strings to match (e.g. {dog,pony})

And wildcards can be combined. Some examples:

ls *.txt        # lists all files with names ending in ".txt"
ls [a-z]*.txt   # does the same but only lists files starting with a lowercase letter
ls [ABhi]*      # lists all filenames whose 1st letter is A, B, h, or i
ls *.{txt,tsv}  # lists filenames ending in either .txt or .tsv

Exercise 4-2

Design a wildcard that will match the files haiku.txt and mobydick.txt but not jabberwocky.txt in your Home directory.

...

There are always multiple ways of doing things in Unix. Here are two possible answers:

ls [hm]*.txt
ls {haiku,jabberwocky}.txt 

File attributes

...

ls *.{txt,tsv}  # lists filenames ending in either .txt or .tsv

Exercise 4-2

Design a wildcard that will match the files haiku.txt and mobydick.txt but not jabberwocky.txt in your Home directory.

ls [hm]*.txt
ls {haiku,jabberwocky}.txt 

Finding file systems

So if you're on a new system, how do you know what file systems are available to you? The df (disk free) command will list all the available file systems. As its name suggests, it also shows file system sizes, so it's always good to use df -h (human readable) to see sizes in more readable form. Also, there can be many many file systems available on any given system, so always pipe the output to more.

df -h | more

This produces a rather busy display like this one:

Image Added

Fortunately, you can ignore most of it. Focus on the Mounted on and Size columns.

• Look for the / (forward slash), under the Mounted on column:
  • / (forward slash) is the root of the file system where the operating system is installed
  • Note its Size is 98G with 61G used
• Look for Mounted on entries with large Size numbers:
  • Gigabytes (10^{^9} bytes), Terabytes (10^{^12} bytes), Petabytes (10^{^15} bytes)
• Ignore file systems with names like /run, /dev, /snap, /sys, /boot, /tmp, /var – these are system related
• Here we see a number starting with /stor (/stor, /stor/home, /stor/work, etc.)
  • Note its large Size: 39T - a sign that it's a file system you want to know about.

Here's a similar listing from the Lonestar6 compute cluster at TACC:

Image Added

Here the big important file systems are /home1 (7.0T), /scratch (8.1P) and /work (6.8P). There's also /admin (3.5T) but its name suggests that normal users won't be able to access it.

File attributes

Let's revisit a listing of our Home directory contents. Before we created any files, an ls -l listing looked something like this:

Image RemovedImage Added

Ignoring that "total 77951" line, there are 9 whitespace-separatedcolumns in this long listing:

1. file permissions - a 10-character field
2. number of sub-directories or symbolic linkscomponents associated with a directory - rarely important
3. account name of the file owner
4. Unix group associated with the file
5. file size
6. last modification month
7. last modification day
8. last modification year, or last modification hour/:minute if within the last year
9. file name

Notice I call everything a file, even directories. That's because directories are just a special kind of file – one that contains information about the directory's contents.

Owner and Group

A file's owner is the Unix account that created the file (here student01, me). That account belongs to one or more Unix groups, and the primary group associated with a file is listed in field 4.

The owner will always be a member of the group associated with a file, and other accounts may also be members of the same group. The Unix group for our class is CCBB_Workshops_1. (To see the Unix groups you belong to, just type the groups command.)

Permissions

File permissions ( and some information about the file type) are encoded in that 1st 10-character field. Permissions govern who can access a file, and what actions they are allowed.

• character 1 describes the file type (
  • d for directory
  ,
  • - (dash) for regular file
  ,
  • l for symbolic link
  )
• the remaining 9 characters are 3 sets of 3-character designations
  • characters 2-4: what the owning user account can do
  • characters 5-7: what other members of the associated Unix group can do
  • characters 8-19: what other non-group members (everyone) can do

Each of the 3-character sets describes if read ( r ) write ( w or s ) ) and execute ( x or s ) actions are allowed , or not allowed ( - ).

• read ( r ) access means file contents can be read, and it can be copied
• write ( w or s ) access means a file's contents can be changed, and directory contents can be modified (files added or deleted)
• execute ( x or s )
  • For for files, execute ( x ) means it is a program that can be called/executed
    • e.g. /usr/bin/ls, the file that performs the ls command
  • for directories, execute ( e.g. /usr/bin/ls, the file that performs the ls command)For directories, execute ( x ) means directory operations may be performed/executedthe directorycan be listed and changed into

Examples:

ls -l ~/haiku.txt

...

...

• dash ( - ) in position one signifies this is a regular file
• rw- for owner allows read and write access
• r-- for group permits only read access
• --- for everyone means no access allowed

ls -l /usr/bin/ls

...

...

• /usr/bin/ls is the program that performs the ls command
  • root (the master admin account) is the owner, in the root group
• dash ( - ) in position one signifies this is a regular file
• rwx for owner allows read, write and execute
• r-x for group permits read and execute
• r-x for everyone permits read and execute

ls -ld ~/docs (-d says to list directory information, not directory contents)

...

...

• • x ) means directory operations may be performed/executed
    • the directorycan be listed and changed into

Examples:

ls -l ~/haiku.txt

ls -l /usr/bin/ls

ls -l -d ~/docs (-d says to list directory information, not directory contents)

Exercise 4-3

What are the directory permissions and ownership of /stor/work/CBRS_unix/test? Can you list its contents? Why or why not?

What are the directory permissions and ownership of /stor/work/CBRS_unix/test/anna_only? Can you list its contents? Why or why not?

Basic file manipulation commands

The basic file manipulation commands are:

• touch to create a new, empty file or update the last modification date on an existing one
• mkdir to create a new directory
• cp to copy files (locally, within the current host computer)
• mv to move a file from one name to another (so rename the file)
• rm to remove (permanently delete!) files and directories

Let's practice:

cd                    # make sure you're in your Home directory
mkdir tmp             # create a new directory called "tmp" and change into it
cd ./tmp              # the "./" syntax is optional; means relative to current directory
touch f1.txt          # create empty file called "f1.txt"
ls -l >> f1.txt       # add some text by redirecting (appending) output from ls
mv f1.txt f2.txt      # rename the "f1.txt" file to "f2.txt"
cp f2.txt f3.txt      # copy "f2.txt" to a new file called "f3.txt" in this directory

mkdir dir2            # make a new "dir2" sub-directory
mv *.txt dir2/        # move the "f2.txt" and "f3.txt" to the "dir2" sub-directory
cp dir2/*.txt .       # copy all the *.txt files in the "dir2" directory to this directory  
rm f*.txt             # delete all f*.txt files in the current directory

cp -r dir2/ dir3/     # copy the "dir2" directory and its contents to a new "dir3" directory
rm dir3/f3.txt        # delete the "f3.txt" file from the "dir3" directory
rm -rf dir2           # delete the "dir2" sub-directory and its contents

Notice:

• Both cp and rm have a -r (recursive) option that is used when operating on directories.
• rm has -f (force) option that does not report an error if the file or directory doesn't exist.
• mkdir has a -p (parent) option that does not report an error if a component of a directory path, doesn't exist.
• It is a good idea to specify directory names with a trailing slash ( / ) to indicate that they are directories
• The "./" syntax means "relative to the current directory". It's optional here, but explicit.

See the Create, rename, linkto, delete files and Copying files and directories sections of the Some Linux commands page for more information.