Page Comparison

...

There are 3 local file systems available on any TACC compute cluster (stampede2, lonestar5 lonestar6, etc.), each with different characteristics. All these local file systems are very fast and set up for parallel I/O (Lustre file system).

On stampede2 lonestar6 these local file systems have the following characteristics:

	Home	Work2	Scratch
quota	10 GB	1024 GB = 1 TB	2+ PB (basically infinite)
policy	backed up	not backed up, not purged	not backed up, purged if not accessed recently (~10 days)
access command	cd	cdw2	cds
environment variable	$HOME	$WORK (different sub-directory for each cluster) $STOCKYARD (root of the shared Work file system) $WORK2 (different sub-directory for each cluster)	$SCRATCH
root file system	/home	/work2work	/scratch
use for	Small files such as scripts that you don't want to lose.	Medium-sized artifacts files you don't want to copy over all the time. For example, custom programs you install (these can get large), or annotation file used for analysis.	Large files accessed from batch jobs. Your starting files will be copied here from somewhere else, and your final results files will be copied elsewhere (e.g. stockyard, corral, or your BRCF POD), or your organization's storage area.

When you first login, the system gives you information about disk quota quotas and your compute allocation quota:balance in "SU" (system units).

Code Block

--------------------- Project balances for user abattenh ----------------------
| Name           Avail SUs     Expires | Name           Avail SUs     Expires |
| genomeAnalysisOTH21095       673  2021-03-31 | BioinformaticsResour 1001000  20202022-0609-30 | | UT-2015-05-18MCB21106        1000  2021-03-31 | DNAdenovo 995  2022-12-31 |
| OTH21164     4969  2021-03-31 | | CancerGenetics  1000    4856  2020-09-302022-12-31 | A-cm10OTH21037            4224  8867  20202022-12-31 |
------------------------ Disk quotas for user abattenh ------------------------
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /home1              0.0      1011.07     0.1000           52      153     10000000    0.0200 |
| /work2work             614169.50    1024.0    6016.0150        6109479362     3000000    2.04 |
| /scratch         2676.6       0.0     0.00        32442           0    0.00 |
--------65 |
-------------------------------------------------------------------------------------------

changing TACC file systems

When you first login, you start in your home Home directory. Use the cd, cdw2 and cds commands to change to your other file systems. Notice how your command prompt helpfully changes to show your location.

Code Block

language	bash
title	Changing file systems at TACC

cdw2cdw
cds
cd

Tip

The cd (change directory) command with no arguments takes you to your home directory on any Linux/Unix system.

The cdw2 cdw and cds commands are specific to the TACC environment.

...

TACC compute clusters now share a common Work file system called stockyard. So files in your Work area do not have to be copied, for example from to stampede2 to ls5 ls6 – they can be accessed directly from either cluster.

...

$STOCKYARD - This refers to the root of your shared Work area
- e.g. /work/01063/abattenh (should be changed to /work2/01063/abattenh soon)
$WORK or $WORK2 - Refers to a sub-directory of the shared Work area that is different for different clusters, e.g.:
- /work/01063/abattenh/lonestarls6 on ls5 ls6
- /work2/01063/abattenh/stampede2 on stampede2

...

Code Block

language	bash
title	The shared BioITeam directory

ls /work2work/projects/BioITeam

Files we will use in this course are in a sub-directory there. The $CORENGS environment variable set in your login profile refers to this path.

...

ranch is a gigantic (multiple PB) tape archive system where researchers can archive data. All TACC users have an automatice automatic 2 TB ranch allocation. UT researchers may request larger (multi-TB) ranch storage allocations through the normal TACC allocation request process.

...

Once that data is staged to the ranch disk it can be copied to other places. However, the ranch file system is not mounted as a local file system from the stampede2 or ls5 ls6 clusters. So remote copy commands are always needed to copy data to and from ranch (e.g. scp, rsync)..g. scp, rsync).

About file systems

File systems are storage areas where files and directories are arranged in a hierarchy. Any computer can have one or more file systems "mounted" (accessible as local storage). The df command can be used on any Unix system to show all the "top-level" mounted file systems. TACC has a lot of temporary file systems, so lets just look at the first 15 and tell df to use "human readable" size formatting with the -h option:

Code Block

language	bash

df -h | head -15

The rightmost "Mounted on" column give the top-level access path. Find /home1, /work, and /scratch and note their sizes!

What do we mean by "hierarchy"? It is like a tree, with the root file system as the trunk, sub-directories as branches, sub-sub-directories as branches from branches (and so forth), with files as leaves off any branch.

your TACC Home directory
using home-directory-relative (~) path syntax

Image Added

But everyone has a Home directory, so you must only be seeing a part of the Home directory hierarchy. To see the absolute path of a directory you're in, use the pwd -P command. Note that absolute paths always start with a forward slash ( / ) denoting its root file system.

Code Block

language	bash

pwd -P
# will show something like this
# /home1/01063/abattenh

That shows you that your Home directory (~) is actually 3 levels down in the /home1 hierarchy:

part of the TACC Home file system
using absolute paths

Image Added

Staging your data

So, your sequencing center has some data for you. They may send you a list of web or FTP links to use to download the data.

...

original – for original sequencing data (compressed FASTQ files)
- sub-directories named, for example, by year_month.<project_<sequencing run/job or project name>
aligned – for alignment artifacts data (BAM files, etc)
- sub-directories named, e.g., by year_month.<project_name>
analysis – further downstream analysis
- reasonably named sub-directories, often by project
refs – reference genomes and other annotation files used in alignment and analysis
- sub-directories for different reference genomes and aligners
- e.g. ucsc/hg38/hg19star, ucsc/sacCer3/bwa, mirbase/v20/bowtie2
code – for scripts and programs you and others in your organization write
- ideally maintained in a version control system such as git, subversion or cvs.
- easiest to name can have separate sub-directories for people, or various shared repositories.

Download from a link – wget

...

The cp command copies one or more files from a local source to a local destination. It has many options, but the most common form is:

cp [options] <source file 1> <source file 2> ... <destination directory>/

...

Here's a fun scavenger hunt for more practice . Issue of the following commands to get practice what you've learned so far:.

Expand

title	Hint

Hit Tab Tab as much as possible to save typing!

...

Code Block

title	Play a scavenger hunt for more practice

cd
cp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/what what
# or using the $CORENGS environment variable
cp -r $CORENGS/linuxpractice/what what
cd what
cat readme

Where are you when you're all done?

Expand

title	Answer

stamp2ls6:~/what/starts/here/changes/the/world

...

Expand

title	Step 1 answer

From inside your ~/what directory:

Code Block

title	Play a scavenger hunt for more practice

mkdir starts
cd starts
cp /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/steps/nextInstr .
cat nextInst

...

Expand

title	Step 3 answer

From inside your ~/what/starts/here directory:

Code Block

title	Play a scavenger hunt for more practice

scp -r /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# or
rsync -ptrvP /work2work/projects/BioITeam/projects/courses/Core_NGS_Tools/linuxpractice/changes/ changes/
# Note: rsync -avP ... will also work, but will report an error because the destination file and
# directory ownership cannot be changed to match the source. But the files will be copied, and
# ownership assigned to you.
 
# Then
cd changes 
more largeFile.txt

...

Versions Compared

Old Version 30

New Version 31

Key

changing TACC file systems

About file systems

Staging your data

Download from a link – wget