...
- wget – retrieves the contents of an Internet URL
- cp – copies directories or files located on any local file system
- scp – copies directories or files to/from a remote system
- rsync – copies directories or files on either local or remote systems
...
There are 3 local file systems available on any TACC compute cluster (stampede2, lonestar6, etc.), each with different characteristics. All these local file systems and your account has a directory in each of the three.
The 3 file local systems have different characteristics, but all are fast and set up for parallel I/O.
...
| Home | Work | Scratch | |
|---|---|---|---|
| quota | 10 GB | 1024 GB = 1 TB | 2+ PB (basically infinite) |
| policy | backed up | not backed up, not purged | not backed up, purged if not accessed recently (~10 days) |
| access command | cd | cdw | cds |
| environment variable | $HOME | $WORK (different sub-directory for each cluster) $STOCKYARD (root of the shared Work file system) | $SCRATCH |
| root file system | /home | /work | /scratch |
| use for | Small files such as scripts that you don't want to lose. | Medium-sized files you don't want to copy over all the time. For example, custom programs you install (these can get large), or annotation file used for analysis. | Large files accessed from batch jobs. Your starting files will be copied here from somewhere else, and your final results files will be copied elsewhere (e.g. stockyard, corral, your BRCF POD, or your organization's storage area. |
When you first login, the system gives you information about disk quotas and your compute allocation balance in "SU" (system units).
| Code Block |
|---|
--------------------- Project balances for user abattenh ---------------------- | Name Avail SUs Expires | Name Avail SUs Expires | | OTH21095 1000 905 20222023-09-30 | MCB21106 1496 995 20222023-1209-3130 | | OTH21164 1000215 20222024-1205-31 | OTH21180 OTH21037 4224899 20222024-1203-31 | ------------------------ Disk quotas for user abattenh ------------------------ | Disk Usage (GB) Limit %Used File Usage Limit %Used | | /home1 scratch 0.07 110.70 0.00 52567 0 0.00 | | /home1 0.0 11.7 0.01 232 0 0.00 | | /work 169.0 1024.0 16.50 7936279361 3000000 2.65 | ------------------------------------------------------------------------------- |
changing TACC file systems
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| cdw
cds
cd
| |||||
cdw # cd $WORK
cds # cd $SCRATCH
cd # cd $HOME |
| Tip |
|---|
The cd (change directory) command with no arguments takes you to your home Home directory on any Linux/Unix system. The cdw and cds commands are specific to the TACC environment. |
...
File systems are storage areas where files and directories are arranged in a hierarchy. Any computer can have one or more file systems " mounted" (accessible as local storage). The df command can be used on any Unix system to show all the "top-level" mounted file systems. TACC has a lot of temporary file systems, so lets just look at the first 15 and tell df to use "human readable" size formatting with the -h option:
| Code Block | ||
|---|---|---|
| ||
df -h | head -15 |
The rightmost "Mounted on" column give the top-level access path. Find /home1, /work, and /scratch and note their sizes Size numbers!
What do we mean by "hierarchy"? It is like a tree, with the root file system (denoted by the leading / ) as the trunk, sub-directories as branches, sub-sub-directories as branches from branches (and so forth), with files as leaves off any branch.
...
But everyone has a Home directory, so you must only be seeing a part of the Home directory hierarchy. To see the absolute path of a directory you're in, use the pwd -P command. Note that absolute paths always start with a forward slash ( / ) denoting its the root file system.
| Code Block | ||
|---|---|---|
| ||
pwd -P # will show something like this # /home1/01063/abattenh |
...
Here's a depiction of the three file systems as seen from your Home directory ( ~ ), showing where the path-valued environment variables represent, and where the three symbolic links (~/CoreNGS, ~/scratch, ~/work) you created in your Home directory point. Notice that both the Work and Scratch file systems have a top-level hierarchy like we saw in Home above.
On many Linux systems, you can use the tree command to view the full file system hierarchy starting from a specified directory:
Staging your data
...