| Table of Contents |
|---|
Files and File systems
First, let's review Intro Unix: Files and File Systems. The most important takeaways are:
- Understanding the tree-like structure of directories and files in the file system hierarchy
- Absolute paths start with a slash ( / ), the root of the file system hierarchy
- More at: Intro Unix: Files and File Systems: The file system hierarchy
- Absolute paths start with a slash ( / ), the root of the file system hierarchy
- Knowing how to navigate the file system using the cd (change directory) command, Tab key completion, and relative path syntax:
- use the dot ( . ) metacharacter for the current directory
- use the dot-dot ( .. ) metacharacters for the parent directory
- More at:
- Selecting multiple files using pathname wildcards (a.k.a. "globbing")
- asterisk ( * ) to match any length of characters
- brackets ( [ ] ) match any character between the brackets, including hyphen ( - ) delimited character ranges such as [A-G]
- braces ( { } ) enclose a list of comma-separated strings to match (e.g. {dog,pony})
- More at: Intro Unix: Files and File Systems: Pathname wildcards
- A basic understanding of file attributes such as
- file type (file, directory)
- owner and group
- permissions (read, write, execute) for the owner, group and everyone
- More at: Intro Unix: Files and File Systems: File attributes
- Familiarly with basic file manipulation commands (mkdir, cp, mv, rm)
Working with remote files
scp (secure copy)
The cp command only copies files/directories with the local host's file systems. The scp command is similar to cp, but scp lets you securely copy files from one machine to another. And also like cp, scp has a -r (recursive) option to copy directories.
scp usage is similar to cp in that it copies from a <source> to a <destination>, but uses remote machine addressing to qualify either the <source> or the <destination> but not both.
Remote machine addressing looks like this: <user_account>@<hostname>:<source_or_destination>
Examples:
Open a new Terminal (Mac) or Command Prompt (Window) window on your local computer (not logged in to your student account), and try the following, using your studentNN account and GSAF pod host.
Note that you will always be prompted for your credentials on the remote host when you execute an scp command.
To copy a remote file:
...
Working with remote files
scp (secure copy)
The cp command only copies files/directories with the local host's file systems. The scp command is similar to cp, but scp lets you securely copy files from one machine to another. And also like cp, scp has a -r (recursive) option to copy directories.
scp usage is similar to cp in that it copies from a <source> to a <destination>, but uses remote machine addressing to qualify either the <source> or the <destination> but not both.
Remote machine addressing looks like this: <user_account>@<hostname>:<source_or_destination>
Examples:
Open a new Terminal (Mac) or Command Prompt (Window) window on your local computer (not logged in to your student account), and try the following, using your studentNN account and GSAF pod host.
Note that you will always be prompted for your credentials on the remote host when you execute an scp command.
To copy a remote file:
| Code Block | ||||
|---|---|---|---|---|
| ||||
# On your local computer - not gsafcomp01 or gsafcomp02 # Be sure to use your assigned student account and hostname # copy "haiku.txt" from your remote student Home directory to your current local directory scp student01@gsafcomp01.ccbb.utexas.edu:~/haiku.txt . # copy "haiku.txt", now in your local current directory, to your remote student # Home directory with the name "haiku2.txt" scp ./haiku.txt student01@gsafcomp01.ccbb.utexas.edu:~/haiku2.txt |
...
| Tip |
|---|
When transferring files between your computer and a remote server, you always need to execute the command on your local computer. This is because your personal computer does not have an entry in the global hostname database (a.k.a. the , whereas the remote computer does. The global Domain Name Service, or DNS), whereas the remote computer does database maps full host names to their IP (Internet Protocol) address. Computers that can be accessed from anywhere on the Internet have their host names registered in DNS. |
wget (web get)
The wget <url> command lets you retrieve the contents of a valid Internet URL (e.g. http, https, ftp).
...
- ln -s <path> says to create a symbolic link link (symlink) to the specified file (or directory) in the current directory
- always use the -s option to avoid creating a hard link, which behaves quite differently
- the default link name corresponds to the last name component in <path>
- you can name the link file differently by supplying an optional link_file_name.
- it is best to change into (cd) the directory where you want the link before executing ln -s
- a symbolic link can be deleted without affecting the linked-to file
- the -f (force) option says to overwrite any existing symbolic link with the same name
...
- find returns a list of matching file paths on its standard output
- ln wants its files listed as arguments, not on standard input
- so the paths are piped to the standard input of xargs
- xargs takes the data on its standard input and calls the specified function (here ln -sf -t .) with that data as the function's argument list.
...
Display lines 7 - 9 of the compressed "jabber.gz" text
| Expand | ||
|---|---|---|
| ||
zcat jabber.gz | cat -n | tail +7 | head -3 |
...
3rd party tool files and streams
In Intro Unix: The Bash shell and commands: Getting help we saw that 3rd Third party bioinformatics tools are often written to perform sub-command processing; that is, they have a top-level program that handles multiple sub-commands. Examples include the bwa NGS aligner and the samtools and bedtools tool suites.
...
| Tip | ||
|---|---|---|
| ||
Many tools write their main output to standard output by default but have options to write it to a file instead. Similarly, tools often write processing status and diagnostics to standard error, and it is usually your responsibility to redirect this elsewhere (e.g. to a log file). Finally, tools may support taking their main input from standard input, but need a "placeholder" argument where you'd usually specify a file. That standard input placeholder is usually a single dash ( - ) but can also be a reserved word such as stdin. |
Now let's see how these concepts fit together when running 3rd party tools.
...
be a reserved word such as stdin. |
Now let's see how these concepts fit together when running 3rd party tools.
Exercise 2-3 bwa mem
Display the bwa mem sub-command usage using the more pager
| Expand | ||
|---|---|---|
| ||
Just typing bwa mem | more doesn't use the more pager! That's because bwa writes its usage information to standard error, not to standard output. So you have to use the funky 2>&1 syntax before piping to more: bwa mem 2>&1 | more |
Where does the bwa mem sub-command write its output?
| Expand | ||
|---|---|---|
| ||
The bwa mem usage says:
This does not specify an output file, so it must write its alignment information to standard output. |
How can this be changed?
| Expand | ||
|---|---|---|
| ||
The bwa mem options usage says:
|
...
This does not specify an output file, so it must write its alignment information to standard output. |
How can this be changed?
| Expand | ||
|---|---|---|
| ||
The bwa mem options usage says:
|
bwa mem also writes diagnostic progress as it runs, to standard error.
| Expand | |||||
|---|---|---|---|---|---|
| |||||
|
Show how you would invoke bwa mem to capture both its alignment output and its progress diagnostics. Use input from a my_fastq.fq file and ./refs/hg38 as the <idxbase>.
...
The cutadapt adapter trimming command reads NGS sequences from a FASTQ file, and writes adapter-trimmed reads to a FASTQ file. Find its usage.
| Expand | ||
|---|---|---|
| ||
cutadapt # overview; tells you to run cutadapt --help for details Note that it also points you to https://cutadapt.readthedocs.io/ for full documentation.
cutadapt -a ADAPTER [options] [-o output.fastq] input.fastq |
Where does cutadapt write its output to from by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The cutadapt usage says that output can be written to a file using the -o option
But the The brackets around [-o output.fastq] suggest this is optional. Reading a bit further we see:
This suggests output can be specified in 2 ways:
|
...
| Expand | ||
|---|---|---|
| ||
The cutadapt usage says an input.fastq file is a required argument:
But again, reading a bit further we see:
This says that the input.fastq file can be provided in one of three compression formats. And the usage also suggests input can be specified in 2 ways:
|
Where does cutadapt write its diagnostic output by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The cutadapt usage doesn't say anything directly about diagnostics:
But again, reading in the Output: options section:
Careful reading of this suggests that: When
|
| Expand | |||||
|---|---|---|---|---|---|
| |||||
|