...
| Code Block | ||
|---|---|---|
| ||
# create a symlink to a non-existent "~../xxx.txt" file, naming the symlink "bad_link.txt" mkdir -p ~/syms; cd ~/syms ln -sf ~../xxx.txt bad_link.txt ls -l |
Now both the symlink and the linked-to file are displayed in red, indicating a broken link.
Multiple files can be linked by providing multiple file name arguments along and using the -t (target) option to specify the directory where links to all the files can be created.
...
Here are some ways to work with a compressed file:
| Code Block | ||
|---|---|---|
| ||
cd ~/gzips cat ../jabberwocky.txt | gzip > jabber.gz # make a compressed copy of the "jabberwocky.txt" file less #jabber.gz make sure you're in your Home directory cat jabberwocky.txt | gzip > jabber.gz # make a compressed copy of the "jabberwocky.txt" file less jabber.gz # use 'less' to view the compressed "jabber.gz" file (q to exit) zcat jabber.gz | wc -l # count lines in the compressed "jabber.gz" file zcat jabber.gz | tail -4 # view the last 4 lines of the "jabber.gz" file zcat jabber.gz | cat -n # view "jabber.gz" text with line numbers (no zcat -n option) zcat jabber.gz | cat -n | tail +6 | head -4 # display lines 6 - 9 of "jabber.gz" text |
...
Recall the three standard Unix streams: they each have a number, a name and redirection syntax:
- standard output is stream 1
- redirect standard output to a file with a the > or 1> operator
- a single > or 1> overwrites any existing data in the target file
- a double >> or 1>> appends to any existing data in the target file
- redirect standard output to a file with a the > or 1> operator
- standard error is stream 2
- redirect standard error to a file with a the 2> operator
- a single 2> overwrites any existing data in the target file
- a double 2>> appends to any existing data in the target file
- redirect standard error to a file with a the 2> operator
We also saw that 3rd party bioinformatics tools are often written as a top-level program that handles multiple sub-commands. Examples include the bwa NGS aligner and samtools and bedtools tool suites. To see their menu of sub-commands, you usually just need to enter the top-level command, or <command> --help. Similarly, sub-command usage is usually available as <command> <sub-command> or <command> <sub-command> --help.
| Tip | ||
|---|---|---|
| ||
Many tools write their main output to standard output by default but have options to write it to a file instead. Similarly, tools often write processing status and diagnostics to standard error, and it is usually your responsibility to redirect this elsewhere (e.g. to a log file). Finally, tools may support taking their main input from standard input, but need a "placeholder" argument where you'd usually specify a file. That standard input placeholder is usually a single dash ( - ) but can also be a reserved word such as stdin. |
Now let's see how these concepts fit together when running 3rd party tools.
Exercise 1-1 bwa aln
Where does the bwa aln sub-command write its output?
| Expand | ||
|---|---|---|
| ||
The bwa aln usage
does not specify an output file, so it must write its alignment information to standard output. |
...
3rd party tool files and streams
In Intro Unix: The Bash shell and commands: Getting help we saw that 3rd party bioinformatics tools are often written to perform sub-command processing; that is, they have a top-level program that handles multiple sub-commands. Examples include the bwa NGS aligner and the samtools and bedtools tool suites.
To see their menu of sub-commands, you usually just need to enter the top-level command, or <command> --help. Similarly, sub-command usage is usually available as <command> <sub-command> or <command> <sub-command> --help.
| Tip | ||
|---|---|---|
| ||
Many tools write their main output to standard output by default but have options to write it to a file instead. Similarly, tools often write processing status and diagnostics to standard error, and it is usually your responsibility to redirect this elsewhere (e.g. to a log file). Finally, tools may support taking their main input from standard input, but need a "placeholder" argument where you'd usually specify a file. That standard input placeholder is usually a single dash ( - ) but can also be a reserved word such as stdin. |
Now let's see how these concepts fit together when running 3rd party tools.
Exercise 2-3 bwa mem
Where does the bwa mem sub-command write its output?
| Expand | ||
|---|---|---|
| ||
The bwa mem usage says:
This does not specify an output file, so it must write its alignment information to standard output. |
How can this be changed?
| Expand | ||
|---|---|---|
| ||
The bwa mem options usage says:
|
bwa mem also writes diagnostic progress as it runs, to standard error. Show how you would invoke bwa mem to capture both its alignment output and its progress diagnostics. Use input from a my_fastq.fq file and ./refs/hg38 as the <idxbase>.
| Expand | ||
|---|---|---|
| ||
Redirecting the output to a file: Using the -o option: |
Exercise 2-4 cutadapt
The cutadapt adapter trimming command reads NGS sequences from a FASTQ file, and writes adapter-trimmed reads to a FASTQ file. Find its usage.
| Expand | ||
|---|---|---|
| ||
cutadapt --help | less Note that it also points you to https://cutadapt.readthedocs.io/ for full documentation.
|
Where does cutadapt write its output to from by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The cutadapt usage says that output can be written to a file using the -o option
But the brackets around [-o output.fastq] suggest this is optional. Reading a bit further we see:
This suggests output can be specified in 2 ways:
|
Where does cutadapt read its input from by default? How can that be changed? Can the input FASTQ be in compressed format?
| Expand | ||
|---|---|---|
| ||
The bwa aln options cutadapt usage says an input.fastq file is a required argument: | ||
| Expand | ||
| ||
cutadapt --help | more Note that it also points you to https://cutadapt.readthedocs.io/ for full documentation.
|
bwa aln also writes diagnostic progress as it runs, to standard error. Show how you would invoke bwa aln to capture both its alignment output and its progress diagnostics. Use input from a my_fastq.fq file and ./refs/hg38 as the <prefix>.
| Expand | ||
|---|---|---|
| ||
Redirecting the output to a file: Using the -f option: |
Exercise 1-2 cutadapt
The cutadapt adapter trimming command reads NGS sequences from a FASTQ file, and writes adapter-trimmed reads to a FASTQ file. Find its usage.
But again, reading a bit further we see:
This suggests input can be specified in 2 ways:
And also says that the input.fastq file can be provided in one of three compression formats. |
Where does cutadapt write its diagnostic output by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The cutadapt usage doesn't say anything about diagnostics:
|
Where does cutadapt write its output to from by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The fastx_trimmer usage says that output is written to a file using the -o option
But the brackets around [-o output.fastq] suggest this is optional. Reading a bit further we see:
x |
Where does fastx_trimmer write its input from by default? How can that be changed?
| Expand | ||
|---|---|---|
| ||
The fastx_trimmer options usage says:
But again, reading in the Output: options section:
Careful reading of this suggests that:
|