Page Comparison

...

Code Block

language	bash

cd 
ln -s -f $SCRATCH scratch
ln -s -f $WORK$WORK2 workwork2
ln -s -f /workwork2/projects/BioITeam

mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /workwork2/projects/BioITeam/commonprojects/bincourses/launcher_maker.py
ln -s -f /work/projects/BioITeam/ls5/opt/cutadapt-1.10/bin/cutadaptCore_NGS_Tools CoreNGS

mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /workwork2/projects/BioITeam/ls5common/opt/multiqc-1.0/multiqcbin/launcher_creator.py
ln -s -f /workwork2/projects/BioITeam/ls5common/opt/samstat-1.09/samstatbin/launcher_maker.py

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

...

Code Block

language	bash

cd
cp .bashrc .bashrc.beforeNGSbeforeNGSTools

Copy and configure the login profile for this class

Code Block

title

language	bash		Copy a pre-configured login script

cd
cp /workwork2/projects/BioITeam/projects/courses/Core_NGS_Tools/tacc/bashrc.corengs.stampede2  .bashrc
chmod 600 .bashrc

# or, using your symlink
cd
cp ~/CoreNGS/tacc/bashrc.corengs.ls5stampede2  .bashrc
chmod 600 .bashrc

...

Code Block

language	bash
title	Copy a pre-configured login script

source ~/.bashrc

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

General

Code Block

language	bash

export ALLOCATION=UT-2015-05-18
export BIWORK=/work2/projects/BioITeam
export CORENGS=$BIWORK/projects/courses/Core_NGS_Tools

export PATH=.:$HOME/local/bin:$PATH

Turn on coloring by file type in the shell:

Code Block

language	bash

# For better colors using a dark background terminal, un-comment this line:
export LS_COLORS=$LS_COLORS:'di=1;33:fi=01:ln=01;36:'

# For better colors using a white background terminal:
export LS_COLORS=$LS_COLORS:'di=1;34:fi=01:ln=01;36:'

# May or may not be needed
export LS_OPTIONS='-N --color=auto -T 0

TACC intro

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Simple commands

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/slurm/simple
cd $SCRATCH/core_ngs/slurm/simple
cp $CORENGS/tacc/simple.cmds .

Wayness commands

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/slurm/wayness
cd $SCRATCH/core_ngs/slurm/wayness
cp $CORENGS/tacc/wayness.cmds .

Anchor
Catchup_idev
Catchup_idev
Start an idev session

To start an a 3-hour idev (interactive development) session:

Code Block

language	bash
title	Start an idev session

idev -p normal -m 120 -N 1 -n 2468 -A UT-2015-05-18 --reservation=CCBBBIO_DATA_week_1

Tip
You can tell you're in a idev session because the hostname command will return a compute node name (e.g. nid00438. c401-041.stampede2.tacc.utexas.edu) instead of a login node name (e.g. login5. login2.stampede2.tacc.utexas.edu).

The n idev session will terminate when the requested time has expired, or you use the exit command.

Working with FASTQ

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Working with some yeast ChIP-seq FASTQ data:

Code Block

language	bash

# Create a $SCRATCH area to work on data for this course,
# with a sub-direct[1ory for pre-processing raw fastq files
mkdir -p $SCRATCH/core_ngs/fastq_prep

# Make symbolic links to the original yeast data:
cd $SCRATCH/core_ngs/fastq_prep
ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz
ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz

# Copy over a small FASTQ file
cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/misc/small.fq .

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Get some FastQC reports for MultiQC:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq
cd $SCRATCH/core_ngs/multiqc/fqc.atacseq
cp $CORENGS/multiqc/fqc.atacseq/*.html .

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

For command-line cutadapt exploration:

Code Block

language	bash

cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .
zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2000 > miRNA_test.fq

For batch cutadapt processing:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/cutadapt
cd $SCRATCH/core_ngs/cutadapt
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R1.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R2.fastq.gz .
cp $CORENGS/tacc/cuta.cmds .

Alignment workflow

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Starting files:

Code Block

language	bash

# FASTA (for building references)
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/*.* $SCRATCH/core_ngs/references/fasta/

# FASTQ (to align)
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

Anchor
Catchup_references
Catchup_references
References

Get a copy of all references we build in the exercises (including FASTA):

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/references
rsync -ptlvrP $CORENGS/references/ $SCRATCH/core_ngs/references/

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

To jump into aligning PE yeast data with BWA

Code Block

language	bash

# Pre-built references
mkdir -p $SCRATCH/core_ngs/references
rsync -avrP $CORENGS/references/ $SCRATCH/core_ngs/references/

# FASTQ (to align)
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

# Alignment directory
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -s -f ../fastq
ln -s -f ../../references/bwa/sacCer3

module load biocontainers  # takes a while
module load bwa
module load samtools

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

To jump into post-alignment manipulation of the yeast_pairedend.bam with samtools:

Code Block

language	bash

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.bam .

module load biocontainers  # takes a while
module load samtools

# If the sorted, indexed BAM is needed:
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.sort* .

SAMTools and BEDTools

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools

Code Block

language	bash
title	Setup for samtools exercises

mkdir -p $SCRATCH/core_ngs/samtools
cd $SCRATCH/core_ngs/samtools
cp $CORENGS/catchup/for_samtools/* .

module load biocontainers  # takes a while
module load samtools

Versions Compared

Old Version 4

New Version Current

Key

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

TACC intro

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Anchor
Catchup_idev
Catchup_idev
Start an idev session

Working with FASTQ

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

Alignment workflow

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Anchor
Catchup_references
Catchup_references
References

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

SAMTools and BEDTools

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools

Page Comparison

Versions Compared

Old Version 4

New Version Current

Key

AnchorCatchup_bashrcCatchup_bashrc.bashrc setup

AnchorCatchup_env_varsCatchup_env_varsEnvironment variables

TACC intro

AnchorCatchup_cmdsCatchup_cmdsCommands files

AnchorCatchup_idevCatchup_idevStart an idev session

Working with FASTQ

AnchorCatchup_yeast_dataCatchup_yeast_dataYeast data

AnchorCatchup_multiqcCatchup_multiqcATACseq data for MultiQC

AnchorCatchup_cutadaptCatchup_cutadaptFASTQ files for cutadapt

Alignment workflow

AnchorCatchup_align_setupCatchup_align_setupAlignment workflow setup

AnchorCatchup_referencesCatchup_referencesReferences

AnchorCatchup_bwa_pe_yeastCatchup_bwa_pe_yeastBWA PE alignment of yeast data

AnchorCatchup_bwa_pe_yeastCatchup_bwa_pe_yeastsamtools manipulation of aligned yeast data

SAMTools and BEDTools

AnchorCatchup_samtoolsCatchup_samtoolsSetup for samtools

Anchor
Catchup_bashrc
Catchup_bashrc
.bashrc setup

Anchor
Catchup_env_vars
Catchup_env_vars
Environment variables

Anchor
Catchup_cmds
Catchup_cmds
Commands files

Anchor
Catchup_idev
Catchup_idev
Start an idev session

Anchor
Catchup_yeast_data
Catchup_yeast_data
Yeast data

Anchor
Catchup_multiqc
Catchup_multiqc
ATACseq data for MultiQC

Anchor
Catchup_cutadapt
Catchup_cutadapt
FASTQ files for cutadapt

Anchor
Catchup_align_setup
Catchup_align_setup
Alignment workflow setup

Anchor
Catchup_references
Catchup_references
References

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
BWA PE alignment of yeast data

Anchor
Catchup_bwa_pe_yeast
Catchup_bwa_pe_yeast
samtools manipulation of aligned yeast data

Anchor
Catchup_samtools
Catchup_samtools
Setup for samtools