Overview

The fastQC tool was presented on the first day of the class as the go to tool for quality control analysis of fastq files, but there is an underlying issue that checking each fastq file is quite daunting and evaluating each file individually can introduce its own set of artifacts or biases. The MultiQC tool represents a tool which works directly on fastQC reports to quickly generate summary reports to both identify samples that are different among a group and to make global decisions about how to treat a set of files.

Learning Objectives

In this tutorial, we will:

1. work with some simple bash scripting from the command line (for loops) to generate multiple fastqc reports simultaneously and look at 272 plasmid samples.
2. work with MultiQC to make decisions about read preprocessing.
3. identify outlier files that are clearly different from the group as a whole and determine how to deal with these files.

Get some data and load fastqc

Copy the plasmid sequencing files found in the BioITeam directory gva_course/plasmid_qc/ to a new directory named GVA_multiqc. There are 2 main ways to do this particularlly since there are so many files (544 total). 

cp -r $BI/gva_course/plasmid_qc/ $SCRATCH/GVA_multiqc

mkdir $SCRATCH/GVA_multiqc 
cp $BI/gva_course/plasmid+qc/* $SCRATCH/GVA_multiqc
cd $SCRATCH/GVA_multiqc

module load fastqc

Use a bash for loop on the command line to generate a fastQC command for all plasmid samples

We are going to construct a single commands file with 544 lines that we can give to the launcher_creator.py script to launch all commands without having to know the name of any single file.  To do so we will use the bash 'for' command.

For loops on the command line have 3 parts:

1. A list of something to deal with 1 at a time. Followed by a ';'
   1. for f in *.gz; in the following example
2. Something to do with each item in the list. this must start with the word 'do'
   1. do echo "fastqc -o fastqc_output $f "; in the following example
3. The word "done" so bash knows to stop looking for more commands.
   1. done in the following example, but we add a final redirect (>) so rather than printing to the screen the output goes to a file (fastqc_commands in this case)

for f in *.gz; do echo "fastqc -o fastqc_output $f" ;done  > fastqc_commands

Use the linux commands head and wc -l to see what the output is.

Next we need to make the output directory for all the fastqc reports to go and make the fastqc_commands file executable.

mkdir fastqc_output
launcher_creator.py -N 5 -n "fastqc" -t 00:30:00 -a "UT-2015-05-18" -j fastqc_commands

sbatch fastqc.slurm

Run MultiQC tool on all fastQC output

Unfortunately, multiqc is not available as a module on lonestar, and it actually ends up being more complicated to access in the BioITeam than it is to install it on your own. Finally, MultiQC provides us with a great opportunity to install a program for ourselves using pip (something that is fairly common among published scripts) and deal with the issue that pip by default tries to install things with administrator privileges which is fine for your personal laptop, but you don't have admin rights on TACC.

From the multiqc web page https://multiqc.info/ we see that the 'quick install' command is listed as pip install multiqc. The trick to getting pip installation commands to work on lonestar is to add --user after install, but before multiqc.

cd $HOME
pip install --user multiqc

You should see multiple different dependancies installing one after another and when it is done you should have it installed in a potentially a new folder .local/bin/ run the following command to verify the installation worked as expected. If you see a help file that is similar to others you have seen in class it worked, if you get an error message let me know.

 ~/.local/bin/multiqc -h

Hopefully by now the job we submitted with all the fastqc commands has finished. Use the showq -u command to check. Assuming it is, run the following command (on the headnode is fine).

cd $SCRATCH/GVA_multiqc
~/.local/bin/multiqc .

Evaluate MultiQC report

As the multiqc_report.html file is a html file, you will need to transfer it back to your laptop to view it. If you do not know how to do this by this point in the course get my attention and we will go through it together as it is a skill you will vitally need going forward.

Optional Exercise

Using information gained in the MultiQC report, modify the bash loop used for the fastQC commands to improve the raw reads.

Return to the Genome Variant Analysis Course 2019 Home Page