Evaluating your raw sequencing data

Before you start the alignment and analysis processes, it us useful to perform some initial quality checks on your raw data. Here we will assume you have data from GSAF's Illumina HiSeq sequencer.

idev -m 60 -q development

Illumina sequence data format (FASTQ)

GSAF gives you paired end sequencing data in two matching fastq format files, contining reads for each end sequenced -- for example Sample_ABC_L005_R1.cat.fastq and Sample_ABC_L005_R2.cat.fastq. Each read end sequenced is representd by a 4-line entry in the fastq file.

A 4-line fastq file entry looks like this:

@HWI-ST1097:104:D13TNACXX:4:1101:1715:2142 1:N:0:CGATGT
GCGTTGGTGGCATAGTGGTGAGCATAGCTGCCTTCCAAGCAGTTATGGGAG
+
=<@BDDD=A;+2C9F<CB?;CGGA<<ACEE*1?C:D>DE=FC*0BAG?DB6

1. Line 1 is the read identifier, which describes the machine, flowcell, cluster, grid coordinate, end and barcode for the read. Except for the barcode information, read identifiers will be identical for corresponding entries in the R1 and R2 fastq files.
2. Line 2 is the sequence reported by the machine.
3. Line 3 is always '+' from GSAF (it can optionally include a sequence description)
4. Line 4 is a string of Ascii-encoded base quality scores, one character per base in the sequence. For each base, an integer quality score = -10 log(probabilty base is wrong) is calculated, then added to 33 to make a number in the Ascii printable character range.

See the Wikipedia FASTQ format page for more information.

Exercise: Examine the 2nd sequence in a FASTQ file

What is the 2nd sequence in the file /corral-repl/utexas/BioITeam/ngs_course/intro_to_mapping/data/SRR030257_1.fastq?

head /corral-repl/utexas/BioITeam/ngs_course/intro_to_mapping/data/SRR030257_1.fastq

Counting sequences

One of the first thing to check is that your fastq files are the same length, and that length is evenly divisible by four. The wc command (word count) using the -l switch to tell it to count lines, not words, is perfect for this:

wc -l $BI/ngs_course/intro_to_mapping/data/SRR030257_1.fastq

Exercise: Counting FASTQ file lines

How many sequences are in the FASTQ file above?

What if your fastq file has been compressed, for example by gzip? You can still count the lines, and you don't have to uncompress the file to do it:

gunzip -c $BI/web/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz | wc -l

Here you use gunzip -c to write decompressed data to standard output (-c means "to console", and leaves the original .gz file untouched). You then pipe that output to wc -l to get the line count.

Exercise: Counting compressed FASTQ lines

How many sequences are in the compressed FASTQ file above?