2020 Pre-processing raw sequences

Before you start the alignment and analysis processes, it us useful to perform some initial quality checks on your raw data. You may also need to pre-process the sequences to trim them or remove adapters. Here we will assume you have paired-end data from GSAF's Illumina HiSeq sequencer.

Setup

Make sure you are logged in to our dedicated login node: login5.ls5.tacc.utexas.edu.

Set up to process the yeast data if you haven't already.

# Create a $SCRATCH area to work on data for this course,
# with a sub-direct[1ory for pre-processing raw fastq files
mkdir -p $SCRATCH/core_ngs/fastq_prep

# Make a symbolic links to the original yeast data:
cd $SCRATCH/core_ngs/fastq_prep
ln -s -f /work/projects/BioITeam/projects/courses/Core_NGS_Tools/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz
ln -s -f /work/projects/BioITeam/projects/courses/Core_NGS_Tools/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz

FASTQ Quality Assurance tools

The first order of business after receiving sequencing data should be to check your data quality. This often-overlooked step helps guide the manner in which you process the data, and can prevent many headaches.

FastQC

FastQC is a tool that produces a quality analysis report on FASTQ files.

Useful links:

• FastQC report for a Good Illumina dataset
• FastQC report for a Bad Illumina dataset
• Online documentation for each FastQC report

First and foremost, the FastQC "Summary" should generally be ignored. Its "grading scale" (green - good, yellow - warning, red - failed) incorporates assumptions for a particular kind of experiment, and is not applicable to most real-world data. Instead, look through the individual reports and evaluate them according to your experiment type.

The FastQC reports I find most useful, and why:

1. Should I trim low quality bases?
   • consult the Per base sequence quality report
     • based on all sequences
2. Do I need to remove adapter sequences?
   • consult the Adapter Content report
3. Do I have other contamination?
   
   • consult the Overrepresented Sequences report
     • based on the 1st 100,000 sequences, trimmed to 75bp
4. How complex is my library?
   • consult the Sequence Duplication Levels report
   • but remember that different experiment types are expected to have vastly different duplication profiles
     
     

Running FastQC

FastQC is available in the TACC module system on ls5. To make it available:

module load fastqc

It has a number of options (see fastqc --help | more) but can be run very simply with just a FASTQ file as its argument.

# make sure you're in your $SCRATCH/core_ngs/fastq_prep directory
cds
core_ngs/fastq_prep
fastqc small.fq

Exercise: What did FastQC create?

-rw-rw-r-- 1 abattenh G-801021 319652 Jun 13 16:50 small_fastqc.html
-rw-rw-r-- 1 abattenh G-801021 344099 Jun 13 16:50 small_fastqc.zip

Let's unzip the .zip file and see what's in it.

unzip small_fastqc.zip

What was created?

drwxrwxr-x 4 abattenh G-801021   4096 Jun 13 16:50 small_fastqc

drwxrwxr-x 2 abattenh G-801021   4096 Jun 13 16:50 Icons
drwxrwxr-x 2 abattenh G-801021   4096 Jun 13 16:50 Images
-rw-rw-r-- 1 abattenh G-801021  77561 Jun 13 16:50 fastqc.fo
-rw-rw-r-- 1 abattenh G-801021  25635 Jun 13 16:50 fastqc_data.txt
-rw-rw-r-- 1 abattenh G-801021 319652 Jun 13 16:50 fastqc_report.html
-rw-rw-r-- 1 abattenh G-801021    446 Jun 13 16:50 summary.txt

Looking at FastQC output

You can't run a web browser directly from your "dumb terminal" command line environment. The FastQC results have to be placed where a web browser can access them. One way to do this is to copy the results back to your laptap (read more at Copying files from TACC to your laptop).

For convenience, we put an example FastQC report at this URL: http://web.corral.tacc.utexas.edu/BioITeam/yeast_stuff/Sample_Yeast_L005_R1.cat_fastqc/fastqc_report.html

Exercise: Based on this FastQC output, should we trim this data?

Using MultiQC to consolidate multiple QC reports

FastQC reports are all well and good, but what if you have dozens of samples? It quickly becomes tedious to have to look through all the separate FastQC reports, including separate R1 and R2 reports for paired end datasets.

The MultiQC tool helps address this issue. Once FastQC reports have been generated, it can scan them and create a consolidated report from all the individual reports.

Whats even cooler, is that MultiQC can also consolidate reports from other bioinformatics tools (e.g. bowtie2 aligner statistics, samtools statistics, cutadapt, Picard, and may more). And if your favorite tool is not known by MultiQC, you can configure custom reports fairly easily. For more information, see this recent Byte Club tutorial on Using MultiQC.

Here we're just going to create a MultiQC report for two paried-end ATAC-seq datasets – 4 FASTQ files total. First stage the data:

mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq
cd $SCRATCH/core_ngs/multiqc/fqc.atacseq
cp $CORENGS/multiqc/fqc.atacseq/*.zip .

You should see these 4 files in your $SCRATCH/core_ngs/multiqc/fqc.atacseq directory:

50knuclei_S56_L007_R1_001_fastqc.zip  5knuclei_S77_L008_R1_001_fastqc.zip  
50knuclei_S56_L007_R2_001_fastqc.zip  5knuclei_S77_L008_R2_001_fastqc.zip  

Now make the MultiQC accessible in your environment. It is not in the standard TACC module system, but is in the BioContainers.

# Load the main BioContainers module, then the multiqc module
module load biocontainers  # may take a while
module load multiqc

# Ask multiqc for its usage information
multiqc --help

Even though multiqc has many options, it is quite easy to create a basic report by just pointing it to the directory where individual reports are located:

cd $SCRATCH/core_ngs/multiqc
multiqc fqc.atacseq

Exercise: How many reports did multiqc find?

[WARNING]         multiqc : MultiQC Version v1.9 now available!
[INFO   ]         multiqc : This is MultiQC v1.7
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching 'fqc.atacseq/'
[INFO   ]          fastqc : Found 4 reports
[INFO   ]         multiqc : Compressing plot data
[INFO   ]         multiqc : Report      : multiqc_report.html
[INFO   ]         multiqc : Data        : multiqc_data
[INFO   ]         multiqc : MultiQC complete

Exercise: What was created by running multiqc?

You can see the resulting MultiQC report here: https://web.corral.tacc.utexas.edu/BioinformaticsResource/CoreNGS/reports/atacseq/multiqc_report.html.

FASTQ or BAM statistics with samstat

The samstat program can also produce a quality report for FASTQ files, and it can also report on aligned sequences in a BAM file.

This program is not available through either of the TACC module systems but is available in the BioITeam directory and was linked into your ~/local/bin directory. You should be able just to type samstat and see some documentation.

Running samstat on FASTQ files

Here's how you run samstat on a compressed FASTQ files. Let's only run it on a few sequences to avoid overloading the system:

cd $SCRATCH/core_ngs/fastq_prep
zcat Sample_Yeast_L005_R1.cat.fastq.gz | head -100000 | samstat -f fastq -n samstat.fastq

This would produce a file named samstat.fastq.html which you need to view in a web browser. The results can be seen here: https://web.corral.tacc.utexas.edu/BioITeam/CoreNGS/reports/samstat.fastq.html.

And here's some example output from bacterial sequences: https://web.corral.tacc.utexas.edu/BioITeam/CoreNGS/SRA/SRR030257_1.fastq.html

Running samstat on BAM files

Before running samstat on a BAM file at TACC, you must make the samtools program accessible by loading the samtools module.

module load samtools

Here's how you can run samstat on a small BAM file. This produces a file named small.bam.html which you need to view in a web browser.

cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/misc/small.bam .
samstat -F 772 small.bam

Note that odd -F 772 option – it's the filtering flags corresponding to samtools view -F <flags>. The value supplied above is what you should supply when you want to analyze only the reads that mapped in your BAM file (BAM files can contain both mapped and unmapped reads).

You can see the results here: https://web.corral.tacc.utexas.edu/BioITeam/CoreNGS/reports/small.bam.html. You can also view sample output for aligned yeast data here: http://web.corral.tacc.utexas.edu/BioITeam/yeast_stuff/yeast_chip_sort.bam.html

Trimming sequences

There are two main reasons you may want to trim your sequences:

• As a quick way to remove 3' adapter contamination, when extra bases provide little additional information
  • For example, 75+ bp ChIP-seq reads – 50 bases are more than enough for a good mapping, and trimming to 50 is easier than adapter removal, especially for paired end data.
  • You would not choose this approach for RNA-seq data, where 3' bases may map to a different exon, and that is valuable information.
    • Instead you would specifically remove adapter sequences.
• Low quality base reads from the sequencer can affect some programs
  
  • This is an issue with sequencing for genome or transcriptome assembly.
  • Aligners such as bwa and bowtie2 seem to do fine with a few low quality bases, soft clipping them if necessary.

There are a number of open source tools that can trim off 3' bases and produce a FASTQ file of the trimmed reads to use as input to the alignment program.

FASTX Toolkit

The FASTX Toolkit provides a set of command line tools for manipulating both FASTA and FASTQ files. The available modules are described on their website. They include a fast fastx_trimmer utility for trimming FASTQ sequences (and quality score strings) before alignment.

FASTX Toolkit is available as a BioContainers module.

module load biocontainers
module spider fastx
module load fastxtools

Here's an example of how to run fastx_trimmer to trim all input sequences down to 50 bases. By default the program reads its input data from standard input and writes trimmed sequences to standard output:

# make sure you're in your $SCRATCH/core_ngs/fastq_prep directory
zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 > trim50_R1.fq

• The -l 50 option says that base 50 should be the last base (i.e., trim down to 50 bases)
• The -Q 33 option specifies how base qualities on the 4th line of each FASTQ entry are encoded.
  
  • The FASTX Toolkit is an older program written in the time when Illumina base qualities were encoded differently, so its default does not work for modern FASTQ files.
  • These days Illumina base qualities follow the Sanger FASTQ standard (Phred score + 33 to make an ASCII character).

Exercise: compressing fastx_trimmer output

How would you tell fastx_trimmer to compress (gzip) its output file?

zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 -z > trim50_R1.fq.gz

zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 | gzip > trim50_R1.fq.gz

Exercise: other fastx toolkit programs

What other FASTQ manipulation programs are part of the FASTX Toolkit?

Note that the FASTX Toolkit also has programs that work on FASTA files. To see them, type fasta_ then tab twice (completion) to see their names.

Adapter trimming with cutadapt

Data from RNA-seq or other library prep methods that result in short fragments can cause problems with moderately long (50-100bp) reads, since the 3' end of sequence can be read through to the 3' adapter at a variable position. This 3' adapter contamination can cause the "real" insert sequence not to align because the adapter sequence does not correspond to the bases at the 3' end of the reference genome sequence.

Unlike general fixed-length trimming (e.g. trimming 100 bp sequences to 50 bp), specific adapter trimming removes differing numbers of 3' bases depending on where the adapter sequence is found.

The cutadapt program is an excellent tool for removing adapter contamination. It is available as a BioContainers module.

module load biocontainers
module spider cutadapt 
module load cutadapt 

cutadapt --help

A common application of cutadapt is to remove adapter contamination from RNA library sequence data. Here we'll show that for some small RNA libraries sequenced by GSAF, using their documented small RNA library adapters.

When you run cutadapt you give it the adapter sequence to trim, and this is different for R1 and R2 reads. Here's what the options look like (without running it on our files yet).

cutadapt -m 22 -O 4 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC <fastq_file>

cutadapt -m 22 -O 4 -a TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA <fastq_file>

Notes:

• The -m 22 option says to discard any sequence that is smaller than 22 bases (minimum) after trimming.
  
  • This avoids problems trying to map very short, highly ambiguous sequences.
• the -O 4 (Overlap) option says not to trim 3' adapter sequences unless at least the first 4 bases of the adapter are seen at the 3' end of the read.
  
  • This prevents trimming short 3' sequences that just happen by chance to match the first few adapter sequence bases.

Figuring out which adapter sequence to use when can be tricky. Your sequencing provider can tell you what adapters they used to prep your libraries. For GSAF's adapter layout, please refer to Illumina - all flavors (USE with Caution, this is outdated but can be useful for a basic understanding of the adapters, the GSAF primarily only uses UDI's for all projects) (you may want to read all the "gory details" below later).

<P5 capture> <indexRead2> <Read 1 primer> [insert] <Read 2 primer> <indexRead1> <P7 capture>

AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC

TCTACACGTTCAGAGTTCTACAGTCCGACGATCA    # small RNA sequencing primer site
CAGGTTCAGAGTTCTACAGTCCGACGATCA        # "other"
TCTACACTCTTTCCCTACACGACGCTCTTCCGATCT  # TruSeq Read 1 primer site. This is the RC of the R2 adapter

cutadapt -a GATCGGAAGAGCACACGTCTGAACTCCAGTCAC

TCTACACGTTCAGAGTTCTACAGTCCGACGATCA    # R1 primer - small RNA sequencing Read 1 primer site, 5' to 3'
TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA    # R2 adapter contaminent (RC of R1 small RNA sequencing Read 1 primer)

Exercise: other cutadapt options

The cutadapt program has many options. Let's explore a few.

How would you tell cutadapt to trim trailing N's?

cutadapt --help | less

How would you control the accuracy (error rate) of cutadapt's matching between the adapter sequences and the FASTQ sequences?

cutadapt --help | less

-e ERROR_RATE, --error-rate=ERROR_RATE
     Maximum allowed error rate (no. of errors divided by
     the length of the matching region) (default: 0.1)

Suppose you are processing 100 bp reads with 30 bp adapters. By default, how many mismatches between the adapter and a sequence will be tolerated?

How would you require a more stringent matching (i.e., allowing fewer mismatches)?

cutadapt example

Let's run cutadapt on some real human miRNA data.

First, stage the data we want to use. This data is from a small RNA library where the expected insert size is around 15-25 bp.

cd $SCRATCH/core_ngs/fastq_prep
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .

Exercise: How many reads are in these files? Is it single end or paired end data?

echo $(( `zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | wc -l` / 4 ))
# or
zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | wc -l | awk '{print $1 / 4}'

Exercise: How long are the reads?

zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2 | tail -1 | wc -c

Adapter trimming is a rather slow process, and these are large files. So to start with we're going to create a smaller FASTQ file to work with.

zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2000 > miRNA_test.fq

Now execute cutadapt like this:

cutadapt -m 20 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC miRNA_test.fq 2> miRNA_test.cuta.log | gzip > miRNA_test.cutadapt.fq.gz

Notes:

• Here's one of those cases where you have to be careful about separating standard output and standard error. 
  • cutadapt writes its FASTQ output to standard output, and writes summary information to standard error.
• In this command we first redirect standard error to a log file named miRNA_test.cuta.log using the 2> syntax, in order to capture cutadapt diagnostics.
  
  • Then standard output is piped to gzip, whose output is the redirected to a new compressed FASTQ file.
  • (Read more about Standard I/O streams)

You should see a miRNA_test.cuta.log log file when the command completes. How many lines does it have?

wc -l miRNA*.log

Take a look at the first 12 lines.

head -12 miRNA_test.cuta.log

It will look something like this:

This is cutadapt 1.14 with Python 2.7.15
Command line parameters: -m 20 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC miRNA_test.fq
Trimming 1 adapter with at most 10.0% errors in single-end mode ...
Finished in 0.02 s (44 us/read; 1.37 M reads/minute).

=== Summary ===

Total reads processed:                     500
Reads with adapters:                       492 (98.4%)
Reads that were too short:                  64 (12.8%)
Reads written (passing filters):           436 (87.2%)

Notes:

• The Total reads processed reads line tells you how many sequences were in the original FASTQ file.
• Reads with adapters tells you how many of the reads you gave it had at least part of an adapter sequence that was trimmed.
  • Here adapter was found in nearly all (98.4%) of the reads. This makes sense given this is a short (15-25 bp) RNA library.
• The Reads that were too short line tells you how may sequences were filtered out because they were shorter than our minimum length (20) after adapter removal (these may have ben primer dimers).
  
  • Here ~13% of the original sequences were removed, which is reasonable.
• Reads written (passing filters) tells you the total number of reads that were written out by cutadapt
  • These are reads that were at least 20 bases long after adapters were removed

paired-end data considerations

Special care must be taken when removing adapters for paired-end FASTQ files.

• For paired-end alignment, aligners want the R1 and R2 fastq files to be in the same name order and be the same length.
• Adapter trimming can remove FASTQ sequences if the trimmed sequence is too short
  
  • but different R1 and R2 reads may be discarded
  • this leads to mis-matched R1 and R2 FASTQ files, which can cause problems with aligners like bwa
•  cutadapt has a protocol for re-syncing the R1 and R2 when the R2 is trimmed.
  
  • See the cutadapt manual for more details (https://cutadapt.readthedocs.org/en/stable/).

running cutadapt in a batch job

Now we're going to run cutadapt on the larger FASTQ files, and also perform paired-end adapter trimming on some yeast paired-end RNA-seq data.

First stage the 4 FASTQ files we will work on:

mkdir -p $SCRATCH/core_ngs/cutadapt
cd $SCRATCH/core_ngs/cutadapt
cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz .
cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R1.fastq.gz .
cp $CORENGS/yeast_stuff/Yeast_RNAseq_L002_R2.fastq.gz .

Now instead of running cutadapt on the command line, we're going to submit a job to the TACC batch system to perform single-end adapter trimming on the two lanes of miRNA data, and paired-end adapter trimming on the two yeast RNAseq FASTQ files.

Paired end adapter trimming is rather complicated, so instead of trying to do it all in one command line we will use one of the handy BioITeam scripts that handles all the details of paired-end read trimming, including all the environment setup.

The name of the script we want is trim_adapters.sh. Just type the full path of the script with no arguments to see its help information:

/work/projects/BioITeam/common/script/trim_adapters.sh

You should see something like this:

trim_adapters.sh 2020_04_20
Trim adapters from single- or paired-end sequences using cutadapt. Usage:

trim_adapters.sh <in_fq> <out_pfx> [ paired min_len adapter1 adapter2 ]

Required arguments:
  in_fq     For single-end alignments, path to input fastq file.
            For paired-end alignemtts, path to the the R1 fastq file
            which must contain the string 'R1' in its name. The
            corresponding 'R2' must have the same path except for 'R1'
  out_pfx   Desired prefix of output files.
Optional arguments:
  paired    0 = single end alignment (default); 1 = paired end.
  min_len   Minimum sequence length after adapter removal. Default 32.
  adapter1  3' adapter. Default GATCGGAAGAGCACACGTCTGAACTCCAGTCAC  (NEB).
            Specifiy 'illumina' for AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC
            (standard Illumina TruSeq3 indexed adapter).
  adapter2  5' adapter. Default TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA (NEB).
            Specifiy 'illumina' for AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTA
            (standard Illumina TruSeq universal adapter).
Environment variables:
  show_only 1 = only show what would be done (default not set)
  keep      1 = keep intermediate file(s) (default 0, don't keep)
  cuta_args other cutadapt options (e.g. '--trim-n --max-n=0.25')

Examples:
  trim_adapters.sh my.fastq.gz h54_totrna_b1 1 40 '' '' '-O 5'

Based on this information, here are the 3 cutadapt commands we want to execute:

/work/projects/BioITeam/common/script/trim_adapters.sh Sample_H54_miRNA_L004_R1.cat.fastq.gz H54_miRNA_L004 0 20
/work/projects/BioITeam/common/script/trim_adapters.sh Sample_H54_miRNA_L005_R1.cat.fastq.gz H54_miRNA_L005 0 20
/work/projects/BioITeam/common/script/trim_adapters.sh Yeast_RNAseq_L002_R1.fastq.gz yeast_rnaseq 1

Let's  put these command into a cuta.cmds commands file. But first we need to learn a bit about Editing files in Linux.

Exercise: Create cuta.cmds file

Use nano or emacs to create a cuta.cmds file with the 3 cutadapt processing commands above. If you have trouble with this, you can copy a pre-made commands file:

cd $SCRATCH/core_ngs/cutadapt
cp $CORENGS/tacc/cuta.cmds .

When you're finished you should have a cuta.cmds file that is 3 lines long (check this with wc -l).

Next create a batch submission script for your job and submit it to the normal queue with a maximum run time of 2 hours.

launcher_creator.py -j cuta.cmds -n cuta -w 3 -t 02:00:00 -q normal -a UT-2015-05-18
sbatch --reservation=intro_NGS cuta.slurm
showq -u

How will you know your job is done?

export EMAIL_ADDRESS="abattenhouse@utexas.edu"

# to see all lines in any log file with the words 'completed successfully':
grep 'completed successfully!' *.log

# or, to simply count how many lines have the the words 'completed successfully':
grep 'completed successfully!' *.log | wc -l

# to see the names of the files that do NOT have the words 'completed successfully'
# note we only look in *.cuta.log files since the pass0, pass1, pass2 log files will
# never contain those words.
grep -L 'completed successfully!' *.cuta.log

You should see several log files when the job is finished:

• H54_miRNA_L004.cuta.log, H54_miRNA_L005.cuta.log, yeast_rnaseq.cuta.log
  • these are the main execution log files, one for each trim_adapters.sh command
• H54_miRNA_L004.acut.pass0.log, H54_miRNA_L005.acut.pass0.log
  • these are cutadapt statistics files for the single-end adapter trimming
  • their contents will look like our small example above
• yeast_rnaseq.acut.pass1.log, yeast_rnaseq.acut.pass2.log
  • these are cutadapt statistics files from trimming the R1 and R2 adapters, respectively.

Take a look at the first 17 lines of the yeast_rnaseq.acut.pass1.log log file:

head -17 yeast_rnaseq.acut.pass1.log

It will look something like this:

This is cutadapt 1.18 with Python 3.7.1
Command line parameters: -m 32 -a GATCGGAAGAGCACACGTCTGAACTCCAGTCAC --trim-n --paired-output yeast_rnaseq_R2.tmp.cuta.fastq -o yeast_rnaseq_R1.tmp.cuta.fastq Yeast_RNAseq_L002_R1.fastq.gz Yeast_RNAseq_L002_R2.fastq.gz
Processing reads on 1 core in paired-end legacy mode ...
WARNING: Legacy mode is enabled. Read modification and filtering options
*ignore* the second read. To switch to regular paired-end mode,
provide the --pair-filter=any option or use any of the
-A/-B/-G/-U/--interleaved options.
Finished in 151.54 s (24 us/read; 2.55 M reads/minute).

=== Summary ===

Total read pairs processed:          6,440,847
  Read 1 with adapter:               3,875,741 (60.2%)
  Read 2 with adapter:                       0 (0.0%)
Pairs that were too short:             112,847 (1.8%)
Pairs written (passing filters):     6,328,000 (98.2%)

• cutadapt started with 6,440,847 read pairs
  
  • 112,847 reads were discarded as too short after the R1 adapter was removed
    • the same 112,847 reads were discarded from both the R1 and R2 files
  • the remaining 6,328,000 were then subjected to pass2 processing.

The resulting pass2 log looks like this:

This is cutadapt 1.18 with Python 3.7.1
Command line parameters: -m 32 -a TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA --paired-output yeast_rnaseq_R1.cuta.fastq -o yeast_rnaseq_R2.cuta.fastq yeast_rnaseq_R2.tmp.cuta.fastq yeast_rnaseq_R1.tmp.cuta.fastq
Processing reads on 1 core in paired-end legacy mode ...
Finished in 83.64 s (13 us/read; 4.54 M reads/minute).

=== Summary ===

Total read pairs processed:          6,328,000
  Read 1 with adapter:                  90,848 (1.4%)
  Read 2 with adapter:                       0 (0.0%)
Pairs that were too short:                   0 (0.0%)
Pairs written (passing filters):     6,328,000 (100.0%)

Total basepairs processed: 1,198,172,994 bp
  Read 1:   639,128,000 bp
  Read 2:   559,044,994 bp
Total written (filtered):  1,197,894,462 bp (100.0%)

• cutadapt started with 6,328,000 pass1 reads
  • no additional reads were discarded as too short after the R2 adapter was removed
  • so new R1 and R2 files, both with 6,328,000 reads, were produced
    • yeast_rnaseq_R1.cuta.fastq.gz and yeast_rnaseq_R2.cuta.fastq.gz

Exercise: Verify that both adapter-trimmed yeast_rnaseq fastq files have 6,328,000 reads

echo "$((`zcat yeast_rnaseq_R1.cuta.fastq.gz | wc -l` / 4))"
zcat yeast_rnaseq_R2.cuta.fastq.gz | wc -l | awk '{print $1/4}'