Overview

Throughout the course we have focused on small data sets, a limited number of samples, and in some cases even purposefully capped the total number of reads you have access to. This has been done for the purpose of time and letting you see the results tick by rather than simply having you come in for 30 minutes, submit a job, and wait an hour (or 6) before it starts running, and have it take another 10 hours to run. The reality is while you will sometimes work with a test sample or a small pilot project, Big Data in Biology means LOTS of data and lots of data means needing to not just identify variants in 1 sample, but to identify commonality across different systems. here we introduce you to bedtools. A program designed to make comparisons across different file types generated from different samples or using different parameters of a given pipeline.

Learning objectives

Review another situation of the importance of knowing what version you are working with, and take note of how older versions may be better or at least have their place when you dont have access to infinite computational resources
Become familiar with how to use bedtools intersect and subtract
Understand when and how bedtools is useful

A note on version control

Previous year's courses dealt a lot with using specific versions of programs, and conflicts that could arise on TACC depending on if you were accessing a program same program from the BioITeam, the TACC module system, or your own installation. With now using conda and being on stampede2 (which doesn't allow access to /corral-repl on idev nodes) all version issues should be eliminated. Friday's closing presentation will detail how to select good tools, what version to select, and when to upgrade to new versions of a tool. The following info box was left in this years tutorial to detail 2 things: 1 if it ain't broke, you probably aren't going to fix it, and 2 newest isn't best if you lack the resources to use it.

GVA2020 version control note

As mentioned several times throughout the course, different versions of software behave differently. Once again we have a situation where the BioITeam bedtools is available by default (version 2.20.1), and a different version is available on stampede (version 2.26.0). A KEY addition to version 2.21 (aka the version after that which is available to you by default through the BioITeam as of this writing) was the ability of bedtools to simultaneously scan multiple files at once rather than having to sequentially scan pairs of files. Using the old version, to identify the common variants of files A, B, C and D, bedtools would have to be invoked a minimum of 3 times:

A & B = E
C & D = F
E & F = G

More broadly, it would have to be invoked a minimum of (number of samples - 1) times. So as you start adding more and more samples, the commands get more and more difficult to write. In some of the other tutorials you will see how we have used command line for loops to systematically deal with these types of situations though this situation would be much more complex likely requiring nested for loops and conditionals. If you had an analysis pipeline worked out to do these types of comparisons for you, upgrading may not be of any interest to you.

With the new version however, such complexity isn't required making the tool more useful to more people though the computational requirements increase substantially. This is especially true when looking for convergence (ie common across all), or worse still thresholds (ie what variants are present in at least x% of the total samples). The larger your data set the more likely this can be a problem. Always make sure your programs are actually completing (NOT just erroring out), and remember there are almost always multiple ways to make the program finish running correctly (access different resources like large-mem nodes on TACC, read the documents, post on forums, reach out to former instructors, etc), or downgrade the version to an earlier version that was less computationally intensive.

Install bedtools version 2.31.0

from https://anaconda.org/bioconda/bedtools

conda create -n GVA-bedtools -c conda-forge -c bioconda bedtools

You can install to a different environment as you wish. Alternatively without conda-forge channel being included you will grab version 2.26 which should also work for the tutorial.

Comparing the results of different mappers using bedtools

Often you want to compare the results of variant calling on different samples or using different pipelines. Bedtools is a suite of utility programs that work on a variety of file formats, one of which is conveniently VCF format. It provides many ways of slicing, dicing, and comparing the information in VCF files. In this tutorial we will use it on .vcf files generated with samtools after mapping with each of 3 different read mappers (bowtie2, bwa and bowtie) to determine what predictions are the same and which are different from the variant calling on reads mapped with different programs.

Getting some data

For now, create a new directory named GVA_bedtools on scratch and copy the respective VCF files to it from $BI/gva_course/bedtools/:

By this point in the class you should know how to do this. Try to do it on your own and then check your work. If you were wrong, ask a question.

mkdir $SCRATCH/GVA_bedtools
cd $SCRATCH/GVA_bedtools
cp  -i $BI/gva_course/bedtools/*.vcf .

Remember the above command is simply 1 possible solution there are multiple ways you could have done this, most commonly recursively copying the entire directly, or copying all the files rather than just the vcf files.

Make sure you are on an idev node for the rest of the tutorial.

Common among all (intersect)

One of if not the most useful links that will be provided in this course is this link to the bedtools intersect page. Not only does it give a very nice graphical interface of what intersect is doing right at the top, but it provides amazing explanations for how to build a good analysis command. If the graphical image is say not what your ideal situation is (i.e. you are interested in seeing where 100% of the samples overlap, not where 1 sample over laps with any of the other samples) there is a command for that as well. Using the link provided, and the bedtools intersect -h command, see if you can identify the regions that were found by all 3 mappers.

Using the subcommands intersect we can find equal and different predictions between mappers. Try to figure out how to to do this on your own first.

Hint:

Remember that adding > output.vcf to the end of a command will pipe the output that is to the terminal into a file, so that you can save it.

Possible solution

conda activate GVA-bedtools
bedtools intersect -a bowtie2.vcf -b bowtie.vcf bwa.vcf -f 1.0 -header > all_aligners_with_header.vcf

The above is 1 possible answer. Did you come up with something different? here is what i thought was important to go into the command line

the use of -f 1.0 ensures that only variants in 100% of all the samples are listed

the -header to keep the header information moving forward.

Evaluating the results

Considering the 4 vcf files:

How many variants did each mapper lead to?
Bowtie 220
Bowtie2 133
bwa 120
Want to see how I figured that out? Expand source
grep -c "^NC_" *.vcf
How many mutations were detected in more than 1 mapper?
124
Want to see how I figured that out? Expand source
grep -c "^NC_" *.vcf
For comparison, compare just the two bowtie mappers ... how many vaiants are the same?
34
Want to see how I figured that out? Expand source
bedtools intersect -a bowtie2.vcf -b bowtie.vcf -f 1.0 -header > bowtie_aligners_with_header.vcf grep -c "^NC_" bow*.vcf

How similar are the bowtie2 and bwa results?

90

Want to see how I figured that out?

bedtools intersect -a bowtie2.vcf -b bwa.vcf -f 1.0 -header > bowtie2_v_bwa_with_header.vcf
grep -c "^NC_" b*.vcf

See if there are other comparisons that you are interested in.

Differential comparison (subtract)

Imagine a data set comprised of "disease state" and "normal state" individuals where the disease state is caused by any of some number of dominant (yet rare) mutations. Knowing that mutation is dominant, means that variants detected in the "normal state" are not of interest. How would one remove the variants identified in the "normal state" from the "diseased state"? How about bedtools subtract? Use the bedtools subtract -h command to see if you can put together a command.

First let's simulate some diseased data (say the intersection of bowtie2.vcf and bwa). Click here for example of how to do this:

bedtools intersect -a bowtie2.vcf -b bwa.vcf -f 1.0 -header > diseased.vcf

Next let's pretend that the bowtie.vcf file represents the intersection of the normal group

bedtools subtract -a diseased.vcf -b bowtie.vcf -f 1.0 -header > causitive.vcf

Using the commands above how many potential diseased mutations are there? how many appear they might be causative?

90, 55

Want to see how I figured that out?

grep -c "^NC_" *.vcf

Given what you saw comparing the the read mappers, imagine the 'diseased' samples are bowtie and bowtie2 results and the healthy results are bwa. How many of the mutations causative?

Need a hint?

#commands you need:
bedtools intersect
bedtools subract
grep

Going further

Hopefully you see how these tools can be useful (if not ask). These are but 2 of the bedtools commands, take a few minutes to look through the other bedtools subcommands that are available using bedtools -h, and bedtools other_command -h

You could go back and do this whole tutorial with the data generated data from the SRR030257 fastq files used in the mapping tutorials if you went back and did the analysis with both trimmed and untrimmed reads.

If you have done the advanced breseq tutorial, looking deeper at the 'gdtools' commands you will see you can do similar comparisons with .gd files from breseq. If you think this type of tutorial would be useful please mention it in the post class survey. Do you see how you could build out a similar set of questions?

Return to GVA2023 page

Bioinformatics Team (BioITeam) at the University of Texas

Bedtools tutorial -- GVA2023