Introduction

Throughout the course we have focused on samll data sets, a limited number of samples, and in some cases even purposefully capped the total number of reads you have access to. This has been done for the purpose of time and letting you see the results tick by rather than simply having you come in for 30 minutes, submit a job, and wait an hour (or 6) before it starts running, and have it take another 10 hours to run. The reality is while you will sometimes work with a test sample or a small pilot project, Big Data in Biology means LOTS of data and lots of data means needing to not just identify variants in 1 sample, but to identify commonality across different systems. here we introduce you to bedtools. A program designed to make comparisons across differnt file types generaterated from different samples or using different parameters of a given pipeline.

A note on version control

As mentioned in our samtools tutorial, different versions of software behave differently. Once again we have a situation where the BioITeam bedtools is available by default (version 2.20.1), and a different version is available on lonestar (version 2.25.0). A KEY addition to version 2.21 (aka the version after that which is available to you by default through the BioITeam as of this writing) was the ability of bedtools to simultaneously scan multiple files at once rather than having to sequencially scan pairs of files. Using the old version, to identify the common variants of files A, B, C and D, bedtools would have to be invoked a minimum of 3 times:

A & B = E
C & D = F
E & F = G

More broadly, it would have to be invoked a minimum of (number of samples - 1) times. So as you start adding more and more samples, the commands get more and more difficult to write. Hopefully after yesterday's tutorials, the sentence "you could write a wrapper to handle this for you" makes sense, and if it doesn't you should ask a question even if the question is "I don't know what question to ask". With the new version however, while the wrapper is not necessary, the computational requirements increase substantially. This is esspecially true when looking for convergence (ie common across all), or worse still thresholds (ie what variants are present in at least x% of the total samples). The larger your data set the more likely this can be a problem. Always make sure your programs are actually completing (NOT just erroring out), and remember there are multiple ways to make the program finish running correctly allbeit at a slower rate (read the documents, post on forums, reach out to former instructors, etc).

Based on what was just said, what do you think the first 2 things you should do are?

Need a hint?

Different versions exist
Can be very computationally intensive

Think you know? Check your guess...

Because different versions exist, you need to make sure that you are using the right version:

Check that you remember how to check if you have access to a command, what version that command is, load a different version, and verify that is the version being used

which bedtools  # check if you have access to bedtools
bedtools --version  # determine what version of bedtools you are using
 
# This will tell you that the bedtools you are using is in the $BI/bin directory and that you are using version v2.20.1-2-g6f0be00
 
module load bedtools  # load the TACC version of bedtools, remember module spider module avail and module list can help you find exactly what module you are looking for
which bedtools
bedtools --version
 
# This will now tell you that you are using version v2.25.0 which is located in the /opt/apps/bedtools/2.25.0/bin dreictory.

Because it is computationally intensive, you need to make sure you are not on the head node

Check that you remember how to get an idev node (remember to change your reservation to day 3)

idev  -m 180 -r CCBB_Bio_Summer_School_2016_Day3 -A UT-2015-05-18 -N 2 -n 8

Surely now that you have executed the above commands nothing could possibly go wrong right?

Hopefully you recognize the bold italic underlined sentence above as a warning that something can (and is likely about to go wrong), but if not ... something is about to go wrong, so try to figure out what that might be, and then click here to see if you are right

If you executed the above commands in the order they are listed from the head node (say because its the first tutorial you ran today and you are not already on a n idev node) you will find that bedtools has reverted to the BioITeam location and version because when you access an idev node it reloads your .bashrc file. Always make sure you are loading tools when and where you need them, and verify their accuracy rather than assuming it. module load bedtools

Objectives

Become familiar with how to use bedtools intersect and subtract
Understand when and how bedtools is useful

Comparing the results of different mappers using bedtools

Often you want to compare the results of variant calling on different samples or using different pipelines. Bedtools is a suite of utility programs that work on a variety of file formats, one of which is conveniently VCF format. It provides many ways of slicing, dicing, and comparing the information in VCF files. In this tutorial we will use it on .vcf files generated with samtools after mapping with each of 3 different read mappers (bowtie2, bwa and bowtie) to determine what predictions are the same and which are different from the variant calling on reads mapped with different programs.

Getting some data to work with

You could do this with your own personally generated data from the SRR030257 fastq files we used in the mapping tutorials. For now, create a new directory named BDIB_bedtools on scratch and copy the respective VCF files to it from $BI/gva_course/bedtools/:

By this point in the class you should know how to do this. Try to do it on your own and then check your work. If you were wrong, ask a question.

mkdir $SCRATCH/BDIB_bedtools 
cd $SCRATCH/BDIB_bedtools
cp  -i $BI/gva_course/bedtools/*.vcf .

Remember the above command is simply 1 possible solution there are multiple ways you could have done this, most commonly recursively copying the entire directly, or copying all the files rather than just the vcd files.

Common among all (intersect)

One of if not the most useful links that will be provided in this course is this link to the bedtools intersect page. Not only does it give a very nice graphical interface of what intersect is doing right at the top, but it provides amazing explanations for how to build a good analysis command. If the graphical image is say not what your ideal situation is (i.e. you are interested in seeing where 100% of the samples overlap, not where 1 sample over laps with any of the other samples) there is a command for that as well. Using the link provided, and the bedtools intersect -h command, see if you can identify the regions that were found by all 3 mappers.

Using the subcommands intersect we can find equal and different predictions between mappers. Try to figure out how to to do this on your own first.

Hint:

Remember that adding > output.vcf to the end of a command will pipe the output that is to the terminal into a file, so that you can save it.

Possible solution

bedtools intersect -a bowtie2.vcf -b bowtie.vcf bwa.vcf -f 1.0 -header> all_aligners_with_header.vcf

The above is 1 possible answer. Did you come up with something different? here is what i thought was important to go into the command line

the use of -f 1.0 ensures that only variants in 100% of all the samples are listed

the -header to keep the header information moving forward.

Differential comparison (subtract)

Imagine a data set comprised of "disease state" and "normal state" individuals where the disease state is caused by any of some number of dominant (yet rare) mutations. Knowing that mutation is dominant, means that variants detected in the "normal state" are not of interest. How would one remove the variants identified in the "normal state" from the "diseased state"? How about bedtools subtract? Use the bedtools subtract -h command to see if you can put together a command.

First let's simulate some diseased data (say the intersection of bowtie2.vcf and bwa). Click here for example of how to do this:

bedtools intersect -a bowtie2.vcf -b bwa.vcf -f 1.0 -header > diseased.vcf

Next let's pretend that the bowtie.vcf file represents the intersection of the normal group

bedtools subtract -a diseased.vcf -b bowtie.vcf -f 1.0 -header > causitive.vcf

Going further

Hopefully you see how these tools can be useful (if not ask). These are but 2 of the bedtools commands, take a few minutes to look through the other bedtools subcommands that are available using bedtools -h, and bedtools other_command -h

Bioinformatics Team (BioITeam) at the University of Texas

Bedtools tutorial -- GVA2016