Overview
Before you start the alignment and analysis processes, it can be useful to perform some initial quality checks on your raw data. If you don't do this (or even if you do), you may notice later that something looks fishy in the the output: for example, many of your reads are not mapping or the ends of many of your reads do not align. Both can give you clues about whether you need to process the reads to improve the quality of data that you are putting into your analysis.
For many years this tutorial is discussed at some length for if it should be included as a main tutorial, if it should be included as an optional tutorial, or if it should be ignored all together as the quality of data increases. Last year Recently a colleague of ours mine spent several days working with and trying to understand some data he got back before reaching out for help, after a few hours of running into a wall, fastqc was used to determine that the library was not constructed correctly in less than 30 minutes. Thus cementing the information as an important tutorial for a quick check which may save you significant amounts of time later on.
...
As we discussed in our first tutorial the head node is a space shared by all and we don't like stepping on each others toes. While the launcher_creator.py helper script makes working with the compute nodes much easier, they still take time to initiate a run (waiting in the que) and if you have errors in your commands your job will fail and you will lose your place in line. An idev (or interactive development session) is a way to move off the head node and onto a single compute node, but work interactively to see if your commands actually work, give you much quicker feedback, and if everything goes as you hope, your data. idev sessions are much more limited in duration and in general its not necessary to see every line a program spits out once you are familiar with the type of data you will get. Additionally, we are going to use a priority access reservation set up special for the BDIB summer school that you normally would not have access to but should guarantee immediate starting of your idev session.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
idev -m 240180 -r CCBB_5.22.17PMDay_1 -A UT-2015-05-18 # This should return the following: # We found an ACTIVE reservation request for you, named CCBB_5.22.17PM_Day_1. # Do you want to use it for your interactive session? # Enter y/n [default y]: # If for any reason you don't see the above message let usme know by raising your hand. # Your answer should be y, which should return the following: # Reservation : --reservation=CCBB_5.22.17PMDay_1 (ACTIVE) # Some of you may see a new prompt stating something like the following: # We need a project to charge for interactive use. # We will be using a dummy job submission to determine your project(s). # We will store your (selected) project $HOME/.idevrc file. # Please select the NUMBER of the project you want to charge.\n # 1 OTHER_PROJECTS # 2 UT-2015-05-18 # Please type the NUMBER(default=1) and hit return: # If you see this message, again let usme know. # You will then see something similar to the following: # job status: PD # job status: R # --> Job is now running on masternode= nid00032...OK # --> Sleeping for 7 seconds...OK # --> Checking to make sure your job has initialized an env for you....OK # --> Creating interactive terminal session (login) on master node nid00032. # If this takes more than 1 minute get ourmy attention. |
Your idev command line contains 3 flags: -m, -r -A. Using the `idev -h` command, can you figure out what these 3 flags mean and what you told the system you wanted to do?
| Expand | ||
|---|---|---|
| ||
From the OPTIONS: section of the idev help output: -m minutes sets time in minutes (default: 30) -r reservation_name requests use of a specific reservation -A account_name sets account name (default: -A none) So you requested an idev node for 240 180 minutes, using the reservation named CCBB_5.22.17PM_Day_1, and asked that it be charged to the account named UT-2015-05-18. |
...
GSAF gives you paired end sequencing data in two matching FASTQ format files, containing reads for each end sequenced: for example, Sample_ABC_L005_R1.cat.fastq and Sample_ABC_L005_R2.cat.fastq. Each read end sequenced is representd represented by a 4-line entry in the FASTQ file.
...
- Line 1 is the read identifier, which describes the machine, flowcell, cluster, grid coordinate, end and barcode for the read. Except for the barcode information, read identifiers will be identical for corresponding entries in the R1 and R2 fastq files.
- Line 2 is the sequence reported by the machine.
- Line 3 is always '+' from GSAF (it can optionally include a sequence description but rarely or never actually does)
- Line 4 is a string of Ascii-encoded base quality scores, one character per base in the sequence. For each base, an integer quality score = -10 log(probabilty probability base is wrong) is calculated, then added to 33 to make a number in the ASCII printable character range.
...
The first order of business after receiving sequencing data should be to check your data quality. This As discussed above, this often-overlooked step helps guide the manner in which you process the data, and can prevent many headaches that could require you to redo an entire analysis after they rear their ugly heads.
...