...

This is probably the longest tutorial in the entire class. It is designed to take between 1/2 and 3/4 of the first class. Do not stress if you feel people are moving through it faster than you are, or if you do not get it done before the next presentation. There will be links back to this tutorial from other tutorials as needed, and by the 2nd half of Wednesday's class when we start with the specialized tutorials, you can circle back to this tutorial as well. 

Objectives:

1. Familiarize yourself with the way course material will be presented.
2. Log into lonestar5.
3. Change your lonestar profile to the course specific format.
4. Refresh understanding of basic linux commands with some course organization.
5. Review use of the nano text editor program, and become familiar with several other text editor programs.

...

There will be 4 types of code blocks used throughout this class. Text inside of code blocks represent "right" answersat least 1 possible correct answer, and should either be typed EXACTLY into the terminal window as they are, or copy pasted with . There is a notable exception . Text that exist within text between <> symbols represent something that you need to replace before sending it to the terminal. Yes, the <> marks themselves also need to be replaced. We try to put informative text within the brackets so you know what to replace it with. If you are ever unsure of what to replace the <> text with, just ask.

1. Visible
   1. These are code blocks that you would have no idea what to type without help. (like when a new command is being introduced)
   2. These will typically be associated with longer/more detailed text above the text box explaining things.

   3. An example code block showing you the command you need to type into the prompt to list what directory you are currently in:

      Code Block
      language bash
      pwd

2. Hinted
   
   1. These are code blocks that you can probably figure out what to type with a hint that goes beyond what the tutorial is requesting. Access the hint by clicking the triangle or hint hyperlink text.
   2. These exist to force you to think about what command you need, and hopefully make some connections to help you remember what you will need to type in the future.
   3. These should all come with additional explanation as to what is going on.
   4. Rather than just expanding these by reflex, I strongly suggest seeing if you can figure out what the command will be, and checking your work

   5. Example:

      Expand
      title what command would you use to Print your current Working Directory
      In this example the letters P W and D are all capitalized to try to help you focus on the command itself Code Block language bash pwd

3. Hidden:
   1. These code blocks represent things that you should have seen several times already, or things that can be succinctly explained.

   2. Example:

      Code Block
      language bash title use the pwd command to print your current working directory collapse true
      pwd

4. Speed bump:

   1. This combines the previous 2 types to deliberately slow you down and be cumbersome. 
   2. If you find yourself consistently wrong about what eventually shows up in the text box, slow down, step back, think about whats going on, and consider asking a question.
   3. These should only come after you have seen the same (or very similar) commands in the other formats previously

   4. Example:

      Expand
      title print your current working directory
      Remember, the command you need is "pwd". Code Block language bash title This command needs no options collapse true pwd

...

ls              #ignore everything that comes after the # mark. There is a problem on this wiki page but things after a # wont effect commands

ls -a

ls -a -1

ls -a -l

ls -a -1

ls -al

...

exitlogout
# or
logoutexit

then log back in:

ssh <username>@ls5.tacc.utexas.edu

If everything is working correctly you should now see a this as your prompt like this:  

tacc:~$

...

cdh
ln -s $SCRATCH scratch
ln -s $WORK work
ln -s $BI BioITeam

Several people report seeing an error message stating "ln: failed to create symbolic link 'BioITeam/BioITeam': Permission denied." This is being investigated, but is not expected to impact today's tutorial.

• Understanding what your .bashrc file actually does.

cat .bashrc

...

• Linux text editors installed at TACC (nano, vi, emacs). These run in your terminal window. vi and emacs are extremely powerful but also quite complex, so nano may be  is the best choice as a first local text editor. Text editors If you are already familiar with one of the other programs you are welcome to continue using it.
• Text editors or IDEs that run on your local computer but have an SFTP (secure FTP) interface that lets you connect to a remote computer (Notepad++ or Komodo Edit). Once you connect to the remote host, you can navigate its directory structure and edit files. When you open a file, its contents are brought over the network into the text editor's edit window, then saved back when you save the file.
• Software that will allow you to mount your home directory on TACC as if it were a normal disk e.g. MacFuse/MacFusion for Mac, or ExpanDrive for Windows or Mac ($$, but free trial). Then, you can use any text editor to open files and copy them to your computer with the usual drag-drop.

...

...

1. The most important thing to get used to is the convention of using . or _ in  _  or capitalizing the first letter in each word in names rather than spaces in names, and limiting your use of any other punctuation. Spaces are great for mac and windows folder names when you are using visual interfaces, but on the command line, a space is a signal to start doing something different. Imagine instead of a BioITeam folder you wanted to make it a little easier to read and wanted to call it "Bio I Team" certainly everyone would agree its easier to read that way, but because of the spaces, bash will think you want to create 3 folders, 1 named Bio another named I and a third named Team. Now this is certainly behavior you can use when appropriate to your advantage, but generally speaking spaces will not be your friend. Early on in my computational learning I was told "A computer will always do exactly what you told it to do. The trick is telling it to do what you want it to do". 
2. Name things something that makes it obvious to you what the contents are not just today but next week, next month, and next year even if you don't touch the it for weeks-months-years.

...

When you execute the ls -1 > whatisHere command, you should have noticed nothing happening happened on the screen, and when you cat the whatisHere file, you should notice the output you would have expected from the ls -1 > whatisHere command. Often it is useful to chain commands together using the output of the first command as the input of the second command. Commands are chained together using the "|" character (shift \ above the return key). Use redirection to put the first 2 lines of the $BI directory contents into the whatisHere file.

...

mv whatisHere BioIteam_contents
mv whatisHere BioIteam_directory_contents
mv whatisHere BioIteam_directory_contents_2020-06

• Understanding TACC

Now that we've been using lonestar for a little bit, and have it behaving in a way that is a little more useful to us, let's get more of a functional understanding of what exactly it is and how it works.

...

 cp $WORK/my_fastq_data/*fastq $SCRATCH/my_project/

...

1. Job: Not required by the script, but likely always will be. This file has a list of commands that you want to run, and is probably biggest part of what makes tacc great... letting you run a large number of commands all at once rather than having to run them 1 at a time on your own computer.
   
2. Time and name: They are required so obviously important for that reason. Also names are important as mentioned above naming everything 'tacc_job' will make your life much more confusing at some point.
3. Wayness: 12 is the default, 48 is the max on lonestar. 
   1. This is a balance of memory and to a lesser extent speed against how long you want to wait for your job to run and SU cost. 
   2. Imagine you have 96 commands you want to run. 12, 24, and 48 as the "w" option will require 8, 4, and 2 remote computers to run at the same time, 2 computers are typically available sooner than a set of 4 and 4 are available sooner than a set of 8. So your job is likely to spend less time 'in the queue' with larger numbers.
   3. We'll go over "SUs" near the end of the course, but for now it is sufficient to say they are a currency for tacc, and that smaller numbers will cost more SUs
   4. For bacterial work, 48 is a much better choice in nearly all situations, unless you are experiencing problems or know you have a massive amount of data.
   5. The downside is if you pick a number that is too small, your commands may have errors and not actually produce results requiring you to start over with longer requests, or smaller "w" values
4. queue: Development and normal are the 2 you are likely to deal with most often. normal "Normal" is a better choice unless you are developing new code .or have odd turn around time requirements.

Running a job

...

# remember that things after the # sign are ignored by bash 
# lines in code blocks like this often will scroll to the right if you have a narrow browser window
cds  # move to your scratch directory
mkdir my_first_job  # make a new folder called "my_first_job"
cd my_first_job  # move into the new folder to make it easier to create a file there
nano commands  

cat commands > commands.out  # this will print the contents of the file you are currently editing to a new file called commands.out
date > date.out  # this will create a file with todays date on it
pwd > current_directory.out  # this will create a file with the current directory in it
echo "my name is <YOURNAME>" >> name.out  # Note that this time we used the append symbol >> not the write symbol > as we plan to put multiple things into the same file. Be sure to replace the <> signs with your name
echo "This is the final result of my first script. It worked how I thought it would, or hopefully have the resources to figure out why it didn't" >> name.out  # this will add another line of text to the name.out file.
# feel free to add up to 7 more lines to your commands file here using the cat/ls/pwd/mkdir/other commands that you know.
# beware using cd commands here as it will change your directory as if you were doing it on an interactive node and may cause you to reference files that don't exist

# write and exit nano now ctrl-o ctrl-x
launcher_creator.py -n "my_first_job" -j commands -t 00:02:00 -a "UT-2015-05-18" # this will create a my_first_job.slurm file that will run for 2 minutes
sbatch my_first_job.slurm  # this will actually submit the job to the Queue Manager and if everything has gone right, it will be added to the development queue.

Interrogating the launcher queue

Here are some of the common commands that you can run and what they will do or tell you:

...

Shows all of your currently submitted jobs, a state of:

"qw" means it is still queued and has not run yet

"r" means it is currently running

...

Delete a submitted job before it is finished running

note: you can only get the job-ID by using showq -u

...

There is no confirmation here, so be sure you are deleting the correct job.

There is nothing worse than deleting a job that has sat a long time by accident because you forgot something on a job you just submitted.

...

# write and exit nano now ctrl-o ctrl-x
launcher_creator.py -n "my_first_job" -j commands -t 00:02:00 -a "UT-2015-05-18" # this will create a my_first_job.slurm file that will run for 2 minutes
sbatch my_first_job.slurm  # this will actually submit the job to the Queue Manager and if everything has gone right, it will be added to the development queue.

Interrogating the launcher queue

Here are some of the common commands that you can run and what they will do or tell you:

If the queue is moving very quickly you may not see much output, but don't worry, there will be plenty of opportunity once you are working on your own data.

...

If (or when) you looked at what our edits to the .bashrc file did, you would have seen that section 1 has a series of "module load XXXX" commands, and a promise to talk more about them later. I'm sure you will be thrilled to learn that now is that time... As a "classically trained wet-lab biologist" one of the most difficult things I have experienced in computational analysis has been in installing new programs to improve my analysis. Programs and their installation instructions tend (or appear) to be written by computational biologists in what at times feels like a foreign language, particularly when things start going wrong. Luckily TACC (and the BioITeam) help get around a large number of these problems by preinstalling many programs if you know where to look.

After explaining the module system which we will use extensively throughout the course, we'll install 3 separate programs that we may use later in the class via 3 different means. This is an incomplete list of ways to install new programs to use, but is meant to be a good working example that you can adapt to install other programs in your future work.

TACC modules

Modules are programs or sets of programs that have been set up to run on TACC. They make managing your computational environment very easy. All you have to do is load the modules that you need and a lot of the advanced wizardry needed to set up the linux environment has already been done for you. New commands just appear.

To see all modules available in the current context, type:

 module avail

Remember you can hit the "q" key to exit out of the "more" system, or just keep hitting return to see all of the modules available. The "module avail" command is not the most useful of commands if you have some idea of what you are looking for. For example imagine you want to align a few million next generation sequencing reads to a genome, but you don't know what your options are. You can use the following command to get a list of programs that may be useful:

module keyword alignment

Note that this may not be an inclusive list as it requires the name of the program, or its description to contain the word "alignment". Looking through the results you may notice some of the programs you already know and use for aligning 2 sequences to each other such as blast and clustalw. Try broadening your results a little by searching for "align" rather than "alignment" to see how important word choice is. When you compare the two sets of results you will see that one of the new results is:

bowtie: bowtie/2.3.4
	Memory-efficient short read (NGS) aligner

 This may sound much better, but you still only have limited information about it. To learn more about a particular program, try the following 2 commands:

module spider bowtie

module spider bowtie/2.3.4

In the first case, we see information about what versions of bowtie lonestar has available for us, but really that is just the same information as we had from our previous search. This can be particularly useful when you know what program you want to use but don't know what versions are available. In the second case we now have more detailed information about the particular version of interest including websites we can go to to learn more about the program itself.

Once you have identified the module that you want to use, you install it using the following command:

 module load bowtie/2.3.4

 Since the module load command doesn't give any output, it is often useful to check what modules you have installed with either of the following commands:

module list
module list bowtie

The first example will list all currently installed modules while the second will only list modules containing bowtie in the name. If you see that you have installed the wrong version of something, a module is conflicting with another, or just don't feel like having it turned on anymore, use the following command:

 module unload bowtie

You will notice when you type module list you have several different modules installed already. These come from both TACC defaults (TACC, linux, etc), and several that are used so commonly both in this class and by biologists that it becomes cumbersome to type "module load python2" all the time and therefore we just have them turned on by default by putting them in our profile to load on startup.  As you advance in your own data analysis you may start to find yourself constantly loading modules as well. When you become tiered of doing this (or see jobs fail to run because the modules that load on the compute nodes are based on your .bashrc file plus commands given to each node), you may want to add additional modules to your .bashrc file. This can be done using the "nano .bashrc" command from your home directory.

Downloading from the web directly to tacc

This is about using the wget command. wget stands for Web get is a simple way of downloading a file from a web address to your current directory. Typically this makes use of the "Copy Link Address" option when you right click on a link in a web browser that would otherwise start a download to your computer. 

Here we will install the trimmomatic read trimming tool. As we will mention in our next tutorial, trimmomatic is a very robust trimming tool that can be integrated into a standard analysis pipeline, and one of the optional tutorials will go over its use.

In a new web browser window/tab, navigate to the trimmomatic home page. Trimmomatic is far above average for as far as programs go, most will not have a user manual, may not have been updated since originally published, etc. This is what makes it such a good tool. For now we will focus on Downloading Trimmomatic section; right click on the 'binary' link for version 0.39 and copy that link address.

Use the wget command to download the link you just copied to a new folder named src in your $WORK directorymany programs if you know where to look.

After explaining the module system which we will use extensively throughout the course, we'll install 3 separate programs that we may use later in the class via 3 different means. This is an incomplete list of ways to install new programs to use, but is meant to be a good working example that you can adapt to install other programs in your future work. If you choose to do one of the optional tutorials that involve the programs installed here the program installation will be covered in more detail at that time.

TACC modules

Modules are programs or sets of programs that have been set up to run on TACC. They make managing your computational environment very easy. All you have to do is load the modules that you need and a lot of the advanced wizardry needed to set up the linux environment has already been done for you. New commands just appear.

To see all modules available in the current context, type:

 module avail

Remember you can hit the "q" key to exit out of the "more" system, or just keep hitting return to see all of the modules available. The "module avail" command is not the most useful of commands if you have some idea of what you are looking for. For example imagine you want to align a few million next generation sequencing reads to a genome, but you don't know what your options are. You can use the following command to get a list of programs that may be useful:

module keyword alignment

Note that this may not be an inclusive list as it requires the name of the program, or its description to contain the word "alignment". Looking through the results you may notice some of the programs you already know and use for aligning 2 sequences to each other such as blast and clustalw. Try broadening your results a little by searching for "align" rather than "alignment" to see how important word choice is. When you compare the two sets of results you will see that one of the new results is:

bowtie: bowtie/2.3.4
	Memory-efficient short read (NGS) aligner

 This may sound much better, but you still only have limited information about it. To learn more about a particular program, try the following 2 commands:

module spider bowtie

module spider bowtie/2.3.4

In the first case, we see information about what versions of bowtie lonestar has available for us, but really that is just the same information as we had from our previous search. This can be particularly useful when you know what program you want to use but don't know what versions are available. In the second case we now have more detailed information about the particular version of interest including websites we can go to to learn more about the program itself.

Once you have identified the module that you want to use, you install it using the following command:

 module load bowtie/2.3.4

 Since the module load command doesn't give any output, it is often useful to check what modules you have installed with either of the following commands:

module list
module list bowtie

The first example will list all currently installed modules while the second will only list modules containing bowtie in the name. If you see that you have installed the wrong version of something, a module is conflicting with another, or just don't feel like having it turned on anymore, use the following command:

 module unload bowtie

You will notice when you type module list you have several different modules installed already. These come from both TACC defaults (TACC, linux, etc), and several that are used so commonly both in this class and by biologists that it becomes cumbersome to type "module load python3" all the time and therefore we just have them turned on by default by putting them in our profile to load on startup.  As you advance in your own data analysis you may start to find yourself constantly loading modules as well. When you become tiered of doing this (or see jobs fail to run because the modules that load on the compute nodes are based on your .bashrc file plus commands given to each node), you may want to add additional modules to your .bashrc file. This can be done using the "nano .bashrc" command from your home directory.

Downloading from the web directly to tacc

This is about using the wget command. wget stands for Web get is a simple way of downloading a file from a web address to your current directory. Typically this makes use of the "Copy Link Address" option when you right click on a link in a web browser that would otherwise start a download to your computer. 

Here we will install the trimmomatic read trimming tool. As we will mention in our next tutorial, trimmomatic is a very robust trimming tool that can be integrated into a standard analysis pipeline, and one of the optional tutorials will go over its use.

In a new web browser window/tab, navigate to the trimmomatic home page. Trimmomatic is far above average for as far as programs go, most will not have a user manual, may not have been updated since originally published, etc. This is what makes it such a good tool. For now we will focus on Downloading Trimmomatic section; right click on the 'binary' link for version 0.39 and copy that link address.

Use the wget command to download the link you just copied to a new folder named src in your $WORK directory.

cd $WORK
mkdir src
cd src

If you already have a src directory, you'll get a very benign error message stating that the folder already exists and thus can not be created. 

wget http://www.usadellab.org/cms/uploads/supplementary/Trimmomatic/Trimmomatic-0.39.zip

You should see a download bar showing you the file has begun downloading, when complete the ls command will show you a new compressed file named "Trimmomatic-0.39.zip". Next we need to uncompress this file, and copy the executable file to a location already in our $PATH variable.

unzip Trimmomatic-0.39.zip
cd Trimmomatic-0.39
cp trimmomatic-0.39.jar $HOME/local/bin

If you don't see the zip file or are unable to cd into the 0.39 directory after unzipping it let the instructor know.

As this tutorial is focused only on downloading interesting programs you may read about, this is the final step. If you do the optional trimmomatic tutorial later in the course, we'll go over some of the nuances of trimmomatic and shortcuts to make it easier to use.

wget alternative

It is always an alternative to download such files directly to your computer using a web browser and then use the scp command to transfer it to TACC. The wget command can help you avoid these intermediate steps and is more convenient most of the time unless you want to install the program on both your laptop and TACC, and have the same operating system on both.

Github

This is about using the git clone command. Git is a command often used for collaborative program development or sharing of files. Some groups also put the programs or scripts associated with a particular paper on a github project and publish the link in their paper or on their lab website. 

Here we will clone the github repository for breseq which is developed by the Barrick lab here at UT and is used to comprehensively analyze haploid microbial genomes to identify all variants present. In some of the initial tutorials everyone will use a version of breseq that is available through the BioITeam, in the optional tutorials you may compile your own copy of breseq from this github project to underscore why binary files are typically preferred, or as a way of easily staying up to date on new developments with the program itself.

Initially cloning a github repository as exceptionally similar to using the wget command to download the repository, it involves typing 'git clone' followed by a web address where the repository is stored. As we did for installing trimmomatic with wget we'll clone the repository into a 'src' directory inside of $WORK.

cd $WORK
mkdir src
cd src

If you already have a src directory, you'll get a very benign error message stating that the folder already exists and thus can not be created. 

wget http://www.usadellab.org/cms/uploads/supplementary/Trimmomatic/Trimmomatic-0.39.zip

You should see a download bar showing you the file has begun downloading, when complete the ls command will show you a new compressed file named "Trimmomatic-0.39.zip". Next we need to uncompress this file, and copy the executable file to a location already in our $PATH variable.

unzip Trimmomatic-0.39.zip
cd Trimmomatic-0.39
cp trimmomatic-0.39.jar $HOME/local/bin

As this tutorial is focused only on downloading interesting programs you may read about, this is the final step. If you do the optional trimmomatic tutorial later in the course, we'll go over some of the nuances of trimmomatic and shortcuts to make it easier to use. This page will be updated to include a link to said tutorial later in the week.

wget alternative

It is always an alternative to download such files directly to your computer using a web browser and then use the scp command to transfer it to TACC. The wget command can help you avoid these intermediate steps and is more convenient most of the time unless you want to install the program on both your laptop and TACC, and have the same operating system on both.

Github

This is done using the git clone command.In a web browser navigate to github and search for 'breseq' in the top right corner of the page. The top result will be for barricklab/breseq; click the green box for 'clone or download' and either control/command + C on the address listed, or click the clipboard icon to copy the repository address. This image may be helpful if you are are having trouble locating the green box. 

git clone https://github.com/barricklab/breseq.git

You will see several download indicators increase to 100%, and when you get your command prompt back the ls command will show a new folder named 'breseq' containing a set of files. If you don't see said directory, or can't cd into that directory let the instructor know.

As with Trimmomatic, these files will require additional work that is somewhat specific to the specific program and there for beyond the scope of this tutorial. A link to the advanced tutorials for getting your own copy of breseq up and running will be added later in the week. 

pip

This is about using the pip3 install command. pip is the standard package manager for the common programing language python. When labs put together new analysis programs/packages, increasingly they try to make these programs available for others to use via pip. pip3 rather than just pip references the current specific version of python.

Here we will install the multiqc analysis program which compiles reports from a program called fastqc about the quality of fastq files from multiple different samples at one time. In the later portion of the class you may choose to work with this program to get a better overall view of multiple fastq files all at once rather than clicking through individual reports.

...

If you still see something else, let the instructor know.

The multiqc tutorial can be found here.

This concludes the the linux and lonestar refresher/introduction tutorial.

Genome Variant Analysis Course 2020 home.

...