Page Comparison

...

Now that we have backed up your profiles so you won't lose any previous settings, you can copy our predefined GVA2021GVA2022.bashrc file from the /corral-repl/utexas/BioITeam/scripts/ folder to your $HOME folder as .bashrc and the predefined GVA2021GVA2022.profile as .profile from the same location before using the chmod command to change the permissions to read and write for the user only.

Code Block

language	bash
title	Copy the course provided .profile file and change its name and permissions
collapse	true

cp /corral-repl/utexas/BioITeam/gva_course/GVA2021GVA2022.bashrc .bashrc
cp /corral-repl/utexas/BioITeam/gva_course/GVA2021GVA2022.profile .profile
chmod 700 .bashrc
chmod 700 .profile

...

Info

title	Understanding why some files start with a "."

In the above code box, you see that the names start with a . when a filename starts with a . it conveys a special meaning to the operating system/command line. Specifically, it prevents that file from being displayed when you use the ls command unless you specifically as for hidden files to be displayed using the -a option. Such files are termed "dot-files" if you are interested in researching them further.

Let's look at a few different ways we will use the ls command throughout the course. Compare the output of the following 4 commands:

Code Block

language	bash
title	Standard output

ls              #ignore everything that comes after the # mark. There is a problem on this wiki page but things after a # wont effect commands

Code Block

language	bash
title	Standard output plus hidden files

ls -a

Code Block

language	bash
title	Standard output plus hidden files in a single column

ls -a -1

Code Block

language	bash
title	Standard output plus hidden files in a single column with additional information

ls -a -l

Throughout the course you will notice that many options are supplied to commands via a single dash immediately followed by a single letter. Usually when you have multiple commands supplied in this manner you can combine all the letters after a single dash to make things easier/faster to type. Experiment a little to prove to yourself that the following 2 commands give the same output.

Code Block

language	bash
title	Standard output plus hidden files in a single column

ls -a -1

ls -al

While knowing that you can combine options in this way helps you analyze data faster/better, the real value comes from being able to decipher commands you come across on help forums, or in publications.

For ls specifically the following association table is worth making note of, but if you want the 'official' names consider using the man command to bring up the ls manual.

flag	association
-a	"all" files
-l	"long" listing of file information
-1	1 column
-h	human readable

Getting back to your profile... Since .bashrc is executed when you login, to ensure it is set up properly you should first logout:

...

Code Block

title	Creating a shortcut to the main Lonestar working directories

cdh
ln -s $SCRATCH scratch
ln -s $WORK work
ln -s $BI BioITeam

Several In previous years, several people have report seeing an error message stating "ln: failed to create symbolic link 'BioITeam/BioITeam': Permission denied." This is being investigated, but is not expected to impact today's tutorialseems to be related to different project allocations. I do not think it will be an issue for anyone this year.

Understanding what your .bashrc file actually does.

Expand

title	While interesting and useful information to have, understanding it is not critical to variant analysis. I suggest you to look through this information after you complete the rest of the tutorial, in your free time, or when you need to modify your profile or bashrc files in the future.

Info

Let's look at what your .bashrc profile actually does. Use the cat command to print contents of the .bashrc file to the screen.

Code Block

language	bash
title	Print the contents of the .profile file to the screen

cat .bashrc

This will print several lines of text to the terminal window. Let's look at what some of these lines do with a little more information:

lines that start with #
- Any line begins with a # symbol, is "commented out". Anything after a # symbol will not be executed by any program. Programers commonly make use of behavior to leave notes for others, or even themselves at a later date as to what particular lines of a script are actually doing.
Section 1 has multiple lines involving "module load <NAME>"
- This loads different modules by default. We have included basic ones that will help with basic TACC things. After we review the use of the nano text editor we'll go into more depth with TACC modules. But for now trust us when we say that not having to load a bunch of modules every time you log into TACC is a good thing.
- In previous years the module system was used more extensively. Here we will attempt to We now rely more on miniconda installations for increased portability. If you find yourself working within TACC (or equivalent resources), the module system (or similar systems) can be very advantageous.
Section 2 has multiple lines starting with "export"
- The export lines define shell variables for example BI and PATH. You've already seen how using $BI can come in handy accessing our shared course directory. As for PATH, that is a well-known environment variable that defines a set of directories where the shell will look when you type in a program's name. Our shared profile adds the common course directories that we copied at the start of this tutorial and your local ~/local/bin directory (which does not exist yet) to the location list. You can see the entire list of locations by doing this:
  Code Block
  language bash
  title How to see where the bash shell looks for programs
  echo $PATH
  As you can see, there are a lot of locations on the path. That's because when you load modules at TACC (see above), that mechanism makes the programs available to you by putting their installation directories on your $PATH.
umask 002
- The umask command is used to set the default permissions of newly created files and directories limiting the need to use the chmod command. umask functions as the inverse of chmod meaning that it subtracts the values from the default permissions. In this case the command umask 002 is the equivalent of the command chmod 775 for directories, and chmod 664 for files. in summary, having this command in your .profile gives all new files you create read and write access to both you and your group while giving read only access to everyone else.
PS1='tacc:\w$ '
- The PS1='tacc:\w$ ' line is a special setting that tells the shell to display the current directory as part of its prompt. It saves you typing pwd all the time to see where you are in the directory hierarchy. Try using the mkdir command to make a new directory called tmp and and change into that directory to see what it does to your prompt. This page maybe useful if you want to further customize your prompt after the course.
  Code Block
  language bash
  title See how your prompt reflects your current directory
  collapse true
  mkdir tmp cd tmp
- Your prompt should have changed from: "tacc:~$"to now be "tacc:~/tmp$". Your prompt now tells you you are in the tmp subdirectory of your home directory (~). See if you can figure out how to return to your home directory without expanding the code block. Expand the following code block to see the different ways of returning to your home directory.
  Code Block
  language bash
  title How to return to your home directory
  collapse true
  cd cdh cd $HOME cd ~ cd -
  The last example in the above code block will return you to your previous directory. In this case, that means the home directory, but it can be very useful in other situations when you change directories to do something in 1 place then need to hop back to where you were, or if you mistakenly leave a directory.

...

Linux text editors installed at TACC (nano, vi, emacs). These run in your terminal window. vi and emacs are extremely powerful but also quite complex, so nano is is the best choice as a first local text editor. It is also powerful enough that you can still accomplish whatever you are working on, it just might be more difficult if you try to do more complex edits. If you are already familiar with one of the other programs you are welcome to continue using it.
Text editors or IDEs that run on your local computer but have an SFTP (secure FTP) interface that lets you connect to a remote computer (Notepad++ or Komodo Edit). Once you connect to the remote host, you can navigate its directory structure and edit files. When you open a file, its contents are brought over the network into the text editor's edit window, then saved back when you save the file.
Software that will allow you to mount your home directory on TACC as if it were a normal disk e.g. MacFuse/MacFusion for Mac, or ExpanDrive for Windows or Mac ($$, but free trial). Then, you can use any text editor to open files and copy them to your computer with the usual drag-drop.

...

The most important thing to get used to is the convention of using . _ or capitalizing the first letter in each word in names rather than spaces in names, and limiting your use of any other punctuation. Spaces are great for mac and windows folder names when you are using visual interfaces, but on the command line, a space is a signal to start doing something different. Imagine instead of a BioITeam folder you wanted to make it a little easier to read and wanted to call it "Bio I Team" certainly everyone would agree its easier to read that way, but because of the spaces, bash will think you want to create 3 folders, 1 named Bio another named I and a third named Team. Now this is certainly behavior you can use when appropriate to your advantage, but generally speaking spaces will not be your friend. Early on in my computational learning I was told "A computer will always do exactly what you told it to do. The trick is correctly telling it to do what you want it to do".
Name things something that makes it obvious to you what the contents are not just today but next week, next month, and next year even if you don't touch the it for weeks-months-years.
Prefixing file/folder names with international date format (YYYY-MM-DD) will ensure that listing the contents will print in an order in which they were created. This can be useful when doing the same or similar analysis on new samples as new data is generated.

...

Stampede2 is a computer cluster connected to three file servers (each with unique characteristics), and other computer infrastructure. For the purpose of this class, and your own work, you only need to understand the basics of the 3 file servers to know how to use them effectively. The 3 servers are named, "HOME", "WORK2", and "SCRATCH", and we will work with them all over the next 5 days

	$HOME	$WORK2$WORK	$SCRATCH
Purged?	No	No	Files can be purged if not accessed for 10 days.
Backed Up?	Yes	No	No
Capacity	10GB	1TB	Basically infinite.
Commands to Access	cdh cd $HOME/	cdw cd $WORK/	cds cd $SCRATCH/
Purpose	Store Executables	Store Files and Programs	Run Jobs
Time spent	When modifying basic settings	When installing new programs; Storing raw or final data	When analyzing data

...

Code Block

language	bash
title	Example command for copying data from a $WORK directory to $SCRATCH . This command is only an example of something you may use in the future. As you do not have any fastq files on $WORK, or at least likely do not have them in a folder titled 'my_fastq_data' if you tried this command you would be expected to get a message stating no such file or directory found.

 cp cp $WORK2$WORK/my_fastq_data/*fastq $SCRATCH/my_project/

...

Versions Compared

Old Version 1

New Version 2

Key

Understanding what your .bashrc file actually does.