Overview:

Stampede2 is tremendously powerful and capable of doing many things, but as most of you are probably being slightly frustrated by, it doesn't have much in the way of a GUI (graphical user interface). The lack of a GUI means it can't visualize graphs or other meaningful representations of our data. In order to do these types of things, we have to get our data off of stampede2 and onto our own computers.

You recall one of the first things you did in this class was use the cp command to copy files from the the BioITeam directories to your home directory. Unfortunately sampede2 can't access the directories on your personal computer so the cp command won't work. Instead we have to use the "secure copy command" (scp). Copying using  scp works using the same 3 parts as the cp command:

1. the command itself (now scp instead of cp)
2. The file you want to securely copy 
3. The location you want the new file to go to

The difference is the scp command includes more detailed information on the path of where the file is, or where the file is going using syntax similar to that used in the other "secure" command you are familiar with: ssh.

Objectives:

1. Transfer a file from stampede2 to your local computer.
2. Transfer a file from your local computer to stampede2.

Tutorial:

• Setup

It is often useful to set up 2 different terminal windows to scp between a remote computer and your local computer. This is not required by any means, but is often helpful. The image below shows 2 separate windows where I have transferred a file named README from a folder inside the BioITeam materials to my local computer. Notice the one on the left corresponds to my local computer while the other is actually inside of the BioITeam folder on TACC. 

• Transfer from stampede2 to your computer

Before you begin, be sure you have 2 terminal windows open as shown in the "setup" section above. For simplicity the window corresponding to your Local computer will be referred to as "Left" while the window where you have logged into the Remote computer will be referred to as "Right" as is depicted in the image above.

In the Right window (Remote):

1. Navigate to the directory that contains the file you want to transfer.

   cd /corral-repl/utexas/BioITeam/ngs_course

2. Use the pwd command to print the current working directory

   pwd

3. Copy the address by highlighting the text and hitting control/command + C

   Explaining why the 2 window 'recommended practice' isn't actually necessary.

   If you are sure you know the address your file resides at from root, you do not need to do this step. In the experience of your instructor, being able to type the address out with tab keys in the remote window will safe a lot of typos when the next steps are completed.

In the Left window (Local):

1. Navigate to the directory that you want to copy the file to.

   cd Downloads

2. The scp command will have the following parts: scp <USERNAME>@<Remote_computer_address>:<pathway_to_file_you_want_to_transfer_starting_with_a_/_mark_and_ending_with_the_filename> <location_you_want_the_file_copied_to>

   use the scp command to transfer the remote file to to your local computer

   scp <USERNAME>@stampede2.tacc.utexas.edu:/corral-repl/utexas/BioITeam/ngs_course/README . 

   *note the required punctuation:

   1. @ symbol following your username (same as is done in the ssh command)
   2. : following the remote computer address. Missing this mark is the 2nd most common mistake people make
   3. / immediately after the : mark. Missing this mark is the most common mistake people make.

   Explaining why this is the only actual required command

   The final . on the command is not a required piece of punctuation, but instead is the symbol for the current directory. Step 1 could have been skipped, and the "." replaced with "Downloads" to have the same effect.

3. Upon hitting enter you should be prompted for your tacc password, and 1 time code.

4. You can then navigate to that folder using your operating system to see the new file and choose what to open it with (in this case it is a text file)

• Transfer from your computer to stampede2

Before you begin, be sure you have 2 terminal windows open as shown in the image in "setup" section above. For simplicity the window corresponding to your Local computer will be referred to as "Left" while the window where you have logged into the Remote computer will be referred to as "Right" as is depicted in the image above.

In the Right window (Remote):

1. Navigate to the directory that you want to transfer the file to..

   cd $SCRATCH

   Please do not upload files to the BioITeam folder

   In the previous example we copied a file from the BioITeam to your local computer. Make sure you change your directory before copying a file back to TACC. If you copy a file into that BioITeam space it creates more work for your instructor of having to clear out that space.

2. Use the pwd command to print the current working directory

   pwd

3. Copy the address by highlighting the text and hitting control/command + C

   Explaining why the 2 window 'recommended practice' isn't actually necessary.

   If you are sure you know the address your file resides at from root, you do not need to do this step. In the experience of your instructor, being able to type the address out with tab keys in the remote window will safe a lot of typos when the next steps are completed.

In the Left window (Local):

1. Navigate to the directory that has the file you want to copy.

   cd Downloads

2. The scp command will have the following parts: scp <filename> <USERNAME>@<Remote_computer_address>:<location_you_want_the_file_copied_to_starting_with_a_/_mark>

   use the scp command to transfer the remote file to to your local computer

   scp README <USERNAME>@stampede2.tacc.utexas.edu:<pwd_you_copied_in_the_right_window>

   *note the required punctuation:

   1. @ symbol following your username (same as is done in the ssh command)
   2. : following the remote computer address. Missing this mark is the 2nd most common mistake people make
   3. / immediately after the : mark. Missing this mark is the most common mistake people make.

   Explaining why this is the only actual required command

   As above, if you know the path to the directory you want to copy your file to, you do not need the Right window with the remote connection. Unfortunately, just as the remote computer doesn't know anything about your local computer, your local computer doesn't know anything about the nice shortcuts that exist on TACC such as the $SCRATCH variable. This means that in order to do this with a single window you would need to know what $SCRATCH is from memory.

   Even assuming you did know the path to the directory you want, it has been extremely rare in my experience that transferring a file to TACC is the last step. Almost always it is actually the first step where once the file is transferred you immediately begin working on it there.

3. Upon hitting enter you should be prompted for your tacc password, and 1 time code.

4. Switch back to the other window and use the ls command to see the file.

Alternatives:

1. Students sometimes ask if you can use scp in the window logged into TACC and transfer a file to their local computer. The answer is yes you can, but it is more complicated. For those wanting to figure this out for themselves, this question has been asked and discussed on stackexchange and should work for you. This is something your instructor has never done personally but is willing to answer questions about, or provide additional resources to help you do this for yourself if you think it important.
2. For those using putty, I remind you that using the power shell with openSSH installed from windows10 will allow you to use the scp command listed above. For those wanting help using power shell see this tutorial.
3. Another alternative for putty users is WinSCP. This link will take you to the installation instructions.
   1. Once installed, recommend using the 'commander' interface until you are more familiar with things. 
   2. This separate powerpoint provides some additional information.

While you can return to Genome Variant Analysis Course 2022 Home Page, it is more likely that you want to use the back arrow on your browser to return to the page you were just on.