The Educational PODs

The Educational pod (EDU pod) is a special BRCF pod dedicated to educational support; that is, to be used exclusively for classroom instruction. The AMD GPU pod (AMDG pod) is also available for use by academic courses that need GPU support, as well as for qualified research groups. Its equipment was generously donated to UT by Austin's own Advanced Micro Devices.

The operations and management of these pods differ somewhat from standard BRCF pods. This page, intended for instructors and their TAs, describes the operational policies and procedures for the EDU and AMDG pods. See also this recording of our September 2024 Educational pods onboarding session.

• Quick Links:
  • EDU pod virtual host: edupod.cns.utexas.edu
    • e.g. https://edupod.cns.utexas.edu for the RStudio and JupyterHub server web applications
    • the AMDG pod does not have a virtual host name
  • Understanding and troubleshooting the R Studio Server and JupyterHub web applications
    • About R and R Studio Server
      
    • About Python and JupyterHub server
  • BRCF support email: rctf-support@utexas.edu
  • EDU pod Course Management interface: 
    • https://rctf-account-request.icmb.utexas.edu/edu-pod/ 
      • accessible only from the UT campus network or with the UT VPN service active
    • AMDG pod course management is performed by BRCF staff
  • About the AMDG pod GPU-enabled software environment: AMD GPU servers
• Important Notes:
  • Each pod has common Home, Work and Scratch storage areas.
  • Account EIDs must be entered in lowercase when logging in, either by SSH or to the RStudio Server/JupyterHub web applications.
    • Please let your students know to enter their EIDs using all lowercase letters.
  • EDU and AMDG pod course accounts are deactivated a few weeks after each semester ends (see Data/account retention)
    
    • Please let your students know this so they can ensure any files they want to retain in their Home directories are transferred elsewhere.

Overview

EDU pod servers

The EDU pod consists of three Dell PowerEdge R640 compute servers (educcomp01, educcomp02, educcomp04) each with 52 cores/104 hyper-threads and 1TB of RAM. All server hostnames are in the ccbb.utexas.edu domain (e.g. educcomp01.ccbb.utexas.edu). The compute servers share a common high-densiity storage server (educstor01) with standard /stor/home, /stor/work and /stor/scratch file systems.

There is also a virtual host name, edupod.cns.utexas.edu, that acts as a front-end, redirecting requests to specific back-end compute server based on server load. This virtual hostname can be used to access SSH, R Studio Server and JupyterHub servers unless a specific host is required (e.g. if a tmux or screen session is desired).

AMDG pod servers

The AMDG pod consists of three custom compute servers (amdgcomp01, amdgcomp02, amdgcomp03) each with 8 AMD Radeon Instinct MI-100 GPUs w/32GB onboard GPU RAM. Each compute server also has 128 cores and 512G of system RAM. All server hostnames are in the ccbb.utexas.edu domain (e.g. amgcomp01.ccbb.utexas.edu). The compute servers share a common high-densiity storage server (amdgstor01) with standard /stor/home, /stor/work and /stor/scratch file systems.

Because the workflows on the AMDG pod are expected to be significantly different from those on the standard EDU pod, the AMDG pod does not have a virtual host. Instead, users or instructors are expected to gauge the current load on the compute servers (see Monitoring GPU server resources) and select appropriate servers.

Use of EID accounts

Since the EDU and AMDG pods are used to support official UT courses, UT EIDs are used for all accounts (rather than user-configured accounts on BRCF research pods). Likewise, authentication is performed using UT EID authentication protocols rather than central password storage.

User roles

There are 4 roles: Admin, Instructor, TA, and Student.

The single Admin account is maint. Instructor, TA, and Student accounts are UT EIDs.

For the EDU pod, the Admin, Instructors and TAs have access to the Class Management web application (https://rctf-account-request.icmb.utexas.edu/edu-pod/ accessible only from the UT campus network or with the UT VPN service active). Students do not have access to the Class management application.

In addition to the Admin, Instructors and TAs also have sudo access on both pods, but should use it sparingly; for example, only to look into student Home directories which have permissions configured to allow access only by the Home directory owner. 

sudo should not be used for any of the following, as these are functions performed by the BRCF automated deployment system:

• Installing software that requires sudo
  • Contact us at rctf-support@utexas.edu to request installation of any software requiring sudo.
• Creating or modifying user accounts or groups
• Changing permissions on top-level directories in Home, Work and Scratch areas
  
  • Any such changes will be reverted by the deployment system

Class/semester groups

Courses have a class id designation (e.g. SDS375) and a semester (e.g. Spring2024).

Pod administration policies revolve around these class and class/semester concepts:

• Each class/semester combination has an associated Unix group (e.g. SDS375_Spring2024)
  • and a corresponding Work area directory (e.g. /stor/work/SDS375_Spring2024)
• Each class also has an associated Unix group (e.g. SDS375)
  • and a corresponding Work area directory (e.g. /stor/work/SDS375)
• Users associated with a course are assigned to both the class and class/semester Unix groups
  • the class/semester Unix group is their primary group

The goal is to allow access to both general course materials that should be available to any semester (via the class Unix group) as well as specific materials for a particular semester of a course (via the class/semester Unix group).

Note that some instructors, TAs, and students may be associated with multiple courses, hence belong to multiple class and class/semester groups. A list of an account's group membership, with the primary group first, can be displayed with the groups command, or sudo groups <eid>. Any user can change their primary group via newgrp <alternate_group_name>.

Course management

Courses for both EDU and AMDG pods are initially created by BRCF staff based on communication with instructors.

A course offering is a combination of a class identifier (e.g. SDS375) and a semester (e.g. Spring2024). and are generally referred to using their class/semester designation (e.g. SDS375_Spring2024).

The class designation may also include the instructor's initials (e.g. SDS320E_LG) when the same course is taught by multiple instructors (e.g. SDS320E_SR_Fall2024 and SDS320E_SR_Fall2024).

Multiple sections of a course generally share a single master course designation unless otherwise requested by the instructor.

Once a course has been created, the next step is managing its enrollment. When a course's enrollment is updated, the Course Management system will

• Deploy EID-named account information for the Instructor, TAs and Students to the appropriate pod, along with Unix group associations and Home directories
• Ensure Instructor and are TAs members of the sudo group
  
  

AMDG pod enrollment management

All enrollment management for AMDG pod courses is performed by BRCF staff. Instructors should email their rosters to rctf-support@utexas.edu and notify us when students are added/dropped.

Note that non-academic Research groups using the AMDG pod will also be assigned class/semester and class "pseudo-course" designations. In this case the semester simply reflects when the "pseudo-course" was created, and standard end-of-semester processing is not performed.

EDU pod enrollment management

EDU pod instructors can email rosters and subsequent add/drop information to rctf-support@utexas.edu to have us assist.

Once their course is initially created, EDU pod Instructors and TAs can also manage their enrollment by logging on to the EDU pod Course Management application, available from the UT Campus network or with the UT VPN service active:

• https://rctf-account-request.icmb.utexas.edu/edu-pod/

Student enrollment can be managed in the Edit Edu Course page. Steps to access it:

• Logon to the EDU pod Course Management application using your UT EID
  • you will see a list of your class/semester courses
• Select the "(full edit)" link next to the desired course
  
  • this will take you to the course's Edit Edu Course page

Managing student enrollment can be done either individually or through bulk upload of a Class Roster on the Edit Edu Course page.

Individual account management

To add a student (or TA) to a course individually, the account first needs to be configured if it is not already in the system (that is, the EID has not yet been associated with any course).

• On the Edit Edu Course page, a set of up to 20 of enrolled students will appear in the right-hand list box of the "Students" (or "TAs") area.
  • For technical reasons, all students cannot be loaded into this list box, and the 2nd number is actually the total number of EIDs known to the system.
• To add an EID, type it into the "Search" text box directly under the "Students" area (or under "TAs" if adding a TA).
  • If the EID is known to the system, it will appear in the area's left-hand list box
    • If it is not already enrolled, a plus sign ( "+" ) will appear to the right of the EID
      • clicking on the plus sign will add it to the right-hand list of students (or TAs)
      • clicking on the big red "Update" button will finalize the addition
    • If the EID is already enrolled, no plus sign will appear.
  • If the EID is not yet known to the system, the left-hand list box will be empty.
    • In this case the EID needs to be added to the system

To add an EID to the Course Management system:

• First click on the "Edu Accounts" menu item in the left margin menu list, then on the "Add new" link that appears.
  • This will take you to the Add new Account page.
• Type the EID in the "Eid*" text box then press Tab.
  • The "Firstname", "Lastname" and "Email" fields will be populated for valid EIDs.
• Click on the big red "Publish" button to add the EID to the system.

After adding the EID, return to the Edit Edu Course page:

• First click on the "Edu Courses" menu item in the left margin menu list
• Click on the "Title" of your course in the Edu Courses page
• In the the Edit Edu Course page, type the EID into the "Search" text box directly under the "Students" (or "TAs") area.
  • It should now appear in the area's left-hand list box with a plus sign ( "+" )
  • Click the plus sign to add the EID, then click the big red "Update" button to finalize the addition.

A similar process can be used to deactivate accounts for dropped students using the "Dropped students" area.

Bulk roster upload

A properly formatted Class Roster should be in CSV format, with a column named EID or UTEID that lists the student UT EIDs. Other columns may be present but will be ignored and all EIDs will be converted to lowercase.

To upload a Class Roster from the Edit Edu Course page:

• Click the "Select File" button in the "Class Roster" area 
• Make sure the "Upload files" tab is selected on the Select File page.
• Either drag/drop the Roster file onto the "Drop files to upload" area, or use the "Select Files" button to browse to it and "Open" it.
  • The file will appear "checked" in the your "Media Library" tab of the Select File page.
  • Click the "Use File" button in the lower right corner.
  • This will take you back to the Edit Edu Course page, with the Roster file now in the "Class Roster" area.
• Click the big red "Update" button to attempt the roster upload.
  
  • Check the "Status of last update" text area to determine if the upload was successful.

A set of up to 20 of enrolled students will appear in the right-hand list box of the "Students" area. For technical reasons, all students cannot be loaded into this list box; however you can check whether a particular student is enrolled by typing the EID into the "Search" text box directly under the "Students" area as described above.

Accessing pod resources

SSH access

All accounts can use SSH to access command-line computation resource. On the EDU pod, the preferred hostname is edupod.cns.utexas.edu, the virtual load-balancing proxy host. On either pod, specific pod compute servers can also be accessed (e.g. amdgcomp01.ccbb.utexas.edu).

SSH access from outside the UT campus network requires either the use of UT VPN service, or public key encryption as described at Passwordless Access via SSH.

Command line software access

Both EDU and AMDG pods has a large variety of bioinformatics utilities and tools installed that can be invoked from a Linux command line.

Users can install additional software that does not require administrative (sudo) access, or instructors can request that additional programs be installed via the rctf-support@utexas.edu support email.

The AMDG pod also has several GPU-enabled applications available. See AMD GPU servers for more information.

Web-based RStudio Server and JupyterHub applications

Web-based applications for R Studio Server and JupyterHub are available from a browser menu.

The preferred URL for the EDU pod is https://edupod.cns.utexas.edu, the front-end virtual host that load balances requests, directing them to one of the back-end physical compute servers.

AMDG pod users should access a specific compute server, and ensure its current load is appropriate (see Monitoring GPU server resources):

Both R Studio Server and JupyterHub web environments have a large number of packages available. Instructors can also request that additional software be installed. Such requests should ideally be made at least two weeks before the start of a course.

See About R and R Studio Server for information about the R Studio Server web application and the command-line R environments, and how to troubleshoot common problems.

See About Python and JupyterHub server for more information about the JupyterHub web application and the command-line Python environments, and how to troubleshoot common problems.

File systems and File management

File system organization

Like standard BRCF pods, both EDU and AMG pods have Home, Work and Scratch areas, although they are managed somewhat differently since they follow the class and class/semester concepts described above:

• /stor/home
  
  • Contains a Home directory for each user
    
    • Meant for student-specific work
    • Will be the current directory when a user accesses the RStudio Server or JupyterHub web applications, or logs in via SSH
  • Home directory permission only allow access by the owning account so that students cannot see each others' work
    
    • instructors & TAs can view via sudo
  • The Home directory group is the user's default Unix group (a class/semester group).
  • There are no quotas applied to Home directories so that users can store large data files there.
  • Snapshots (copy-on-write method) are enabled for Home directories to allow most deleted user files to be restored easily
    • Home directories are also backed up weekly
• /stor/work
  • The Work area has a course/semester directory for each course offering (e.g. /stor/work/SDS375_Spring2024)
    • Intended for resources specific to a particular class/semester, as well as multi-user projects at the instructor's discretion
      • E.g., instructors can stage assignments and data files to the shared class/semester Work directory
    • Owned by the class/semester Unix group
    • Permissions allow read/write by all users in the class/semester Unix group.
  • The Work area has a class directory for each course (e.g. /stor/work/SDS375)
    • Intended for resources common to many course offering semesters
    • Owned by the class Unix group
    • Permissions allow read/write by the class Unix group.
  • Work directories are fully backed up weekly, but do not have snapshots enabled
• /stor/scratch
  • The Scratch area also has class/semester and class directories for each course, similar to those in the Work area.
    
    • e.g. /stor/scratch/SDS375_Spring2024, /stor/scratch/SDS375
  • Scratch directories are not backed up.
    • Intended for externally downloaded resources

Managing assignment files

There are a number of options for managing homework assignment files, for example:

• Post assignments to Canvas. Have students download to their personal computers, then Upload to their Home directory in RStudio Server or JupyterHub.

  • When the assignment is complete, students can copy the files from their Home directory back to their personal computer using a remote file system transfer tool such as scp. This transfer must be initiated from the user's computer, as shown here. Then the completed assignment can be uploaded to Canvas from there.

  • Note: There is a 64 Megabyte file-size limit on uploading files in both JupyterHub and RStudio Server, so an alternate method should be used for files larger than 64M.

# From the student's personal computer
cd my_homework_directory
scp amb599@educcomp01.ccbb.utexas.edu:~/homework1.amb599.R . 

• Instructors can post assignments to their shared /stor/work/<Class_Semester> directory as describe in the Posting assignments to the shared course directory section.

  • After posting an assignment, instructors can have students copy it to their Home directory using a Terminal pane in RStudio Server, or JupyterHub, renaming it in the process. When finished, use a Terminal pane to copy the assignment to the shared /stor/work/<Course_Semester> directory.

# Copy/rename homework assignment to the user's Home directory,
# which is the default current directory in an RStudio Server or JupyterHub Terminal window.
cp /stor/work/MOL290C_Fall2024/homework1.R  ~/homework1.amb599.R

# When complete, copy the finished assignment back to the shared class/semester directory
# Note the -p option ensures the file's date/time stamp is preserved
cp -p homework1.amb599.R /stor/work/MOL290C_Fall2024/ 

Posting assignments to the shared course directory

Instructors can stage files in the shared /stor/work/<Class_Semester> directory in several ways:

• Use a remote file transfer command line utility such as scp, to copy files from your computer. The scp command-line program is available on Mac computers in the Terminal application, and on Windows computers in both Command Prompt and Windows PowerShell applications. For example:

# From the instructor's personal computer
cd my_assignments_directory
scp homework1.R iyervr@educcomp01.ccbb.utexas.edu:/stor/work/MOL290C_Fall2024/

• A graphical remote file transfer program can also be used, such as WinSCP on Windows or Cyberduck on Macs.

Other pod policies

Data/account retention

In order to manage pod accounts and data efficiently, both student accounts and Home directories are removed after courses complete.

Once a semester is over, all EDU and AMDG pod Home and Work area data are first archived to TACC's ranch tape archive system for long-term retention. Courses are then marked as "No longer active" in the Educational pod Class Management application, which will cause all student and TA accounts and Home directories to be removed from the system. Note that instructor accounts are not affected, nor accounts for students that are still enrolled in other courses.

Class/semester and class directory data is not removed, although class/semester directories may be moved underneath the corresponding class directory.

Maintenance windows

In order to address emerging security issues, occasional maintenance is performed on both EDU and AMDG pods, generally monthly. Note that during maintenance, pod resources are not available.

Dates for maintenance are posted in advance on the BRCF Users wiki home page. Instructors may request variances from the posted dates/times, and these will be honored to the extent possible.

Occasionally a problem will arise requiring immediate attention in order to restore a pod to its standard operating state. In such cases an Emergency Maintenance window will be opened, and email sent to all affected users.