Our clusters use the Linux Operating system System as their core together with some add-on tools provided by the San Diego Super Computing Center. Together the complete package as is known as ROCKS. Since UNIX is the core of the operating system part of the process of wielding the power of the cluster is learning how use the various UNIX provided, and other commands to manipulate your data.
To There are a number of resources available to help in this process
Our clusters are Linux based, so to get the maximum use of them you'll need to become familiar with the UNIX command line, which is a powerful text based interface which lets you type in commands to the computer for evaluation. This can be a bit of a barrier to getting your work done, but with perseverance you can learn enough UNIX to be at least able to use the cluster. While we can answer some quick, "how do I do this", or "why is this" type questions, in the long term it's better that you bite the bullet and get this done. Here are some resources that we have run across
...
:
- Computer Sciences offers a beginners UNIX course, CS108 (and as well various 1 hour programming classes in C++, Perl, Ruby, Python, etc. This are all offered under the listing CS105 Programming Languages)
- The Texas Advanced Computing Center or TACC offers a UNIX class every semester. (Note: This is a place holder in case people see this advertised. As of Feb, 2011 this was not an adequate class).
- O'Reilly and Associates publishes very good books that range from beginner to reference. In particular, Learning the UNIX Operating System is very good.
- Computer Science offers CS 108: UNIX, 1 hour course that is meant for beginners. If you don't take the class you might sit in, or get the book Linux: The Textbook which is pricey, but also very good.
- Several people have taken courses offered by Perl Source Informatics but they can be pricey.
You may also find loads of stuff using Google, and hopefully we will write our own version. It's just the case that right now we need more work on infrastructure, so spending time writing up own on version of things is counterproductive. If someone is willing to buy some of these reference texts, and donate them to use, we are willing to serve as the library. We will also be willing to track interest in the Perl Source Informatics courses as they do have bulk purchasing pages.
Initially when you use the cluster you can use some of our pre-canned scripts. After awhile you will probably be interested in write your own, or perhaps you will be forced to do so. While you can continue to use our scripts as templates for your own you might also want to get some books on scripting, and programming. First, you can get quite decent scripting done just by learning more about the shell we use which is the BASH shell. O'Reilly and Associates ahs a book on BASH which is called Learning the BASH Shell. This would also be useful for people wanting just to learn more about the powerful things they can do on the command line, or who want more information about the things they can do to customize their environment. They also have books on Perl, and Python for people interested in using BioPerl, or BioPtyhon. Computer Science also has Perl, Python, C++, and possibly other programming classes listed under the CS 105 class headinghttp://x.cygwin.com/. You will also want to read our documentation on the SGE Batch system which you will use to submit jobs to the cluster.Labels parameters
...
- .
- Several people have recommended Practical Computing for Biologists
- CCBB is currently working on producing a set of docs to be used in a tutorial session.
There of course may be many other resources that can be located using Google.
Besides learning to use commands, you will at least need to learn some basic shell scripting so that you can write job scripts to submit to the cluster's batch queuing system. The cluster requires those job scripts be written in the syntax of the BASH shell, so our UNIX accounts are created using BASH. You might want to pick up a book on BASH. The O'Reilly and Associates book Learning the BASH Shell is very good. In the event that you need something more than what we can provide with 3rd party software, you may want to learn to program in a language such as C, C++, or Fortran. Alternatively, you may want use one of the more advanced UNIX scripting languages such as perl, or python which we install with their add-on modules BioPerl, and BioPython (among others). Again O'Reilly and Associates is a good source for books on these subjects, as might be the Computer Science classes. The Department of Statistics and Scientific Computing also offers entry level programming courses.
To use 3rd party software we have added, you will want to learn to use the module command. You can also see the software that we have available on our systems.
Here is a A Little UNIX Dictionary which will let you use many of the buzzwords that you herehear.
People that want to run X-Windows (UNIX Graphical User Interface) apps, can read about this on the Using X-Windows at CCBB page.
For more cluster specific information, our Documentation page has a link for Cluster Specific Documentation.
Here is a tutorial showing how to manage disk space in UNIX.