XSEDE Resources

Comet (SDSC)

Comet specs

Link to Comet user guide

To gain access

  1. Create user account at xsede.org
  2. Tell Mike to add you to the project
  3. Log into Comet!

Logging into Comet

From a terminal (Mac OSX): 

ssh -X [username]@comet.sdsc.edu

Navigate to your storage directory on lab storage on Oasis: 

cd /oasis/projects/nsf/[XSDEDE Project ID]/[username]

*See below for XSEDE project ID number

Set up software environment

Open the file ~/.bashrc using 'vi' and include the following lines at the bottom: 

export MODULEPATH=/share/apps/compute/modulefiles/applications:$MODULEPATH
module load relion/2.0.3

Comet Job Monitoring

Check status of job :

squeue -u [username]

RELION GPU Job Submission

First, note that the optimal submission for RELION is 2 nodes total. This optimizes speed vs. 'cost' (SUs). Read more here.

To submit a RELION GPU job:

1. From your project directory on Oasis, open the RELION GUI: 

$ relion

2. Input parameters related to your job & select 'Print command'. Copy the output text after the command relion_refine or relion_refine_mpi.

3. Create a new file (e.g. relion_gpu_submit.run)

4. Open file using 'vi' to include this template information: 

#!/bin/bash
#SBATCH --job-name="reliongpu"
#SBATCH --output="reliongpu.%j.%N.out"
#SBATCH --partition=gpu
#SBATCH --nodes=2                   #For 1 node
#SBATCH --ntasks-per-node=24
#SBATCH --no-requeue
#SBATCH --gres=gpu:4
#SBATCH -t 12:00:00                  #Edit to give approx. time in hours here
#SBATCH --mail-type=begin,end
#SBATCH --mail-user=                #Input email address here
#SBTACH --export=ALL
#SBATCH -A [ID]                     #Input XSEDE Project ID here
export MODULEPATH=/share/apps/compute/modulefiles/applications:$MODULEPATH
module load relion/2.0.3
date
ibrun -np 10 relion_refine_mpi [paste relion command here] --j 6

5. XSEDE Project ID number to include (need to be signed into UMich account & need permission to be able to view file)

6. Submit to Comet: 

$ sbatch [filename]

RELION CPU Job Submission (e.g. CTFFIND4)

To submit a RELION CPU job:

1. From your project directory on Oasis, open the RELION GUI: 

$ relion

2. Input parameters related to your job & select 'Print command'. Copy the output text after the command relion_refine or relion_refine_mpi.

3. Provide path to CTFFIND4 executable: 

/oasis/projects/nsf/[XSEDE project number]/micianfr/CTFFIND4/ctffind

4. Create a new file (e.g. relion_cpu_submit.run)

5. Open file using 'vi' to include this template information: 

#!/bin/bash
#SBATCH --job-name="relioncpu"
#SBATCH --output="relioncpu.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1                   #For 1 node
#SBATCH --ntasks-per-node=24
#SBATCH --no-requeue
#SBATCH -t 2:00:00                  #Edit to give approx. time in hours here
#SBATCH --mail-type=begin,end
#SBATCH --mail-user=                #Input email address here
#SBTACH --export=ALL
#SBATCH -A [ID]                     #Input XSEDE Project ID here
export MODULEPATH=/share/apps/compute/modulefiles/applications:$MODULEPATH
module load relion/2.0.3
date
ibrun -np 5 --tpr 4 relion_refine_mpi [paste relion command here]

6. XSEDE Project ID number to include (need to be signed into UMich account & need permission to be able to view file)

7. Submit to Comet: 

$ sbatch [filename]