NERSC Submission Examples

Pantheon Example

This bash script will run the Pantheon example in the CosmoSIS Standard Library. Put this in a text file, e.g. pantheon.sub, and then submit with srun pantheon.sub. This particular example is quick and small enough for the debug queue:

#!/bin/bash
# The line above must be the first of the script

# You can use debug for jobs of less than 30 minutes.
#SBATCH --qos=debug

# Wall-clock time limit
#SBATCH --time=00:10:00

# See https://docs.nersc.gov/jobs/policy/ for info on numbers
# of nodes you can have in different queues. One node would be
# fine for this particular task as it's really fast.
#SBATCH --nodes=2

# Run on Haswell CPUs. I haven't tried KNL, it might work.
#SBATCH --constraint=haswell

# This assumes you want the LSST NERSC account
#SBATCH --account=m1727

# This can be anything you like.
#SBATCH --job-name=pantheon-cosmosis

# Job output will be sent to pantheon-cosmosis.log
#SBATCH --output=%x.log

# You will get an email when the job completes or times out
#SBATCH --mail-type=END

# Now the actual work ....


# Let's run with 4 threads per process. Each node has 32 cores,
# so if we use 2 nodes that's 64 cores total, so we want 64/4 = 16 processes.
# There's also the "hyperthread" thing which could let us have twice as many,
# but the instructions about that are mixed - somewhere it says not to use it.
# We export OMP_NUM_THREADS so that it is used internally in camb and elsewhere.
# NUM_PROCESSES we just use below to tell srun how many tasks to launch
# It is worth exploring the best numbers to use for your specific chains.

NUM_PROCESSES=16
export OMP_NUM_THREADS=4

# Run the setup script
source $CFS/des/zuntz/cosmosis-global/setup-cosmosis-nersc

# Launch the program. Don't forget the --mpi flag!
srun -u -n ${NUM_PROCESSES} --cpus-per-task ${OMP_NUM_THREADS} cosmosis examples/pantheon.ini --mpi