Cluster & HTCondor

Biochemistry laboratories - 201

Jean-Yves Sgro [email protected]


*Note*: To see as slides click on **"Gift icon"** at the top right. To revert back to page view click on **"book icon"** when on the first slide.

Set-up

  • check your name on the attendees list
  • choose an iMac
  • login with your NetID ( @wisc.edu is not necessary)
  • If this is the first time this Mac "sees" you it will go through a quick set-up: simply skip using "Apple ID"

Finding Terminal on the iMac

Method 1

  • Click on the "Finder" at the bottom left of the "Dock" on the bottom of the screen. This will open a new window.
  • On the left click on "Applications"
  • In the alphabetical list open folder Utilities
  • double click to open Terminal

Method 2

  • Spotlight Search: Click the magnifying glass at the top right corner of your screen
  • type Terminal
  • press the return key

Cluster & HTCondor

The Biochemistry Computational Cluster (BCC) is a High Throughput Computing (HTC) environment within the UW-Madison Biochemistry Department.

             submit_node
                  |
                  |-[2TB disk]
        -------------------------
        |  |  |  |  |  |  |  |  |
        N1 N2 N3 N4 N5 N6 N7 N8 N9

HTCondor

  • HTCondor is a “scheduler” system that dispatches compute jobs to 1 or more "compute nodes."

  • Using HTCondor is the only approved method for performing high throughput computing on the BCC Linux cluster.

  • Jobs have to be ready to be processed by HTCondor as jobs cannot be interactive on the cluster.

Cluster access overview

  1. Text-based access only
  2. No graphical interface
    1. Login info ssh [email protected]
  3. Linux OS: Scientific Linux
  4. VPN access

Process

Submit a "job file" that contains information on how to run the software

  • submit file: myjob.sub
  • executable: myjob.sh

Information about software and file transfer

Getting ready

Invert the process:

  • define software and files (with dependecies)
  • create script to run "hands-off"
  • create submit file

QuickStart

Login - [replace myname with your login name.]

ssh [email protected]

Then move to /scratch and create a directory with your name and another directory within to work with.

$ cd /scratch
$ mkdir myname  #replace myname with e.g. YOUR ID
$ mkdir quickstart
$ cd myname/quickstart

Create an simple executable test file

Create execute file (e.g. with nano)

$ nano hello.sh

File content:

#!/bin/sh
echo "Hello World"

Make file executable:

$ chmod u+x hello.sh

Create a simple submit file

$ nano hello.sub

File content:

executable = hello.sh
should_transfer_files = Yes

output = hello.out
error  = hello.err
log    = hello.log

queue

Submit job

$ condor_submit hello.sub

Check output

$ ls
hello.err  hello.log  hello.out  hello.sh  hello.sub

On-line resources

Resource Link
HTCondor Quick Start Guide http://research.cs.wisc.edu/htcondor/manual/quickstart.html
Complete manual* http://research.cs.wisc.edu/htcondor/manual/

*You can check which manual you need by checking which version of HTCondor is installed with command: condor_version

In [ ]: