ANSYS on bwUniCluster 2.0
Reaching HE ANSYS license servers from bwUniCluster over a SSH tunnel.
ANSYS packages are connected to university's FLEXlm servers through two ports: 1055 and 2325.
Besides these 2 ports, an extra port, required by the FLEXlm daemon, should be tunneled as well. In the example below we named it DAEMON_PORT and set it to an arbitrary port, 49100.
This port is either random or, usually configured by the license servers administrator. Later in the scripts it will be referred with a placeholder <DAEMON_PORT>
The following command will create an SSH master control socket with the respective ports tunneled through the comserver.hs-esslingen.de:
Useful: SSH-Tunnel will need a passwordless connection from the cluster to the comserver.hs-esslingen.de through a SSH-key.
DAEMON_PORT=49100
ssh -MS ansys-socket -fnNT -L 2325:lizenz-ansys.hs-esslingen.de:2325 \
-L 1055:lizenz-ansys.hs-esslingen.de:1055 -L \
${DAEMON_PORT}:lizenz-ansys.hs-esslingen.de:${DEAMON_PORT} \
userID@comserver.hs-esslingen.de |
where, userID is the id of the user on comserver.hs-esslingen.de
After you finish your simulations, don't forget to close the socket with:
| ssh -S tunnel-socket -O exit userID@comserver.hs-esslingen.de |
ANSYS Fluent sbatch job using HE managed licenses.
As you may know from, bwUniCluster 2 Slurm common features, in order to start a job on the cluster you will need to send a specific batch script to sbatch Linux application.
Below, you may consult a sample of a script which will start a FLUENT job on bwUniCluster, using ANSYS v.2020R2, a single node, 28 processors.
The job will be using Esslingen's university ANSYS license manager, with the host name lizenz-ansys.hs-esslingen.de.
Note: Value placeholders in the script will be:
- HE user ID - <USER_ID>
- FLEXlm Daemon Port - <D_PORT>
FLEXlm DAEMON_PORT variable will be
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=28
# Queue class wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=single
# Maximum run time of job
#SBATCH --time=2:00:00
# Give job a reasonable name
#SBATCH --job-name=fluent-test
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=fluent-test-%j.out
# File name for error output
#SBATCH --error=fluent-test-%j.out
# send an e-mail when a job begins, aborts or ends
#SBATCH --mail-type=ALL
# e-mail address specification
#SBATCH --mail-user=<USER_ID>@hs-esslingen.de
echo "Starting at "
date
# load the software package
module load cae/ansys/2022R2_no_license
HE_USER_ID=<USER_ID>
HE_LIZENZ_SERVER='lizenz-ansys.hs-esslingen.de'
HE_COM_SERVER='comserver.hs-esslingen.de'
# start a SSH tunnel, creating a control socket.
DAEMON_PORT=<D_PORT>
ssh -M -S ansys-socket -fnNT -L 2325:${HE_LIZENZ_SERVER}:2325 \
-L 1055:${HE_LIZENZ_SERVER}:1055 \
-L ${DAEMON_PORT}:${HE_LIZENZ_SERVER}:${DAEMON_PORT} \
${HE_USER_ID}@${HE_COM_SERVER}
# export license environment variables
export ANSYSLMD_LICENSE_FILE=1055@localhost
export ANSYSLI_SERVERS=2325@localhost
# Create the hosts file 'fluent.hosts'
HOSTS="fluent.hosts"
scontrol show hostname ${SLURM_JOB_NODELIST} > ${HOSTS}
# set number of nodes variable
nrNodes=${SLURM_NTASKS}
echo "number of nodes: $nrNodes"
# run fluent in parallel, where fluentJournal.jou is a fluent Journal File
echo "Starting fluent..."
fluent 3d -t$nrNodes -g -env -pib -mpi=openmpi -cnf=${HOSTS} -i fluentJournal.jou &&
# close the SSH control socket
ssh -S ansys-socket -O exit ${HE_USER_ID}@${HE_COM_SERVER}
[[ -f ${HOSTS} ]] && rm -rf ${HOSTS}
echo "Run completed at "
date |
Download FLUENT Script
CFX sbatch job on a single node using HE managed licenses.
The script below, will start a CFX5 job on bwUniCluster, using ANSYS v.19.2, on 40 processors of a single node.
This script can also be downloaded clicking the liks in the page footer.
The job in an CFX5 def file named cfx_example.def will be using Esslingen's university ANSYS license manager, with the host name lizenz-ansys.hs-esslingen.de.
Note: Value placeholders in the script will be:
- HE user ID - <USER_ID>
- FLEXlm Daemon Port - <D_PORT>
FLEXlm DAEMON_PORT variable will be
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=40
# Queue class wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=single
# Maximum run time of job
#SBATCH --time=8:00:00
# Give job a reasonable name
#SBATCH --job-name=cfx5-job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=cfx-test-%j.out
# File name for error output
#SBATCH --error=cfx-test-%j.out
# send an e-mail when a job begins, aborts or ends
#SBATCH --mail-type=ALL
# e-mail address specification
#SBATCH --mail-user=<USER_ID>@hs-esslingen.de
echo "Starting at "
date
# load the software package
module load cae/ansys/2022R2_no_license
HE_USER_ID=<USER_ID>
HE_COM_SERVER='comserver.hs-esslingen.de'
HE_LIZENZ_SERVER='lizenz-ansys.hs-esslingen.de'
INPUT='cfx_example.def'
# start a SSH tunnel, creating a control socket.
DEAMON_PORT=<D_PORT>
SOCKET_NAME="cfx-socket"
[[ -f ${SOCKET_NAME} ]] && rm -f ${SOCKET_NAME}
ssh -M -S ${SOCKET_NAME} -fnNT -L 2325:${HE_LIZENZ_SERVER}:2325 \
-L 1055:${HE_LIZENZ_SERVER}:1055 \
-L ${DEAMON_PORT}:${HE_LIZENZ_SERVER}:${DEAMON_PORT} \
${HE_USER_ID}@${HE_COM_SERVER}
# export license environment variables
export ANSYSLMD_LICENSE_FILE=1055@localhost
export ANSYSLI_SERVERS=2325@localhost
cfx5solve -batch -def $INPUT -part=${SLURM_NTASKS}
# close the SSH control socket
ssh -S ${SOCKET_NAME} -O exit ${HE_USER_ID}@${HE_COM_SERVER}
echo "Run completed at "
date |
Download CFX Script
CFX sbatch job on multiple nodes using HE managed licenses.
The script below, will start a CFX5 job on bwUniCluster, using ANSYS v.19.2, 4 nodes with 40 tasks each, totaling in 160 processors.
This script can also be downloaded clicking the liks in the page footer.
The job in an CFX5 def file named cfx_example.def will be using Esslingen's university ANSYS license manager, with the host name lizenz-ansys.hs-esslingen.de.
Note: Value placeholders in the script will be:
- HE user ID - <USER_ID>
- FLEXlm Daemon Port - <D_PORT>
Note: You will also have to load system/ssh_wrapper/0.1 to allow cfx5solve node-communication over ssh.
#!/bin/bash
# Allocate one node
#SBATCH --nodes=4
# Number of program instances to be executed
#SBATCH --ntasks-per-node=40
# Queue class wiki.bwhpc.de/e/BwUniCluster_2.0_Batch_Queues
#SBATCH --partition=multiple
# Maximum run time of job
#SBATCH --time=8:00:00
# Give job a reasonable name
#SBATCH --job-name=cfx5-job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=cfx-test-%j.out
# File name for error output
#SBATCH --error=cfx-test-%j.out
# send an e-mail when a job begins, aborts or ends
#SBATCH --mail-type=ALL
# e-mail address specification
#SBATCH --mail-user=<USER_ID>@hs-esslingen.de
echo "Starting at "
date
# load the software package
module load system/ssh_wrapper/0.1
module load cae/ansys/2022R2_no_license
HE_USER_ID=<USER_ID>
HE_COM_SERVER='comserver.hs-esslingen.de'
HE_LIZENZ_SERVER='lizenz-ansys.hs-esslingen.de'
INPUT="cfx_example.def"
# start a SSH tunnel, creating a control socket.
DEAMON_PORT=<D_PORT>
SOCKET_NAME="cfx-socket"
[[ -f ${SOCKET_NAME} ]] && rm -f ${SOCKET_NAME}
ssh -M -S ${SOCKET_NAME} -fnNT -L 2325:${HE_LIZENZ_SERVER}:2325 \
-L 1055:${HE_LIZENZ_SERVER}:1055 \
-L ${DEAMON_PORT}:${HE_LIZENZ_SERVER}:${DEAMON_PORT} \
${HE_USER_ID}@${HE_COM_SERVER}
# export license environment variables
export ANSYSLMD_LICENSE_FILE=1055@localhost
export ANSYSLI_SERVERS=2325@localhost
# create hostslist
export jms_nodes=`srun hostname -s`
export hostslist=`echo $jms_nodes | sed "s/ /,/g"`
cfx5solve -batch -def $INPUT -par-dist ${hostslist}
# close the SSH control socket
ssh -S ${SOCKET_NAME} -O exit ${HE_USER_ID}@${HE_COM_SERVER}
echo "Run completed at "
date |
Download CFX Script
