Jtdcftul

I can't seem to run MPI jobs using slurm. Any help or advice?

Running a local home based mini cluster to use all of my processors. I'm using 18.04, and have installed the stock openmpi and slurm packages. I have a small test program that I use to show which cores I'm running on. When I run with mpirun I get:

$ mpirun -N 3 MPI_Hello

Process 1 on ubuntu18.davidcarter.ca, out of 3

Process 2 on ubuntu18.davidcarter.ca, out of 3

Process 0 on ubuntu18.davidcarter.ca, out of 3

When I run with srun I get:

$ srun -n 3 MPI_Hello

Process 0 on ubuntu18.davidcarter.ca, out of 1

Process 0 on ubuntu18.davidcarter.ca, out of 1

Process 0 on ubuntu18.davidcarter.ca, out of 1

I've done this many times with different arguments (--mpi=pmi2, --mpi=openmpi, etc) and can confirm that instead of running a job with n parallel threads it runs n single threaded jobs. n times the work with 1/n times the expected resources per job.

This is my test program:

#include <stdio.h>

#include <mpi.h>



int main(int argc, char** argv) {

        int numprocs, rank, namelen;

        char processor_name[MPI_MAX_PROCESSOR_NAME];



        MPI_Init (&argc, &argv);

        MPI_Comm_size(MPI_COMM_WORLD, &numprocs);

        MPI_Comm_rank(MPI_COMM_WORLD, &rank);

        MPI_Get_processor_name(processor_name, &namelen);



        printf("Process %d on %s, out of %dn", rank, processor_name, numprocs);



        MPI_Finalize();

}

This is my /etc/slurm-llnl/slurm.conf:

# slurm.conf file generated by configurator easy.html.

# Put this file on all nodes of your cluster.

# See the slurm.conf man page for more information.

#

ControlMachine=ubuntu18.davidcarter.ca

#ControlAddr=

#

#MailProg=/bin/mail

MpiDefault=none

MpiParams=ports=12000-12100

ProctrackType=proctrack/pgid

ReturnToService=1

SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid

#SlurmctldPort=6817

SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid

#SlurmdPort=6818

SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd

SlurmUser=slurm

#SlurmdUser=root

StateSaveLocation=/var/lib/slurm-llnl/slurmctld

SwitchType=switch/none

TaskPlugin=task/affinity

#

#

# TIMERS

#KillWait=30

#MinJobAge=300

#SlurmctldTimeout=120

#SlurmdTimeout=300

#

#

# SCHEDULING

FastSchedule=1

SchedulerType=sched/backfill

#SchedulerPort=7321

SelectType=select/linear

#

#

# LOGGING AND ACCOUNTING

AccountingStorageType=accounting_storage/none

ClusterName=cluster

#JobAcctGatherFrequency=30

JobAcctGatherType=jobacct_gather/linux

#SlurmctldDebug=3

SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log

#SlurmdDebug=3

SlurmdLogFile=/var/log/slurm-llnl/slurmd.log

#

#

# COMPUTE NODES

#NodeName=compute[1-4] Sockets=1 CoresPerSocket=2 RealMemory=1900 State=UNKNOWN

#NodeName=compute[1-2] Sockets=1 CoresPerSocket=4 RealMemory=3800 State=UNKNOWN

NodeName=compute1 Sockets=8 CoresPerSocket=1 RealMemory=7900 State=UNKNOWN

NodeName=ubuntu18 Sockets=1 CoresPerSocket=3 RealMemory=7900 State=UNKNOWN



#

# Partitions

PartitionName=debug Nodes=ubuntu18 Default=YES MaxTime=INFINITE State=UP OverSubscribe=NO

PartitionName=batch Nodes=compute1 Default=NO MaxTime=INFINITE State=UP OverSubscribe=NO

PartitionName=prod Nodes=compute1,ubuntu18 Default=NO MaxTime=INFINITE State=UP OverSubscribe=NO