- To use Eddie's system MPI. This is the preferred way as I'd expect it to be fine-tuned for ECDF hardware. I'm using the wiki page as guidance, but it seems to be outdated since I can't even load the modules I need:
- Install and use the latest version of openmpi. So far it fails to install with this message:
Think it's missing some libraries, or whatever, but should be fixablear: ***/topographica/external/openmpi-1.5.3/ompi/contrib/vt/vt/util/.libs/libutil.a: No such file or directory
$ module add intel/cce
ModuleCmd_Load.c(199):ERROR:105: Unable to locate a modulefile for 'intel/cce'
$mpicc -v
-bash: mpicc: command not found
Ideally I'd want to use both and see the difference in performance
[update] Fixed. This issue with openmpi has been raised before, cleaning up all traces of the old install and re-configuring resulted in clean build.
[update] Fixed. This issue with openmpi has been raised before, cleaning up all traces of the old install and re-configuring resulted in clean build.
Just contact ECDF's support email address (or maybe it's a web form); they have very responsive tech support and getting "module add" to work should be very quick for them.
ReplyDeleteBTW, please post a sample graph and explanation for me to study when I get a chance.
You should email this address:
ReplyDeleteis.helpline@ed.ac.uk
In the subject, make it clear that eddie/ecdf is the topic.
Usually you get a quick and helpful response.
Chris
P. S. I guess the documentation is particularly out of date at the moment because they just did a big upgrade, didn't they? Are you subscribed to the ecdf-users mailing list?
Could it be something like this:
ReplyDeletemodule add openmpi/infiniband/intel/latest
or e.g.
module add openmpi/infiniband/gcc/latest
(from https://www.wiki.ed.ac.uk/display/ecdfwiki/Using+Environment+Modules).
Or you could try:
module available
to see if you can see a suitable looking one to pick.
There could be some useful advice in the Eddie Mark 2 quickstart:
https://www.wiki.ed.ac.uk/display/ecdfwiki/Eddie+Mark+2+-+User+instructions
Or maybe in the mpi tutorials:
https://www.wiki.ed.ac.uk/display/ecdfwiki/MPI+Tutorials
Or maybe the list of environments is helpful:
https://www.wiki.ed.ac.uk/display/ecdfwiki/Parallel+Environments
Chris
Try:
ReplyDeletemodule avail
This should tell you what modules you can load and you should be able to find the correct one for mpi.
adrian
fantastic! thanks. Also, by any chance do you have any mpi batch job submission script for Eddie that I could use as an example? The simpler the better. Cheers
ReplyDeleteThere seems to be an example job script here:
ReplyDeletehttps://www.wiki.ed.ac.uk/display/ecdfwiki/OpenMPI+-+openmpi+over+infiniband
(including the source code of a sample mpi program).
Chris
thanks! unfortunately it doesn't work, although it might be due to the modules. I'm working on it
ReplyDeleteHaving experimented with eddie I would suggest the following for compiling an mpi code:
ReplyDeletemodule load intel/compiler
module load openmpi
then mpicc should be available. Note that this loads the infiniband mpi by default, so you might want to change this to the gigabit ethernet infiniband depending on the number of processors/cores you want to use.
Assuming you've used the above then you should be able to submit the following script to run your job (where my program is called sharpen):
#!/bin/sh
#$ -N sharpen
#$ -cwd
#$ -pe openib_smp12_qdr 12
#$ -R y
#$ -l h_rt=00:10:00
. /etc/profile.d/modules.sh
module load intel/compiler
module load openmpi
mpirun -np $NSLOTS ./sharpen
This runs my program on 12 cores for a maximum of 10 minutes
Note that the infiniband stuff requires you to use multiples of 12 cores for you jobs. If you want use less than 12 cores you can either use the above script and replace $NSLOTS with the actual number of cores you want to use (providing it's less than the multiple of 12 you've specified in the -pe line), or you need to use the gigabit ethernet library instead. If you want to use gigabit ethernet I can provide example scripts for that too, let me know.
cheers
adrianj
That's awesome! Thanks. A few questions:
ReplyDelete1) What is the main difference between infiniband and gigabit ethernet?
2) If I want to run my code in pure MPP mode, i.e. using only 1 core per machine (given the number of nodes can be up to 100) how do I need to modify the script?
3)Yes, could you please post here or email me the script for gigabit ethernet, I think I'll need it
Cheers,
Konstantin
Infiniband is a very high performance network, get much lower latencies and higher bandwidth than gigE.
ReplyDeleteFor Gigabit Ethernet use the following for compiling:
module load intel/compiler
module load openmpi/ethernet
And this script for running:
#!/bin/sh
#$ -N sharpen_ge
#$ -cwd
#$ -pe openmpi_* 8
#$ -l h_rt=00:10:00
. /etc/profile.d/modules.sh
module load intel/compiler
module load openmpi/ethernet
mpirun -np $NSLOTS ./sharpen
(this will run on 8 cores)
If you want to under-subscribe nodes you should be able to specify it in the mpirun line, something like:
mpirun -n TOTALNUMBER -npernode 1
where the number you put in the request line, i.e.:
#$ -pe openmpi_* 8
should match the total number of cores you are requesting (i.e. the number of nodes*number of cores per node).
However, I've not done this on eddie yet so it may require slightly different syntax, I'll check.
cheers
adrianj
great, thanks! should keep me going for now
ReplyDeleteOk, to under-populate nodes on Eddie you need the following script:
ReplyDelete#!/bin/sh
#$ -N sharpen_ge
#$ -cwd
#$ -pe openmpi_* 24
#$ -l h_rt=00:10:00
. /etc/profile.d/modules.sh
module load intel/compiler
module load openmpi/ethernet
mpirun -np 2 --bynode ./sharpen_ge
The above uses 2 nodes, 1 process on each nodes, and uses the gigabit ethernet network.
To do the same for infiniband use the following:
#!/bin/sh
#$ -N sharpen_ib
#$ -cwd
#$ -pe openib_* 24
#$ -l h_rt=00:10:00
. /etc/profile.d/modules.sh
module load intel/compiler
module load openmpi
mpirun -np 2 --bynode ./sharpen_ib
fantastic! would be great to know some "magic word" to get turnaround time to something reasonable. However, it could be some temporary issue, I'll try using eddie again tonight
ReplyDeleteI found turnaround wasn't so bad today on both the gigabit and infiniband resources.
ReplyDeleteOne thing I should have mentioned is that the --bynode flag uses round robin allocation of processes, so if for example you are using 2 nodes then MPI rank 0 goes onto node 1 and rank 1 onto node 2 then if you've a rank 3 it would be on node 1 and rank 4 node 2, etc...
This is often an issue for performance benchmarking although I think for your program it's not a big deal.
adrianj