progress update

Finally fixed the parallel version of the topographica script and completed the timing code. Giant thank you to Chris for helping out with the CF distribution. As soon as I wake up - hector and new graphs.

[update] not quite as soon...

HECToR, 15 nodes, 4 cores per node, 100x100 density, 1000 iterations

Just remembered that I wanted to post the graphs for this run:

Timings first:
 
Looks like even on 60 cores computation still takes most of the run time.

Speedup:


It seems that 52 cores (13 nodes) provide the best efficiency and it doesn't make sense to use any more processors for this density.

Parallel Efficiency:
cpu_cores: 1    E = 99.2%
cpu_cores: 4    E = 97.5%
cpu_cores: 8    E = 97.3%
cpu_cores: 12   E = 96.7%
cpu_cores: 16   E = 96.5%
cpu_cores: 20   E = 96.0%
cpu_cores: 24   E = 92.9%
cpu_cores: 28   E = 94.7%
cpu_cores: 32   E = 91.0%
cpu_cores: 36   E = 86.9%
cpu_cores: 40   E = 93.3%
cpu_cores: 44   E = 90.2%
cpu_cores: 48   E = 87.8%
cpu_cores: 52   E = 87.4%
cpu_cores: 56   E = 80.5%
cpu_cores: 60   E = 76.4%

HECToR, 100x100, 1000 iterations, 7 nodes (1-168 cores)

In this experiment I was using 7 MPP nodes, subscribing from 1 to 24 cores on each node. The results are quite interesting. First, the speedup:

Some quick conclusions:

• there is only so much that the code can scale (at least in MPP+SMP mode)
• there is a "peak performance" point somewhere between 112 and 168 cores after which it doesn't make anymore sense to add any more computational power since it negatively affects the performance
• best performance can be achieved if nodes are undersubscribed
• up to 4 cores used per node can give very close to linear performance. Let's look at parallel efficiency values:

cpu_cores: 1    E = 99%
cpu_cores: 7    E = 99%
cpu_cores: 14   E = 97%
cpu_cores: 28   E = 94%
cpu_cores: 56   E = 81%
cpu_cores: 112  E = 68%
cpu_cores: 168  E = 37%

I will have to verify the last point in the next experiment where I run the code on 1-60 processors, subscribing only up to 4 CPUs per node. In the current experiment 28 cores (4 per node) provide 94% efficiency which sounds fairly good to me.

Next, parallel timings:

It is clear from the graph that starting from 4 cores per node processes start under-performing and create visible load imbalance. There is no actual load imbalance: all nodes have exactly the same amount of work to do, but probably due memory accessing bottlenecks some processors wait longer to get their data from memory while others have to wait in the collective routing for the former to complete their computations. In addition to that, the increasing time spent in the MPI_Gather collective is definitely not representing the real communication overheads: if it was then the broadcasting time would be increasing as well because the amount of data broadcast by node 0 on each iteration is greater than the amount of data it has to gather by the corresponding collective routine. The only explanation for why broadcasting time is very small is that it happens after another collective - MPI_Gather, which implicitly synchronises all nodes, and the actual communication time for this routine is very small.

Hector, 100x100, 50 to 500 iterations

Conclusions:

• Although somewhat deviating randomly from run to run by 1-2 seconds, the overhead is static (i.e. does not increase for longer simulations)
• As such, it shouldn't be worried about

Job launching overhead

Using minimum density of 1 neuron on one node:

Conclusions:

• The overhead is somewhat random 
• The overhead doesn't seem to depend on the number of cores used on the same node
• Finally, the overhead can be anywhere between 0.5 and 4 seconds

Next I'm going to try and see if the overhead stays the same while the program run time increases.

Results from Hector

SMP run on one 24-core node


Overall timing capped by the wallclock timer (/usr/bin/time)

 Timings for activation only:

Finally, the speed-up (parallel activation against serial):

I should definitely try the same density on a larger number of processors to see if the speed-up tails off at some point and when.

Next, results obtained with a 200x200 simulation on 4 nodes, using up to 96 cores:

It seems to be achieving only sub-linear speed-ups, although the scaling is pretty good. I need to try a smaller density with a larger number of processors to see what can really be achieved in terms of performance. However, it seems that over a 100 times the serial is quite realistic, given there's enough CPUs in use.

My own version of deleting a line from a file in C

Modified version of this: http://bhagwat-masalkar.blogspot.com/2008/10/delete-line-in-file-c-programming.html

   
    FILE *f;
    f = fopen(argv[4],"a");//filename as command line argument                                                                                

    fseek(f, 0, SEEK_END); //                                                                                                                 
    if (ftell(f) == 0)//new file, add opening root xml tag                                                                                    
      fprintf(f,"<runs>\n");
    else{ // file wasn't empty, delete the closing root xml tag                                                                               
      fclose(f);

      char tmp_str[100];
      FILE *temp;

      f = fopen(argv[4],"r");
      temp = fopen("timings.tmp.xml","wb");

      while(fgets(tmp_str,100,f) != NULL){
        if(strstr(tmp_str,"</runs>"))
          break;
        else
          fputs(tmp_str,temp);
      }

      fclose(f);
      fclose(temp);
      char * command = //concatenating strings to build up a unix command                                                                     
        malloc(snprintf(NULL, 0, "mv timings.tmp.xml %s", argv[4]) + 1);
      sprintf(command,"mv timings.tmp.xml %s",argv[4]);
      system(command);
      f = fopen(argv[4],"a");
    }

    // ... and this is why I hate C sometimes

timings.0.xml

Here's what an entry in my timings files might look like:

<run size='2' density='400' m='20' n='20' iterations='10'>
  <main_tot name='main: Total'>1.05683208</main_tot>
  <main_utils name='main: Utils'>1.04948020</main_utils>
  <main_create_cfs name='main: Creating CFs'>0.00234818</main_create_cfs>
  <main_seq_act name='main: Serial Activate'>0.00202990</main_seq_act>
  <main_mpi_act name='main: Parallel Activate'>0.00296593</main_mpi_act>
  <act_seq_tot name='serial: Total'>0.00202894</act_seq_tot>
  <act_seq_utils name='serial: Utils'>0.00002003</act_seq_utils>
  <act_seq_dp name='serial: Dot Product'>0.00197291</act_seq_dp>
  <act_seq_of name='serial: Output Function'>0.00001287</act_seq_of>
  <act_mpi_tot name='parallel: Total'>0.00296402</act_mpi_tot>
  <act_mpi_utils name='parallel: Utils'>0.00016284</act_mpi_utils>
  <act_mpi_distr_weights name='parallel: Distributing Weights'>0.00114608</act_mpi_distr_weights>
  <act_mpi_bc name='parallel: Broadcasting Inputs'>0.00003695</act_mpi_bc>
  <act_mpi_dp name='parallel: Dot Product'>0.00095177</act_mpi_dp>
  <act_mpi_gather name='parallel: Gathering Outputs'>0.00061488</act_mpi_gather>
  <act_mpi_of name='parallel: Output Function'>0.00001335</act_mpi_of>
</run>

HECToR and my humble app

Just finished pulling my code from the repository. Edited my Makefile (for C Activation, not Topographica) to use cc (Cray's C compiler) instead of gcc. Compiling failed at first complaining about some mpi stuff but suddenly started working (I didn't get how I fixed it, but whatever). Time to make a first run!

Side note: it actually feels pretty cool to have 3 terminal windows all ssh'ed into different supercomputers, one of which being UK national service :P

Submitted my job. 1 core on one processor with a small density size, just to check if it works. Wasn't sure which budget to use, so decided to go with the one we used for our msc practicals (it's the only one I know anyway). Job submitted ok and just finished running. Error. Why does everything that has to do with programming never seems to work the first time you run it? "I don't know, said Hector, but you're..."

Attempting to use an MPI routine before initializing MPICH

I'm just grateful it attempted to run my script and there were no problems with the budget. Ok, how the hell do I initialise MPICH?

Tried loading xt-mpich2/5.1.4 module, it conflicted with xt-mpi, so I had to unload it. What is mpt anyway? What's really bothering me (after googling up the problem) is that probably the error is due to the fact that I compiled the code with a set of libraries different from the ones used for running my code. I wonder why. I used cc (default compiler on Cray) and now using default MPI libraries and that just doesn't seem to work.

I looked up my PD and AT notes: we did use Hector for exercises, but it was for co-arrays and UPC. Couldn't find anything on MPI. Voodoo-ing again...

[update] I got it! So ridiculous... Basically, I was calling MPI_Wtime() before MPI_Init() and after MPI_Finalize. While openmpi allows doing it, mpich seems not to. Glad I've solved it.

And here's the result of tonight's effort (using 1 to 24 cores of one node, i.e. full SMP):

The speed-ups and parallel efficiency values for dot-product are (I'll improve my plotting script to display speed-ups soon):
2 cores: x1.99 (E=0.99)
4 cores: x3.91 (E=0.9775)
8 cores: x6.40 (E=0.8)
16 cores: x11.87 (E=0.74)
24 cores: x20.94 (E=0.87) <- seems to be hitting the cache, or at least close to

HECToR and Git

Having talked to Adrian today, I'm switching to HECToR from ECDF to do my benchmarking experiments. ECDF's turn-around time is just horrible (at least for me!), hopefully the national service will be better. First of all I am going to see how my C code scales on one 24-core die by running it on 1, 2, 4, 8, 16 and 24 cores. The largest number that shows linear or close enough to linear performance improvement will be used in future for distributed-memory experiments: e.g. if 8 cores on one node provide close to 8 times better performance than 1, and then it tails off at 16 and 24, I will use 8 cores per die for scaling experiments. Thus, if the maximum number of cores that I want to use to get an idea of my app's scaling properties is 100, then I will have to use 12 dies, using 8 cores per die (96 in total) and wasting 16 (192 in total).

At the moment I'm experiencing difficulties with installing git on hector. It won't compile, giving me the following message:

make
GIT_VERSION = 1.7.6
    * new build flags or prefix
    CC daemon.o
PGC-W-0095-Type cast required for this conversion (./git-compat-util.h: 254)
PGC-S-0137-Incompatible prototype declaration for function inet_ntop (./git-compat-util.h: 413)
PGC/x86-64 Linux 10.9-0: compilation completed with severe errors
make: *** [daemon.o] Error 2

looks like there's some problem with the C compiler. I tried changing the makefile to explicitly use gcc (it's said to build fine with it, not sure about other compilers. Besides, I managed to build it on ECDF without any trouble), but no luck. At the moment I'm trying out some voodoo stuff, if nothing works I'll just create a new svn repository for topographica and use it as version control between hector and jupiter.

[update] I think I slightly progressed. Specifying "gcc all" as CC option got me a bit further:

make CC=gcc all

Now most of it (I'd like to believe) compiles but breaks once it gets to http.o:

CC http.o
In file included from http.c:1:
http.h:6:23: error: curl/curl.h: No such file or directory
http.h:7:23: error: curl/easy.h: No such file or directory

and a giant bunch of errors after.

[update 2] Fantastic! The solution was found here. In particular, I set a few environment variables:

NO_CURL=1 NO_EXPAT=1 NO_SVN_TESTS=1 NO_TCLTK=1 make CC=gcc all 

Make completes without errors now all that's left is make install.....

NO_CURL=1 NO_EXPAT=1 NO_SVN_TESTS=1 NO_TCLTK=1 make CC=gcc all install

and ...

git --version
git version 1.7.6

feels good :)

Graphs (Jupiter)

Long awaited graphs! I haven't managed to run my code on ECDF yet, so for now I've only got data for Jupiter (cluster of 3 8-core boxes). It would be too early to make any comments on scaling, but at least we can see what the graphs will look like.

The problems size used here was 22500 (i.e. 150x150 sheets), along the X dimension is the number of processors (MPI_Size), along Y - time. Components timed:

#real - the overall runtime (taken from /usr/bin/time)
main_mpi_act - MPI Activation time
main_seq_act - same for the serial
main_create_cfs - initialising connection fields
main_utils - irrelevant stuff

First observations:

• I should give more meaningful names to labels (i.e. "Parallel Activation time" instead of main_mpi_act)
• Parallel time going up with the number of processors??? Really? However, the reason for this will be clear from the next graph

act_mpi_tot - total time spent inthe parallel activation code
act_mpi_of - output functions (the hysteresis function in my case)
act_mpi_gather - gathering output activities
act_mpi_dp - computing dot-product
act_mpi_bc - broadcasting the input matrix
act_mpi_distr_weights - distributing the initial sets of weights
act_mpi_utils - shizzle

By just looking at this I can tell that optimising initial weights distribution could be a lot more important than we thought. Of course it's a one-off operation and it's contribution to the overall time is going to be smaller for longer simulations (mine were only 20 iterations long), but it's still massive! Anyway, here's the kind of graph we're probably most interested in (same as above, but without weights distribution):

The only values of X (MPI_Size) worth attention here are 1,2 and 3 as only these are pure distributed-memory runs (MPP), the rest are mixed SMP-MPP (i.e. more than 1 core used per machine) and are less important for now.

Dot product seems to scale linearly, but with the addition of MPI calls the overall activation time (without weights distribution) scales somewhat sub-linearly. Anyway, I won't make any conclusions about scaling until I get my results from ECDF. Besides, these results cannot be trusted since someone was using half of Jupiter cores while I was running my experiments.

As a side note: it would probably make sense to optimise the weights distribution process in my C code. It shouldn't be hard and will save a lot of CPU time on ECDF.


[update] Implemented local weights initialisation: now each node creates it's own set of weights instead of doing it on the master node and then broadcasting this massive set of data to all nodes. Suddenly the graphs look a lot nicer (also, no one was using jupiter except me this time). Same as before, 150x150 problem size, but this time 50 iterations instead of 20.

Overall run statistics:
 MPI only:

progress update

Finished the script for making good-looking component scaling graphs. At the moment working on getting my C code to compile and run on ECDF. Basically, I have two options:

1. To use Eddie's system MPI. This is the preferred way as I'd expect it to be fine-tuned for ECDF hardware. I'm using the wiki page as guidance, but it seems to be outdated since I can't even load the modules I need:

$ module add intel/cce
ModuleCmd_Load.c(199):ERROR:105: Unable to locate a modulefile for 'intel/cce'

It looks like it is necessary to load some modules for MPI to work:

$mpicc -v
-bash: mpicc: command not found

Adrian, I need your help here
2. Install and use the latest version of openmpi. So far it fails to install with this message:
   

   
   ar: ***/topographica/external/openmpi-1.5.3/ompi/contrib/vt/vt/util/.libs/libutil.a: No such file or directory
   

   Think it's missing some libraries, or whatever, but should be fixable
   

Ideally I'd want to use both and see the difference in performance

[update] Fixed. This issue with openmpi has been raised before, cleaning up all traces of the old install and re-configuring resulted in clean build.