new pmi

What should I do with the optimised PMI module? (I mean what would be the best place to put it?)

Merging

topographica_mpi is now history. topographica --mpi has come to replace it. Working on merging with the current SVN version at the moment.

up

Weights generation problem is fixed now and the parallel code produces correct results: I modified the timed_topographica.sh script to work with lissom and it shows that the activity of V1 is the same in serial and in parallel. Next I need to move pmi.init into the commandline module to get rid of topographica_mpi once and for all and then - diff-ing and merging with the svn version.

Sorry for being so slow - I really do have very little free time. I remember someone said that while working in a start-up I won't have life. Well, yes.

I'm back

quick update: looks like I've cleaned up most of the cruft in the code, however now have to deal with error messages from cf.py that still tries to generate weights in parallel. On a bright side, I've run tests with lissom_oo_or_mpi.ty (before getting rid of the parallel weights generation code on the mpi-side) and it seems to be working fine. Anyway, the short-run plan is to fix the bug with weights, then move the pmi initialisation code away from the topographica launch script and then run some tests for correctness with lissom. Hoping to achieve that tomorrow.

Performance results for PMI-optimised version

Slightly better than it used to be.

Parallel Efficiency:
cpu_cores: 1    E = 99.5%
cpu_cores: 2    E = 96.6%
cpu_cores: 4    E = 91.7%
cpu_cores: 8    E = 79.7%
cpu_cores: 16   E = 75.3%
cpu_cores: 24   E = 67.6%

And simulation total against node activation:

Topograhica: topo.sim.run time and MPI node activation time

The gap between the total simulation time (topo.sim.run()) and activation on the nodes (MPI_CFProjection_node.activate()), as we observed in C-Topo simulations, ideally should be irrelevantly small. Proof is here:

main: Parallel Activate (the black line) is the equivalent of Simulation Total in Topographica. Everything else (the colour-filled areas), combined, is the equivalent of MPI node activation. It's clear that the activation time in C is
a) completely dominated by the dot-product computation
b) takes about as much time as the total simulation.

Not the case with Topographica. I will assume that I have this bit nailed. Another thing to notice on the first figure is that the time difference between simulation total and activation increases with the number of cores used: there is almost none on 2 cores and it becomes quite visible on 24. I think because of this the overhead is unlikely to come from the serial bits of the application. What else? The only thing that comes to my mind is communication overheads which, in turn, could be due to one of two reasons: inefficient use of MPI routines (unlikely, this was an SMP run on 1 node, there wasn't supposed to be any "real" communication involved) or .... something else (I was thinking hard what to put here, got nothing so far). Anyhow, I need to test this assumption first, which won't be too hard: I'll place my timers in PMI's invoke and call methods that manage all communication in parallel topographica and see how much time is being spent there.

Topographica: Serial vs Parallel on 1 core

  <run n='100' m='100' iterations='1000' cpu_cores='1' density='10000'>
    <sim_tot_mpi name='Simulation total'>214.378264904</sim_tot_mpi>
    <sim_tot_seq name='Simulation total'>212.18565011</sim_tot_seq>
  </run>

it doesn't look like PMI/mpi4py adds much static overhead.

Detailed timings

Some more detailed timings for Topographica:

"Simulation total" 16.2974710464
"MPI_CFP.activate()" 13.5669333935
"MPI_CFP_n.activate()" 11.1868505478
"MPI_CFP_n.activate():response_fn" 11.1458990574

MPI_CFP.activate() - the activate function of MPI_CFProjection class. This class contains serial code and through PMI calls functions of MPI_CFProjection_node
MPI_CFP_n.activate() - the activate function of MPI_CFProjection_node (parallel CFProjection)

MPI_CFP_n.activate():response_fn - response function call (response_fn mainly consists of the dot-product implementation)

Here we can see that most of the parallel activation time is spent doing dot-products (11.15 s out of 11.19). Parallel activation takes 11.19 seconds out of 13.57 seconds (or 82%) of serial-side activation call which consists only of the PMI call (the bridge between the serial and parallel implementations: it distributes input data to nodes and invokes their methods specified in the PMI call) and re-combining the data. Here's the source code for this function:

    
def activate(self, input_activity):
        activity_list = pmi.invoke(self.pmiobj,'activate',input_activity)

        self.activity = numpy.array([])
        for activity_row in activity_list:
            self.activity = numpy.append(self.activity, activity_row)
        self.activity = self.activity.reshape(self.activity_shape[0],self.activity_shape[1])

The missing 18% are probably lost communicating doing some python stuff. In all that wouldn't be too bad if not the fact that serial-side activation takes 83% of the total Simulation time while the parallel-side activation contributes to 68% of the total topo.sim.run time. This means that about 32% of simulation time is lost doing something unknown. I don't think it's likely that this is pure Python overhead because in this post, comparing serial versions of Topographica's matchmodel and C-Topo I've shown that the overhead is fairly constant and small. For example in this run of a 100x100 density, Topographica is slower than C only by 2 seconds:
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 100

Serial: 18.9039599895
=== Sequential Time ===
Dot Product: 16.5311
-----------------------
Total: 16.5367

"Serial" is topographica's time and the one at the bottom is for C-Topo.

Here's some sample output of C-Topo with the same density, same number of processors (I left only the important entries):

  <run n='100' density='10000' m='100' iterations='1000' cpu_cores='24'>
    <main_mpi_act name='main: Parallel Activate'>10.80980301</main_mpi_act>
    <act_mpi_dp name='parallel: Dot Product'>9.99707174</act_mpi_dp>
  </run>

The total simulation time here is main_mpi_act, 10.81 seconds. 9.997 seconds of which is spend doing dot-products. That is 92%, as opposed to Topographica's 68% which more than 20% less efficient. The challenge now is to figure out where this inefficiency comes from.

C vs Python. Hector, 13 nodes, 1-52 cores

A few more graphs.

Topographica:

C-Topo

I'm not sure what's causing performance go down at 52 cores for C. Last time in a similar experiment (except with 10000 density instead of 14400) 52 cores were showing very good efficiency. This could be a one-off glitch. Anyway, the point of this post is: topographica is still worse, even when subscribing only 4 cores per node.

C vs Python. Hector, 1 node SMP, 1-24 cores

Speedups:

Topographica:

C-Topo:



Parallel Efficiencies:

                Topographica  C-Topo
cpu_cores: 1    E = 99.0%     E = 99.5%
cpu_cores: 2    E = 95.4%     E = 98.3%
cpu_cores: 4    E = 89.3%     E = 96.3%
cpu_cores: 8    E = 78.4%     E = 91.7%
cpu_cores: 16   E = 66.1%     E = 90.6%
cpu_cores: 24   E = 55.4%     E = 86.2%

script output

luckily missing module warnings disappeared after I removed module swaps from .bashrc

TOPOGRAPHICA RUNS

### Topographica ###
Application 1013295 resources: utime ~1199s, stime ~63s real 83.19 user 0.00 sys 0.00

### C-Topo ###
Application 1013303 resources: utime ~301s, stime ~3s real 14.70 user 0.18 sys 0.01

### Comparing the results of c-topo and topographica ###
Results matched

Graphs will shortly follow, first for single-node run.

update

Good news and somewhat confusing news. The good news is: it finally runs and produces timings!!! I couldn't be happier about it. The confusing news is this: I have no idea why it runs, and the output it produces is even more confusing (script output,not the timings; timings are fine). When I first tried running topographica from my /work directory it ran but probably froze, because it only completed when it ran out of allocated time (10 minutes). Here's what I found in the output file:

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

Of course they are not loaded! I swapped PrgEnv-pgi for PrgEnv-gnu and xt-mpt for xt-mpich2 (mpi4py wouldn't build otherwise). Then I tried swapping the modules back, without recompiling mpi4py. And it worked! Why did it not segfault? Apparently, this means that you can build mpi4py with one set of MPI libraries and run with another?.. However, that's not it: despite the correct execution, the new output file still contained

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

huh? Ok, I'm just happy it runs, at last. I have to admit, making topographica run on Hector was a royal pain in the...

the full error output

messages between these ">>>>>" "<<<<<" are printed exceptions that break execution, I added the print statements into weave/catalog.py:

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04' <<<<< >>>> [Errno 17] File exists: '/home/d04' <<<<< Traceback (most recent call last): File "/var/spool/alps/1009053/topographica_mpi", line 15, in <module> process_argv(argv[1:]) File "/esfs2/d04/d04/s0676011/topographica/topo/misc/commandline.py", line 474, in process_argv execfile(filename,__main__.__dict__) File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ cp.pbs.o321605 create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/basic.py", line 31, in <module> from topo.base.patterngenerator import PatternGenerator,Constant File "/esfs2/d04/d04/s0676011/topographica/topo/base/patterngenerator.py", line 238, in <module> from topo.misc.inlinec import inline,c_header File "/esfs2/d04/d04/s0676011/topographica/topo/misc/inlinec.py", line 59, in <module> import weave File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/__init__.py", line 9, in <module> from blitz_tools import blitz File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/blitz_tools.py", line 11, in <module> import inline_tools File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/inline_tools.py", line 15, in <module> function_catalog = catalog.catalog() File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 351, in __init__ sys.path.append(default_dir()) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 202, in default_dir create_dir(path) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 131, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Weave

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<<

The directory exists (I mkdir'ed it), obviously. I even chmod'ed 777 permissions for it. Seems like the root of the problem is the same as with matplotlib: Hector's backend can't access files and directories on the frontend. Except for unlike with matplotlib, I can't tell topographica to ignore Weave

Are there any shared directories I could make use of? Otherwise, I'm really puzzled...

One thing I don't understand: my C app creates an xml file (timings), which I'm quite sure it does on the backend (the file is created from withing the main body of the code) and that does not cause any trouble with permissions. However, for python libraries it does - could the problem be somewhere else?

Weird error message

Not sure what's going on there (from somewhere in the middle of the same error output as the last message in the previous post was taken from):

File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info

What is it importing from basic?.. That's the contents of topographica's root directory

Topographica on Hector

Today Adrian helped me out with the libgfortran bug: the library I was looking for was inside one of the subdirectories of gcc, namely /opt/gcc/4.5.2/snos/lib. I have no clue why find /opt/ -name 'libgfortran.so.3' wasn't picking it up.


The next issue - topographica not being able to find it's source code (the topo module). This was fixed by adding  the following line into my .pbs script:

export PYTHONPATH=$PBS_O_WORKDIR/topo:$PYTHONPATH

The next issue was matplotlib unable to create .matplotlib directory anywhere:

RuntimeError: Failed to create /home/d04/d04/s0676011/.matplotlib; 
consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data

I tried $HOME, $PBS_O_WORKDIR, work and topographica directories and even ./  . Nothing helped, I'm still not sure why. Neither did any solutions I found on the web. What fixed that problem was tracing the import and adding exception to the file that was importing matplotlib (topo/misc/commandline.py):

try:
    # Import matplotlib, if available, and use a non-GUI backend by default
    import matplotlib
    matplotlib_imported=True
    from matplotlib import rcParams
    rcParams['backend']='Agg'
except ImportError:
    print "Warning: Could not import matplotlib; pylab plots will not work."
except RuntimeError as e: #KKALERT: added this to be able to run on hector                   
    print "Warning: pylab plots will not work:",e

Thing I'm stuck with at the moment:

File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 129, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Starting to think that launching Topographica on Hector is a project on it's own...

mpi4py and hector

Trying to compile mpi4py with gcc didn't get me anywhere, even after following what I was suggested at the mpi4py google group. I did manage after all to compile it with gcc after swapping several libraries. Now however, I'm stuck with this:

Traceback (most recent call last):
  File "mpitest.py", line 1, in <module>
    from mpi4py import MPI
ImportError: libgfortran.so.3: cannot open shared object file: No such file or directory

I've spent a whole day trying fix this and didn't get any further this error message. The library just isn't there and I'm not 100% sure that this is what mpi4py is trying to find. Why does it even need fortran's libraries? i'm stuck...

mpi4py and Cray's compiler

They don't seem to work together. Basically, the problem is that mpi4py is by default compiled with gcc, however I need to compile it with cc instead to be able to run parallel jobs with aprun. Now, cc gives a bunch of unrecognised compiler flag options (it doesn't understand gcc's flags) when trying to build mpi4py with it. I think I tried pretty much everything I could with what I found on the web to make it work, but no luck so far... I think the options now are:

1) get mpi4py to compile with cc
2) use MPI that is based on gcc (OpenMPI? MPICH2?) if that's possible on Hector.

I posted about this on mpi4py's google group board, seems like they already had to deal with this issue before, so hoping to get some help from there.

Also, am I getting it right: cc actually uses Portland's compiler (pgcc) under the hood? It appears so from the error messages I'm getting

progress update

Finally fixed the parallel version of the topographica script and completed the timing code. Giant thank you to Chris for helping out with the CF distribution. As soon as I wake up - hector and new graphs.

[update] not quite as soon...

HECToR, 15 nodes, 4 cores per node, 100x100 density, 1000 iterations

Just remembered that I wanted to post the graphs for this run:

Timings first:
 
Looks like even on 60 cores computation still takes most of the run time.

Speedup:


It seems that 52 cores (13 nodes) provide the best efficiency and it doesn't make sense to use any more processors for this density.

Parallel Efficiency:
cpu_cores: 1    E = 99.2%
cpu_cores: 4    E = 97.5%
cpu_cores: 8    E = 97.3%
cpu_cores: 12   E = 96.7%
cpu_cores: 16   E = 96.5%
cpu_cores: 20   E = 96.0%
cpu_cores: 24   E = 92.9%
cpu_cores: 28   E = 94.7%
cpu_cores: 32   E = 91.0%
cpu_cores: 36   E = 86.9%
cpu_cores: 40   E = 93.3%
cpu_cores: 44   E = 90.2%
cpu_cores: 48   E = 87.8%
cpu_cores: 52   E = 87.4%
cpu_cores: 56   E = 80.5%
cpu_cores: 60   E = 76.4%

HECToR, 100x100, 1000 iterations, 7 nodes (1-168 cores)

In this experiment I was using 7 MPP nodes, subscribing from 1 to 24 cores on each node. The results are quite interesting. First, the speedup:

Some quick conclusions:

• there is only so much that the code can scale (at least in MPP+SMP mode)
• there is a "peak performance" point somewhere between 112 and 168 cores after which it doesn't make anymore sense to add any more computational power since it negatively affects the performance
• best performance can be achieved if nodes are undersubscribed
• up to 4 cores used per node can give very close to linear performance. Let's look at parallel efficiency values:

cpu_cores: 1    E = 99%
cpu_cores: 7    E = 99%
cpu_cores: 14   E = 97%
cpu_cores: 28   E = 94%
cpu_cores: 56   E = 81%
cpu_cores: 112  E = 68%
cpu_cores: 168  E = 37%

I will have to verify the last point in the next experiment where I run the code on 1-60 processors, subscribing only up to 4 CPUs per node. In the current experiment 28 cores (4 per node) provide 94% efficiency which sounds fairly good to me.

Next, parallel timings:

It is clear from the graph that starting from 4 cores per node processes start under-performing and create visible load imbalance. There is no actual load imbalance: all nodes have exactly the same amount of work to do, but probably due memory accessing bottlenecks some processors wait longer to get their data from memory while others have to wait in the collective routing for the former to complete their computations. In addition to that, the increasing time spent in the MPI_Gather collective is definitely not representing the real communication overheads: if it was then the broadcasting time would be increasing as well because the amount of data broadcast by node 0 on each iteration is greater than the amount of data it has to gather by the corresponding collective routine. The only explanation for why broadcasting time is very small is that it happens after another collective - MPI_Gather, which implicitly synchronises all nodes, and the actual communication time for this routine is very small.

Hector, 100x100, 50 to 500 iterations

Conclusions:

• Although somewhat deviating randomly from run to run by 1-2 seconds, the overhead is static (i.e. does not increase for longer simulations)
• As such, it shouldn't be worried about

Job launching overhead

Using minimum density of 1 neuron on one node:

Conclusions:

• The overhead is somewhat random 
• The overhead doesn't seem to depend on the number of cores used on the same node
• Finally, the overhead can be anywhere between 0.5 and 4 seconds

Next I'm going to try and see if the overhead stays the same while the program run time increases.

Results from Hector

SMP run on one 24-core node


Overall timing capped by the wallclock timer (/usr/bin/time)

 Timings for activation only:

Finally, the speed-up (parallel activation against serial):

I should definitely try the same density on a larger number of processors to see if the speed-up tails off at some point and when.

Next, results obtained with a 200x200 simulation on 4 nodes, using up to 96 cores:

It seems to be achieving only sub-linear speed-ups, although the scaling is pretty good. I need to try a smaller density with a larger number of processors to see what can really be achieved in terms of performance. However, it seems that over a 100 times the serial is quite realistic, given there's enough CPUs in use.

My own version of deleting a line from a file in C

Modified version of this: http://bhagwat-masalkar.blogspot.com/2008/10/delete-line-in-file-c-programming.html

   
    FILE *f;
    f = fopen(argv[4],"a");//filename as command line argument                                                                                

    fseek(f, 0, SEEK_END); //                                                                                                                 
    if (ftell(f) == 0)//new file, add opening root xml tag                                                                                    
      fprintf(f,"<runs>\n");
    else{ // file wasn't empty, delete the closing root xml tag                                                                               
      fclose(f);

      char tmp_str[100];
      FILE *temp;

      f = fopen(argv[4],"r");
      temp = fopen("timings.tmp.xml","wb");

      while(fgets(tmp_str,100,f) != NULL){
        if(strstr(tmp_str,"</runs>"))
          break;
        else
          fputs(tmp_str,temp);
      }

      fclose(f);
      fclose(temp);
      char * command = //concatenating strings to build up a unix command                                                                     
        malloc(snprintf(NULL, 0, "mv timings.tmp.xml %s", argv[4]) + 1);
      sprintf(command,"mv timings.tmp.xml %s",argv[4]);
      system(command);
      f = fopen(argv[4],"a");
    }

    // ... and this is why I hate C sometimes

timings.0.xml

Here's what an entry in my timings files might look like:

<run size='2' density='400' m='20' n='20' iterations='10'>
  <main_tot name='main: Total'>1.05683208</main_tot>
  <main_utils name='main: Utils'>1.04948020</main_utils>
  <main_create_cfs name='main: Creating CFs'>0.00234818</main_create_cfs>
  <main_seq_act name='main: Serial Activate'>0.00202990</main_seq_act>
  <main_mpi_act name='main: Parallel Activate'>0.00296593</main_mpi_act>
  <act_seq_tot name='serial: Total'>0.00202894</act_seq_tot>
  <act_seq_utils name='serial: Utils'>0.00002003</act_seq_utils>
  <act_seq_dp name='serial: Dot Product'>0.00197291</act_seq_dp>
  <act_seq_of name='serial: Output Function'>0.00001287</act_seq_of>
  <act_mpi_tot name='parallel: Total'>0.00296402</act_mpi_tot>
  <act_mpi_utils name='parallel: Utils'>0.00016284</act_mpi_utils>
  <act_mpi_distr_weights name='parallel: Distributing Weights'>0.00114608</act_mpi_distr_weights>
  <act_mpi_bc name='parallel: Broadcasting Inputs'>0.00003695</act_mpi_bc>
  <act_mpi_dp name='parallel: Dot Product'>0.00095177</act_mpi_dp>
  <act_mpi_gather name='parallel: Gathering Outputs'>0.00061488</act_mpi_gather>
  <act_mpi_of name='parallel: Output Function'>0.00001335</act_mpi_of>
</run>

HECToR and my humble app

Just finished pulling my code from the repository. Edited my Makefile (for C Activation, not Topographica) to use cc (Cray's C compiler) instead of gcc. Compiling failed at first complaining about some mpi stuff but suddenly started working (I didn't get how I fixed it, but whatever). Time to make a first run!

Side note: it actually feels pretty cool to have 3 terminal windows all ssh'ed into different supercomputers, one of which being UK national service :P

Submitted my job. 1 core on one processor with a small density size, just to check if it works. Wasn't sure which budget to use, so decided to go with the one we used for our msc practicals (it's the only one I know anyway). Job submitted ok and just finished running. Error. Why does everything that has to do with programming never seems to work the first time you run it? "I don't know, said Hector, but you're..."

Attempting to use an MPI routine before initializing MPICH

I'm just grateful it attempted to run my script and there were no problems with the budget. Ok, how the hell do I initialise MPICH?

Tried loading xt-mpich2/5.1.4 module, it conflicted with xt-mpi, so I had to unload it. What is mpt anyway? What's really bothering me (after googling up the problem) is that probably the error is due to the fact that I compiled the code with a set of libraries different from the ones used for running my code. I wonder why. I used cc (default compiler on Cray) and now using default MPI libraries and that just doesn't seem to work.

I looked up my PD and AT notes: we did use Hector for exercises, but it was for co-arrays and UPC. Couldn't find anything on MPI. Voodoo-ing again...

[update] I got it! So ridiculous... Basically, I was calling MPI_Wtime() before MPI_Init() and after MPI_Finalize. While openmpi allows doing it, mpich seems not to. Glad I've solved it.

And here's the result of tonight's effort (using 1 to 24 cores of one node, i.e. full SMP):

The speed-ups and parallel efficiency values for dot-product are (I'll improve my plotting script to display speed-ups soon):
2 cores: x1.99 (E=0.99)
4 cores: x3.91 (E=0.9775)
8 cores: x6.40 (E=0.8)
16 cores: x11.87 (E=0.74)
24 cores: x20.94 (E=0.87) <- seems to be hitting the cache, or at least close to

HECToR and Git

Having talked to Adrian today, I'm switching to HECToR from ECDF to do my benchmarking experiments. ECDF's turn-around time is just horrible (at least for me!), hopefully the national service will be better. First of all I am going to see how my C code scales on one 24-core die by running it on 1, 2, 4, 8, 16 and 24 cores. The largest number that shows linear or close enough to linear performance improvement will be used in future for distributed-memory experiments: e.g. if 8 cores on one node provide close to 8 times better performance than 1, and then it tails off at 16 and 24, I will use 8 cores per die for scaling experiments. Thus, if the maximum number of cores that I want to use to get an idea of my app's scaling properties is 100, then I will have to use 12 dies, using 8 cores per die (96 in total) and wasting 16 (192 in total).

At the moment I'm experiencing difficulties with installing git on hector. It won't compile, giving me the following message:

make
GIT_VERSION = 1.7.6
    * new build flags or prefix
    CC daemon.o
PGC-W-0095-Type cast required for this conversion (./git-compat-util.h: 254)
PGC-S-0137-Incompatible prototype declaration for function inet_ntop (./git-compat-util.h: 413)
PGC/x86-64 Linux 10.9-0: compilation completed with severe errors
make: *** [daemon.o] Error 2

looks like there's some problem with the C compiler. I tried changing the makefile to explicitly use gcc (it's said to build fine with it, not sure about other compilers. Besides, I managed to build it on ECDF without any trouble), but no luck. At the moment I'm trying out some voodoo stuff, if nothing works I'll just create a new svn repository for topographica and use it as version control between hector and jupiter.

[update] I think I slightly progressed. Specifying "gcc all" as CC option got me a bit further:

make CC=gcc all

Now most of it (I'd like to believe) compiles but breaks once it gets to http.o:

CC http.o
In file included from http.c:1:
http.h:6:23: error: curl/curl.h: No such file or directory
http.h:7:23: error: curl/easy.h: No such file or directory

and a giant bunch of errors after.

[update 2] Fantastic! The solution was found here. In particular, I set a few environment variables:

NO_CURL=1 NO_EXPAT=1 NO_SVN_TESTS=1 NO_TCLTK=1 make CC=gcc all 

Make completes without errors now all that's left is make install.....

NO_CURL=1 NO_EXPAT=1 NO_SVN_TESTS=1 NO_TCLTK=1 make CC=gcc all install

and ...

git --version
git version 1.7.6

feels good :)

Graphs (Jupiter)

Long awaited graphs! I haven't managed to run my code on ECDF yet, so for now I've only got data for Jupiter (cluster of 3 8-core boxes). It would be too early to make any comments on scaling, but at least we can see what the graphs will look like.

The problems size used here was 22500 (i.e. 150x150 sheets), along the X dimension is the number of processors (MPI_Size), along Y - time. Components timed:

#real - the overall runtime (taken from /usr/bin/time)
main_mpi_act - MPI Activation time
main_seq_act - same for the serial
main_create_cfs - initialising connection fields
main_utils - irrelevant stuff

First observations:

• I should give more meaningful names to labels (i.e. "Parallel Activation time" instead of main_mpi_act)
• Parallel time going up with the number of processors??? Really? However, the reason for this will be clear from the next graph

act_mpi_tot - total time spent inthe parallel activation code
act_mpi_of - output functions (the hysteresis function in my case)
act_mpi_gather - gathering output activities
act_mpi_dp - computing dot-product
act_mpi_bc - broadcasting the input matrix
act_mpi_distr_weights - distributing the initial sets of weights
act_mpi_utils - shizzle

By just looking at this I can tell that optimising initial weights distribution could be a lot more important than we thought. Of course it's a one-off operation and it's contribution to the overall time is going to be smaller for longer simulations (mine were only 20 iterations long), but it's still massive! Anyway, here's the kind of graph we're probably most interested in (same as above, but without weights distribution):

The only values of X (MPI_Size) worth attention here are 1,2 and 3 as only these are pure distributed-memory runs (MPP), the rest are mixed SMP-MPP (i.e. more than 1 core used per machine) and are less important for now.

Dot product seems to scale linearly, but with the addition of MPI calls the overall activation time (without weights distribution) scales somewhat sub-linearly. Anyway, I won't make any conclusions about scaling until I get my results from ECDF. Besides, these results cannot be trusted since someone was using half of Jupiter cores while I was running my experiments.

As a side note: it would probably make sense to optimise the weights distribution process in my C code. It shouldn't be hard and will save a lot of CPU time on ECDF.


[update] Implemented local weights initialisation: now each node creates it's own set of weights instead of doing it on the master node and then broadcasting this massive set of data to all nodes. Suddenly the graphs look a lot nicer (also, no one was using jupiter except me this time). Same as before, 150x150 problem size, but this time 50 iterations instead of 20.

Overall run statistics:
 MPI only:

progress update

Finished the script for making good-looking component scaling graphs. At the moment working on getting my C code to compile and run on ECDF. Basically, I have two options:

1. To use Eddie's system MPI. This is the preferred way as I'd expect it to be fine-tuned for ECDF hardware. I'm using the wiki page as guidance, but it seems to be outdated since I can't even load the modules I need:

$ module add intel/cce
ModuleCmd_Load.c(199):ERROR:105: Unable to locate a modulefile for 'intel/cce'

It looks like it is necessary to load some modules for MPI to work:

$mpicc -v
-bash: mpicc: command not found

Adrian, I need your help here
2. Install and use the latest version of openmpi. So far it fails to install with this message:
   

   
   ar: ***/topographica/external/openmpi-1.5.3/ompi/contrib/vt/vt/util/.libs/libutil.a: No such file or directory
   

   Think it's missing some libraries, or whatever, but should be fixable
   

Ideally I'd want to use both and see the difference in performance

[update] Fixed. This issue with openmpi has been raised before, cleaning up all traces of the old install and re-configuring resulted in clean build.

does this sound like an overkill?

100 combined LoC for a python script that builds graphs...

Note: if you're planning to build something like this with matplot don't use pyplot.contour or .contourf unless you want to get some serious headache. It's easier to use pyplot.fill (fillx, etc.). I should've discovered it earlier


[update] 250-something LoC

Current Plan

Quoting Jim:


1. Measure effect of Bare-C timing code by comparing real times for
   versions with and without timing code.  As long as this difference
   is small (under 1%) for sufficiently large problem sizes, then we
   can ignore the timing code.  Otherwise we'll have to account for it
   in any plots with total run time.


2. Finish reporting and plotting, making it very clear (a) how the
   components add up to the total time (and highlighting any
   differences between the sum of the components and the total time),
   and (b) which numbers and plots are comparable between the
   Topographica (hybrid Python+C) and the bare C versions.


3. Prepare good, clear plots of the scaling behavior for us to
   discuss: density versus time (on a single processor), and then
   number-of-CPUs versus time for various (relatively large) density
   values.
   
4. Analyze the plots to understand the scaling behaviour of the bare C
   code with number of CPUs (each with different memory subsystems).
   Is the communication overhead (and any other overheads) low enough
   that it scales well, up to a hundred processors or so?  


5. If the bare-C scaling with CPU is bad, we probably need to focus on the
   communication pattern, reducing time CPUs spend waiting on data.


6. If the bare-C scaling with CPU is good, rejoice and start focusing
   on the Topographica implementation, first making it have the same
   pattern of communication and C computation. Once it appears to be
   doing the same thing as the bare-C version, analyse the scaling
   with CPU for Topographica -- does the Python overhead limit the
   number of processors that are useful?  Is that limit high enough
   for our practical purposes?  If not, we'll need to focus on
   optimizing other parts of Topographica until the scaling behavior
   with CPU is good.


7. Once we have good scaling with CPU for Topographica, for
   sufficiently large problem sizes, we'll be ready to release that to
   everyone to start using.
   
8. Then we can focus on scaling with cores -- can we do anything to
   exploit other cores that share memory hierarchies?  Right now we do
   get some speedup using OpenMP on some shared-memory machines, but
   it's far from ideal. We'll at least want to exploit what speedup we
   already have from using all the cores, and possibly investigate how
   to make it better.  But we're unlikely to get quite that far in
   this project.


Suggested time output:

Total time (CPU usage 96%): 14.301s

Components: init=8.387s + sim.run=5.48s = 13.867s

Unaccounted-for: 0.434s (3.0%)

I'm hoping to finish my plotting script in about 2 days. As soon as that happens I will start using ECDF instead of Jupiter.

Hand-crafted profiling

Just finished placing timers around all the ins and outs in my C code and formatting the outputs. Now the result of a single C vs Python run looks like

./c_vs_python.sh -m 100 -n 100 -i 30 -s examples/matchmodel.ty
TOPOGRAPHICA RUNS

### Python+C ###
Total script init time: 8.38710808754
Serial: 5.48633885384

real    0m14.301s
user    0m13.097s
sys     0m1.021s

### BARE C ###

====================== main() ======================
Total:                                  10.7808
----------------------------------------------------
Utils:                                  0.0557
Create and normalise CFs:               1.4893
Serial activate:                        4.6100
Barrier between serial and parallel:    0.0000
Parallel activate:                      4.6257
=================== activateSeq() ==================
Total:                                  4.6100
----------------------------------------------------
Utils:                                  0.0008
Dot product:                            4.6083
Output function:                        0.0008
=================== activateMPI() ==================
Total:                                  4.6257
----------------------------------------------------
Utils:                                  0.0065
Distributing weights:                   0.0001
Broadcasting inputs:                    0.0000
Dot product:                            4.6174
Gathering activity:                     0.0008
Output function:                        0.0008
====================================================

real    0m10.840s
user    0m10.484s
sys     0m0.303s

### Comparing results ###
Results matched

Timings are printed to screen only for the first node, although in this run I only used just one. For more detailed per-processor results (only for bare C code yet) the timings are printed into separate files (appending if the files exist) called timings.X.out where X corresponds to the rank of the node. A sample file might look like this:

#size 4
#m 100
#n 100
#i 20
main_tot 7.87351489
main_utils 1.07544136
main_create_cfs 0.00000095
main_seq_act 0.00000095
main_mpi_act 2.45419502
main_barrier 4.34387183
act_seq_tot 0.00000000
act_seq_utils 0.00000000
act_seq_dp 0.00000000
act_seq_of 0.00000000
act_mpi_tot 2.45419288
act_mpi_utils 0.00215340
act_mpi_distr_weights 1.10045886
act_mpi_bc 0.00546145
act_mpi_dp 1.34518266
act_mpi_gather 0.00084949
act_mpi_of 0.00001097
#end

I planned this to be more machine-readable for later use in two plotting scripts that I'm currently working on: one would take filename1 filename2 ... "param1 param2 ... >paramN" as arguments to produce N bars where params 1 through N-1 make up the total of paramN (e.g. timings.0.out timings.1.out "act_seq_utils act_seq_dp act_seq_of > act_seq_tot" should show two bars with utils, dot-product and output function displayed as percentages of the total serial activation time). The second script would work with scaling of one parameter against another. For example the total parallel activation time against the number of runs, the number of processors, or the size of the input sheet (m*n). That will be displayed in form of line graphs. Sounds like a lot of work, but in the end I will have a hand-crafted and very customisable set of profiling tools that do exactly what I need, should be worth the effort.

Un-squaring simulations

Got a bit distracted by other work in the previous few days, but thankfully it's all sorted now. Today I fixed a very annoying bug that wasn't allowing me to use rectangular sheets (i.e. with sides of different lengths, as opposed to square sheets): forgot to multiply one value by -1 and there we go, hours of debugging.

Other than that, work in progress on detailed timings and pretty graphics...

Week 3

Another weekly meeting. The aim for this week is to improve my testing script to do graph plotting and measure CPU load. Apart from that I will perform series of parallel scaling experiments on Python and bare C topographicas. If everything goes well, there will be a bunch of colourful graphs here by the end of this week.

C vs Python Part 2

Just for record: there was this annoying issue of output matrix converging to permanent values after a very small number of iterations. Because of using one iteration's output as input on the next iteration, as with purely Lateral model, the output matrix values were very quickly becoming fixed, i.e. not changing irrelevant of how many iterations were run afterwards. This is something you don't want to happen if you have two different versions of the same simulation and want to constantly make sure they do the same thing.

To fix that we changed the model to take external inputs (as with the Afferent model) in the form of a moving dot where the value of 1.0 is moving forward each iteration down an array of zeros, and the value of 0.25 - moving backwards. For instance, on the second iteration the 3 by 3 input matrix would be:

0.0 1.0 0.0

0.0 0.0 0.0

0.0 0.25 0.0

on the fourth:

0.0 0.0 0.0

1.0 0.0 0.25

0.0 0.0 0.0

Despite using external inputs, the outputs of previous iterations are still taken into account: after the new activity matrix has been computed using dot-product response function, a hysteresis function is applied to it the following way(TC is a pre-defined time constant):

new_activity = old_activity + (new_activity - old_activity)*TC

Now the output activity values do change after each iteration, which allows us a fairly reliable test case. I also modified the python script to use the optimised (in C) version of dot-product response function, the difference in performance is quite noticeable and gets bigger for bigger matrix sizes (optimised is the first one):

[jupiter3]sXXXXXX: ./c_vs_python.sh -m 100 -i 40
Serial: 7.49869203568
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 40
Serial: 10.6240258217

Here are the results of C vs Python runs with different densities and numbers of iterations:

[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 50 -i 50
Serial: 0.688660860062
=== Sequential Time ===
Dot Product:    0.471745
-----------------------
Total:          0.472502
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 50 -i 200
Serial: 2.88342404366
=== Sequential Time ===
Dot Product:    1.92014
-----------------------
Total:          1.92326
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 50
Serial: 9.23764681816
=== Sequential Time ===
Dot Product:    7.99027
-----------------------
Total:          7.99298
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 100
Serial: 18.9039599895
=== Sequential Time ===
Dot Product:    16.5311
-----------------------
Total:          16.5367
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 150 -i 20
Serial: 17.1115911007
=== Sequential Time ===
Dot Product:    15.8554
-----------------------
Total:          15.858
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 150 -i 40
Serial: 34.4383950233
=== Sequential Time ===
Dot Product:    32.8897
-----------------------
Total:          32.8947
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 170 -i 20
Serial: 28.8636169434
=== Sequential Time ===
Dot Product:    26.6929
-----------------------
Total:          26.6962
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 170 -i 40
Serial: 59.0316541195
=== Sequential Time ===
Dot Product:    53.9727
-----------------------
Total:          53.9792

Notice that the timings for both versions of the code are very close with some amount of relatively static overhead to the Python code which decreases for bigger problem sizes: ~50% for m=50, ~10% for m=170

This looks like a very good starting base for my comparative performance investigations.

C vs Python

First of all, I built a script to automate topographica and c runs and compares the results. Called it c_vs_python.sh. Because of the output stabilising issue (after a certain number of iterations the output matrix ) I can't say I'm 100% sure we've built matching models, but I'd say my confidence in their correctness is about 90%. I ran it with different matrix sizes and numbers of iterations, and the results were matching in each case up to the 8th digit of precision (please correct me if I'm wrong, but isn't that the most you can get out of 32 bit float?).

The script also measures the simulation time (not the total run time!) for both runs. Both serial runs, no MPI involved yet. I don't know, either I'm timing things in a wrong way, or C is indeed that much faster than Python. I truly hope it's the former, because otherwise it will be really hard to compare the parallel performance...

-m - matrix height/width (it's square)

-i - the number of iterations

./c_vs_python.sh -m 120 -i 20
Serial: 9.2740278244
=== Sequential Time ===
Dot Product:    6.20771
Normalise:      0.00157809
Copy Activity:  0.000202894
-----------------------
Total:          6.20958

Toporaphica's time is at the top (9.27s), C - at the bottom (6.2s). 3:2. This ratio doesn't depend on the number of iterations, but seems to decrease slightly for bigger matrices:

[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 170 -i 20
Serial: 31.0484178066
=== Sequential Time ===
Dot Product:    25.015
Normalise:      0.00315642
Copy Activity:  0.000400066
-----------------------
Total:          25.0187
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 170 -i 40
Serial: 62.2372758389
=== Sequential Time ===
Dot Product:    50.2376
Normalise:      0.00620961
Copy Activity:  0.000801563
-----------------------
Total:          50.2448

vampirtrace

Oh, and I finally got Vampirtrace to work, only to realise that it isn't that great for tracing Python code: it only seems to display communication routine calls, while it would be very useful for me to track other events, e.g. function calls. Ideally I would want to use some tool that combines profiling based on PC sampling together with MPI tracing and visualises that in some human-friendly way. However, even being able to manually set events or labels and track when and how many times the application triggers them would do. MPE seems to be capable of that, I'll try it out later on.

Week 2

Had my second project meeting yesterday, the plan for this week is to modify my C code and Topographica (hopefully only model scripts) to match in terms of the results produced and internal operations. Starting off with lateral-only square all-to-all projections with no learning. This means that the output matrix is produced by dot-producting an input matrix with a 4-dimensional matrix of weights: first two dimensions contain an M*M grid of neurons (at this stage the size and shape of the grid are the same as that of the input and output matrices).  At this stage, for simplicity each neuron is connected to all other neurons in the sheet, and therefore is represented by an M*M matrix of connection weights.Thus, the grid of neurons together with their weights can be expressed as M*M*M*M matrix, input and output - as M*M. Output is generated by dot-producting input with each neuron's weights, the result of each operation is a single number. After M*M such operations full output matrix is produced. "Lateral" term in our case means that the system uses the output matrix from previous iteration as input to produce new output, as opposed to using some external feed. No learning means that the neuron sheet is static at all points of the simulation.

At some point we want the neuron sheet to be M*K*L*N, i.e. use non-square patterns as well as for inputs and outputs; have limited connection fields (i.e. non-all-to-all), some form of learning and, perhaps, afferent projections (taking inputs from the outside). All that will be used to isolate any performance problems and see the potential for parallel scalability.

URL for the experimental MPI code

http://gitorious.org/parallel-topographica/parallel-topographica/trees/master/mpi_experiments

Another update

Spent the last two days trying to do some tracing with vampirtrace so that I don't have to hardcode timers anymore but haven't managed to get anywhere yet. Hoping to get some help on the mpi4py google group, because at the moment I'm in the dead end and out of ideas...

Done some benchmarking with the new python code I built a few days ago and the main conclusion I have to make is: there is indeed a late reduce problem causing communication times to soar, but it's definitely not due to load imbalance. In fact, in my app the load is perfectly balanced. It's just that when several SMP nodes are computing data in parallel, and the computation speed is ultimately limited by memory bandwidth (which is exactly the case with topographica and both my testing apps), the nodes will slow each other down. An interesting observation is that for some reason this behaviour is asymmetric in a way that some nodes will have advantage all the time during one run (next run some other process could dominate). I'm not quite sure why this is happening, and could dig into that if needed, but both my experimental apps demonstrate this behaviour. Here's an output of my C app, after I ran it on the same SMP machine subscribing 6 CPU cores:

=== Sequential Time ===
Dot Product:    14.4671
Normalise:      0.00240159
Copy Activity:  0.000317574
-----------------------
Total:          14.4698

=== Parallel v.1 Time ===
Parallel
dot product:    7.45656
        Broadcasting:   0.00737047
        Dot Product:    4.6
        Gathering:      2.84918
Normalise:      0.00228691
Copy Activity:  0.000320673
-------------------------
Total:          7.45918
######### P0 #########
dot product:    7.45656
        Broadcasting:   0.00737047
        Dot Product:    4.6
        Gathering:      2.84918
######### P1 #########
dot product:    7.45768
        Broadcasting:   0.0125265
        Dot Product:    7.44423
        Gathering:      0.000930071
######### P2 #########
dot product:    7.45741
        Broadcasting:   0.00942779
        Dot Product:    7.44338
        Gathering:      0.00460601
######### P3 #########
dot product:    7.45694
        Broadcasting:   0.00952554
        Dot Product:    7.43021
        Gathering:      0.017205
######### P4 #########
dot product:    7.45681
        Broadcasting:   0.00902605
        Dot Product:    7.05847
        Gathering:      0.389309
######### P5 #########
dot product:    7.45687
        Broadcasting:   0.00766182
        Dot Product:    4.95379
        Gathering:      2.49543

Processes 0 and 5 were dominating this run in that the computations on these have taken 30%-40% less time than on the other nodes. And on the same two processes we can see Gathering taking as much as about 50% of total computation time, a lot more than on the other nodes (notice on process 1 the total Gathering time is less than a millisecond). This is a very clear early reduce pattern that can be observed here: some nodes finish their computations quicker than others and have to wait inside the collective routine. Why isn't Broadcasting taking that much? Well, this collective operation is called very shortly after Gathering, which synchronises all processes, so they are entering the routine roughly at the same time.

My Python app was giving similar results, so there's no point in discussing them. Oh, except one thing: pickling Gather turned out to be not much slower than it's non-pickling version on SMP, and sometimes even faster on MPP!

Now let's see what happens if we run the app with the same problem size on 3 boxes, subscribing only 1 CPU on each machine:

=== Sequential Time ===
Dot Product:    15.3458
Normalise:      0.00247908
Copy Activity:  0.000368357
-----------------------
Total:          15.3486

=== Parallel v.1 Time ===
Parallel
dot product:    5.17208
        Broadcasting:   0.00858498
        Dot Product:    5.15076
        Gathering:      0.0127301
Normalise:      0.00232387
Copy Activity:  0.000324488
-------------------------
Total:          5.17474
######### P0 #########
dot product:    5.17208
        Broadcasting:   0.00858498
        Dot Product:    5.15076
        Gathering:      0.0127301
######### P1 #########
dot product:    5.16623
        Broadcasting:   0.158481
        Dot Product:    5.00647
        Gathering:      0.00128365
######### P2 #########
dot product:    5.16298
        Broadcasting:   0.192611
        Dot Product:    4.96899
        Gathering:      0.0013752

Almost perfectly linear speed-up and communication time kept to minimum. QED.

Now, I'm not saying there's no load imbalance problem in Topographica, to be honest I'd be quite happy if there was because that would give me an interesting and challenging problem to solve and write about in my dissertation. I only want to admit that massive communication overheads that I observed were to a big extent not due to them but rather to SMP memory access bottleneck.

It all seems very obvious now, but I think I just had to demonstrate and prove that. Tomorrow I'm profiling Topographica to see what else is slowing it down and trying to get the effing vampirtrace to work. Also, I now have my very own account on Eddie and will be able to use it as soon as I get get my miserable 1GB disk quota extended to something a bit less depressing. I think I might still have my HECToR account, need to check, but the service is down at the moment anyway.

4 am ffs....

Vampirtrace + mpi4py

Banging my head against the wall: http://groups.google.com/group/mpi4py/browse_thread/thread/f33761d54dbf0fe9

Load Balance

While testing my code, I had a very interesting observation. I added computations into the loops with collective routines: now on each iteration nodes were not only communicating data, but also computing dot-products of local sets of distributed data with rows of a local array loc_arr (each row of loc_arr is dot-product-ed with the local set). I defined it as

loc_arr = beta(5,20,size=(ceil(N/100),N))

Initially it was a square NxN matrix but I divided the number of elements along the first dimension by a hundred to reduce the overall runtime (I just needed some computations there, not necessarily too heavy): fewer elements in first dimension => fewer dot-products => less time spent doing them. While changing the value of the divisor I noticed something interesting: computation time was changing, as expected, but communication time was changing as well! Furthermore, the communication time differed between application runs. I couldn't get my head around this for some time until I realised: of course, Early Reduce (well, scatter/... in my case) problem! Since I'm measuring time only on the master node, it could arrive to the communication routine while some other node hasn't finished computing! To check this I put barriers before each communication call, and - here we are, comm times went down, and became stable.

Now this made me think of the following: we were always assuming that data distribution in Topographica is load-balanced. Frankly, I wasn't even thinking about it until now. And suddenly, it seems I found the source of the problem with Topographica's massive communication overheads: poor load balance! I really have to look into the distribution patterns, and test this assumption, but I have a strong reason to believe it.

Apart from that, why the hell did my test code un-balance?! All nodes have exactly the same amount of data to compute, plus the computation time taken by all nodes was nearly the same! Some OS-related issues? I don't think so: in my benchmarking code in C there are no barriers before communication routines, and the timings are always stable... I should probably start using ECDF/HECToR to exclude any OS-related overheads anyway.

hacking on...

Quick summary for the day

Upgrade to new versions of mpi4py and OpenMPI went successfully. Then I decided to check whether my assumption that I could get some performance out of replacing the currently used in Topographica collective routines with different versions was correct. In particular, I wanted to see whether Scatter would work faster than scatter and Gather - than gather. The assumption was based on this conversation on mpi4py googlegroup: http://groups.google.com/group/mpi4py/browse_thread/thread/a1741896517c7d12/5c5e604047365348?lnk=gst&q=konstantin#5c5e604047365348. This way I though I would re-familiarise myself with topographica code and also hope to get some performance improvement.

To summarise, scatter/gather involve pickling while Scatter/Gather just make use of the underlying C routines MPI_Gather() and MPI_Scatter(). Thus, the advantage of the first way is that it can communicate arbitrary Python objects while the latter is meant to be faster.

I decided to write a simple Python application that gathers and scatters (later on added broadcasting) large arrays of floats in a loop, 100 times each, and measures the execution  time for each routine. N controlled the size of individual chunks, so the master node (rank 0) would gather/scatter N*P elements, where P is the number of processes used. After having finished the coding I played around with different values of N and P, also running my code in SMP mode and MPP (i.e. using a cluster of 3 machines). Here are some results:


N=10,000
SMP, 3 nodes:


scatter time: 0.0482320785522
Scatter time: 0.0166721343994
bcast time: 0.0553240776062
Bcast time: 0.041867017746
gather time: 0.0940809249878
Gather time: 0.0110039710999


MPP, 3 nodes:

scatter time: 0.177076816559

Scatter time: 0.176827907562

bcast time: 0.211786031723

Bcast time: 0.213672876358

gather time: 0.22430896759

Gather time: 0.168394088745

N=50,000

SMP, 3 nodes:

scatter time: 0.220118045807

Scatter time: 0.0812599658966

bcast time: 0.321494102478

Bcast time: 0.180558919907

gather time: 0.190404176712

Gather time: 0.0511260032654

MPP, 3 nodes:

scatter time: 3.98046803474

Scatter time: 0.681133985519

bcast time: 1.60409212112

Bcast time: 1.39482522011

gather time: 0.849080085754

Gather time: 0.734710931778

N=200,000

SMP, 3 nodes

scatter time: 1.79250884056

Scatter time: 0.383674144745

bcast time: 2.98570203781

Bcast time: 0.743601083755

gather time: 1.25536489487

Gather time: 0.288822889328

MPP, 3 nodes

scatter time: 3.97047996521

Scatter time: 2.73933315277

bcast time: 6.40952301025

Bcast time: 5.25424790382

gather time: 3.54560399055

Gather time: 3.84726691246

As we can see, SMP benefits from non-pickling routines the most, and the difference in performance seems to increase with the amount of data in each packet sent. On the contrary, MPP hasn't shown any significant difference between the performance of two routines. Except in the Scatter/scatter case. I think there must be a hint there: what if the performance is limited by the interconnect bandwidth? Thing is, in my experiment, data is sent in loops, without any interruption or work done between the communication routine calls. This is quite far from what a real application would look like, so I decided to put some work there. Now each node on each gathering/scattering iteration will do some computations. To make it a bit similar to topographica, each node will compute a dot-product of the distributed array and a local set of data. I'll measure both the computation and gathering/scattering time.

Parallel Topographica part 2

This is the first post for my MSc project. After the first meeting today my task for now is to review my old TODO list (which meant quick skim through, shift-select and delete), re-familiarise with the code and report on the current state.

Topographica install went ok, parallel stuff works as well. I checked if there were new versions of OpenMPI and mpi4py released, and yes they were: mpi4py 1.2.2 and OpenMPI 1.4.3. I successfully upgraded both, everything still seems to run. Think I'll go ahead and run a quick simulation in parallel just to make sure nothing has been broken.

This blog isn't dead yet;)

Hi. A quick update. I finished my activation simulator a month or two ago. Surprisingly, the new approach (doing computations and communication in parallel) doesn't seem to be any faster than the old one (compute, then communicate) even on a cluster with slow TCP/IP interconnect, which is a bit disappointing. However, I also discovered that communication overheads in my simulator are significantly smaller than in Topographica, and I am sure that communication patterns in both cases are the same or very similar. This gives us some hope that we will be able to get good speed-ups with MPI on distributed systems. At the moment I'm very busy with my end of term work, so I'll probably resume my research in April and start full-time development in June. That's all I had to say.