Performance results for PMI-optimised version

Slightly better than it used to be.

Parallel Efficiency:
cpu_cores: 1    E = 99.5%
cpu_cores: 2    E = 96.6%
cpu_cores: 4    E = 91.7%
cpu_cores: 8    E = 79.7%
cpu_cores: 16   E = 75.3%
cpu_cores: 24   E = 67.6%

And simulation total against node activation:

Topograhica: topo.sim.run time and MPI node activation time

The gap between the total simulation time (topo.sim.run()) and activation on the nodes (MPI_CFProjection_node.activate()), as we observed in C-Topo simulations, ideally should be irrelevantly small. Proof is here:

main: Parallel Activate (the black line) is the equivalent of Simulation Total in Topographica. Everything else (the colour-filled areas), combined, is the equivalent of MPI node activation. It's clear that the activation time in C is
a) completely dominated by the dot-product computation
b) takes about as much time as the total simulation.

Not the case with Topographica. I will assume that I have this bit nailed. Another thing to notice on the first figure is that the time difference between simulation total and activation increases with the number of cores used: there is almost none on 2 cores and it becomes quite visible on 24. I think because of this the overhead is unlikely to come from the serial bits of the application. What else? The only thing that comes to my mind is communication overheads which, in turn, could be due to one of two reasons: inefficient use of MPI routines (unlikely, this was an SMP run on 1 node, there wasn't supposed to be any "real" communication involved) or .... something else (I was thinking hard what to put here, got nothing so far). Anyhow, I need to test this assumption first, which won't be too hard: I'll place my timers in PMI's invoke and call methods that manage all communication in parallel topographica and see how much time is being spent there.

Topographica: Serial vs Parallel on 1 core

  <run n='100' m='100' iterations='1000' cpu_cores='1' density='10000'>
    <sim_tot_mpi name='Simulation total'>214.378264904</sim_tot_mpi>
    <sim_tot_seq name='Simulation total'>212.18565011</sim_tot_seq>
  </run>

it doesn't look like PMI/mpi4py adds much static overhead.

Detailed timings

Some more detailed timings for Topographica:

"Simulation total" 16.2974710464
"MPI_CFP.activate()" 13.5669333935
"MPI_CFP_n.activate()" 11.1868505478
"MPI_CFP_n.activate():response_fn" 11.1458990574

MPI_CFP.activate() - the activate function of MPI_CFProjection class. This class contains serial code and through PMI calls functions of MPI_CFProjection_node
MPI_CFP_n.activate() - the activate function of MPI_CFProjection_node (parallel CFProjection)

MPI_CFP_n.activate():response_fn - response function call (response_fn mainly consists of the dot-product implementation)

Here we can see that most of the parallel activation time is spent doing dot-products (11.15 s out of 11.19). Parallel activation takes 11.19 seconds out of 13.57 seconds (or 82%) of serial-side activation call which consists only of the PMI call (the bridge between the serial and parallel implementations: it distributes input data to nodes and invokes their methods specified in the PMI call) and re-combining the data. Here's the source code for this function:

    
def activate(self, input_activity):
        activity_list = pmi.invoke(self.pmiobj,'activate',input_activity)

        self.activity = numpy.array([])
        for activity_row in activity_list:
            self.activity = numpy.append(self.activity, activity_row)
        self.activity = self.activity.reshape(self.activity_shape[0],self.activity_shape[1])

The missing 18% are probably lost communicating doing some python stuff. In all that wouldn't be too bad if not the fact that serial-side activation takes 83% of the total Simulation time while the parallel-side activation contributes to 68% of the total topo.sim.run time. This means that about 32% of simulation time is lost doing something unknown. I don't think it's likely that this is pure Python overhead because in this post, comparing serial versions of Topographica's matchmodel and C-Topo I've shown that the overhead is fairly constant and small. For example in this run of a 100x100 density, Topographica is slower than C only by 2 seconds:
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 100

Serial: 18.9039599895
=== Sequential Time ===
Dot Product: 16.5311
-----------------------
Total: 16.5367

"Serial" is topographica's time and the one at the bottom is for C-Topo.

Here's some sample output of C-Topo with the same density, same number of processors (I left only the important entries):

  <run n='100' density='10000' m='100' iterations='1000' cpu_cores='24'>
    <main_mpi_act name='main: Parallel Activate'>10.80980301</main_mpi_act>
    <act_mpi_dp name='parallel: Dot Product'>9.99707174</act_mpi_dp>
  </run>

The total simulation time here is main_mpi_act, 10.81 seconds. 9.997 seconds of which is spend doing dot-products. That is 92%, as opposed to Topographica's 68% which more than 20% less efficient. The challenge now is to figure out where this inefficiency comes from.

C vs Python. Hector, 13 nodes, 1-52 cores

A few more graphs.

Topographica:

C-Topo

I'm not sure what's causing performance go down at 52 cores for C. Last time in a similar experiment (except with 10000 density instead of 14400) 52 cores were showing very good efficiency. This could be a one-off glitch. Anyway, the point of this post is: topographica is still worse, even when subscribing only 4 cores per node.

C vs Python. Hector, 1 node SMP, 1-24 cores

Speedups:

Topographica:

C-Topo:



Parallel Efficiencies:

                Topographica  C-Topo
cpu_cores: 1    E = 99.0%     E = 99.5%
cpu_cores: 2    E = 95.4%     E = 98.3%
cpu_cores: 4    E = 89.3%     E = 96.3%
cpu_cores: 8    E = 78.4%     E = 91.7%
cpu_cores: 16   E = 66.1%     E = 90.6%
cpu_cores: 24   E = 55.4%     E = 86.2%

script output

luckily missing module warnings disappeared after I removed module swaps from .bashrc

TOPOGRAPHICA RUNS

### Topographica ###
Application 1013295 resources: utime ~1199s, stime ~63s real 83.19 user 0.00 sys 0.00

### C-Topo ###
Application 1013303 resources: utime ~301s, stime ~3s real 14.70 user 0.18 sys 0.01

### Comparing the results of c-topo and topographica ###
Results matched

Graphs will shortly follow, first for single-node run.

update

Good news and somewhat confusing news. The good news is: it finally runs and produces timings!!! I couldn't be happier about it. The confusing news is this: I have no idea why it runs, and the output it produces is even more confusing (script output,not the timings; timings are fine). When I first tried running topographica from my /work directory it ran but probably froze, because it only completed when it ran out of allocated time (10 minutes). Here's what I found in the output file:

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

Of course they are not loaded! I swapped PrgEnv-pgi for PrgEnv-gnu and xt-mpt for xt-mpich2 (mpi4py wouldn't build otherwise). Then I tried swapping the modules back, without recompiling mpi4py. And it worked! Why did it not segfault? Apparently, this means that you can build mpi4py with one set of MPI libraries and run with another?.. However, that's not it: despite the correct execution, the new output file still contained

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

huh? Ok, I'm just happy it runs, at last. I have to admit, making topographica run on Hector was a royal pain in the...

the full error output

messages between these ">>>>>" "<<<<<" are printed exceptions that break execution, I added the print statements into weave/catalog.py:

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04' <<<<< >>>> [Errno 17] File exists: '/home/d04' <<<<< Traceback (most recent call last): File "/var/spool/alps/1009053/topographica_mpi", line 15, in <module> process_argv(argv[1:]) File "/esfs2/d04/d04/s0676011/topographica/topo/misc/commandline.py", line 474, in process_argv execfile(filename,__main__.__dict__) File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ cp.pbs.o321605 create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/basic.py", line 31, in <module> from topo.base.patterngenerator import PatternGenerator,Constant File "/esfs2/d04/d04/s0676011/topographica/topo/base/patterngenerator.py", line 238, in <module> from topo.misc.inlinec import inline,c_header File "/esfs2/d04/d04/s0676011/topographica/topo/misc/inlinec.py", line 59, in <module> import weave File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/__init__.py", line 9, in <module> from blitz_tools import blitz File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/blitz_tools.py", line 11, in <module> import inline_tools File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/inline_tools.py", line 15, in <module> function_catalog = catalog.catalog() File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 351, in __init__ sys.path.append(default_dir()) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 202, in default_dir create_dir(path) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 131, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Weave

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<<

The directory exists (I mkdir'ed it), obviously. I even chmod'ed 777 permissions for it. Seems like the root of the problem is the same as with matplotlib: Hector's backend can't access files and directories on the frontend. Except for unlike with matplotlib, I can't tell topographica to ignore Weave

Are there any shared directories I could make use of? Otherwise, I'm really puzzled...

One thing I don't understand: my C app creates an xml file (timings), which I'm quite sure it does on the backend (the file is created from withing the main body of the code) and that does not cause any trouble with permissions. However, for python libraries it does - could the problem be somewhere else?

Weird error message

Not sure what's going on there (from somewhere in the middle of the same error output as the last message in the previous post was taken from):

File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info

What is it importing from basic?.. That's the contents of topographica's root directory

Topographica on Hector

Today Adrian helped me out with the libgfortran bug: the library I was looking for was inside one of the subdirectories of gcc, namely /opt/gcc/4.5.2/snos/lib. I have no clue why find /opt/ -name 'libgfortran.so.3' wasn't picking it up.


The next issue - topographica not being able to find it's source code (the topo module). This was fixed by adding  the following line into my .pbs script:

export PYTHONPATH=$PBS_O_WORKDIR/topo:$PYTHONPATH

The next issue was matplotlib unable to create .matplotlib directory anywhere:

RuntimeError: Failed to create /home/d04/d04/s0676011/.matplotlib; 
consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data

I tried $HOME, $PBS_O_WORKDIR, work and topographica directories and even ./  . Nothing helped, I'm still not sure why. Neither did any solutions I found on the web. What fixed that problem was tracing the import and adding exception to the file that was importing matplotlib (topo/misc/commandline.py):

try:
    # Import matplotlib, if available, and use a non-GUI backend by default
    import matplotlib
    matplotlib_imported=True
    from matplotlib import rcParams
    rcParams['backend']='Agg'
except ImportError:
    print "Warning: Could not import matplotlib; pylab plots will not work."
except RuntimeError as e: #KKALERT: added this to be able to run on hector                   
    print "Warning: pylab plots will not work:",e

Thing I'm stuck with at the moment:

File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 129, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Starting to think that launching Topographica on Hector is a project on it's own...

mpi4py and hector

Trying to compile mpi4py with gcc didn't get me anywhere, even after following what I was suggested at the mpi4py google group. I did manage after all to compile it with gcc after swapping several libraries. Now however, I'm stuck with this:

Traceback (most recent call last):
  File "mpitest.py", line 1, in <module>
    from mpi4py import MPI
ImportError: libgfortran.so.3: cannot open shared object file: No such file or directory

I've spent a whole day trying fix this and didn't get any further this error message. The library just isn't there and I'm not 100% sure that this is what mpi4py is trying to find. Why does it even need fortran's libraries? i'm stuck...

mpi4py and Cray's compiler

They don't seem to work together. Basically, the problem is that mpi4py is by default compiled with gcc, however I need to compile it with cc instead to be able to run parallel jobs with aprun. Now, cc gives a bunch of unrecognised compiler flag options (it doesn't understand gcc's flags) when trying to build mpi4py with it. I think I tried pretty much everything I could with what I found on the web to make it work, but no luck so far... I think the options now are:

1) get mpi4py to compile with cc
2) use MPI that is based on gcc (OpenMPI? MPICH2?) if that's possible on Hector.

I posted about this on mpi4py's google group board, seems like they already had to deal with this issue before, so hoping to get some help from there.

Also, am I getting it right: cc actually uses Portland's compiler (pgcc) under the hood? It appears so from the error messages I'm getting