new pmi

What should I do with the optimised PMI module? (I mean what would be the best place to put it?)

Merging

topographica_mpi is now history. topographica --mpi has come to replace it. Working on merging with the current SVN version at the moment.

up

Weights generation problem is fixed now and the parallel code produces correct results: I modified the timed_topographica.sh script to work with lissom and it shows that the activity of V1 is the same in serial and in parallel. Next I need to move pmi.init into the commandline module to get rid of topographica_mpi once and for all and then - diff-ing and merging with the svn version.

Sorry for being so slow - I really do have very little free time. I remember someone said that while working in a start-up I won't have life. Well, yes.

I'm back

quick update: looks like I've cleaned up most of the cruft in the code, however now have to deal with error messages from cf.py that still tries to generate weights in parallel. On a bright side, I've run tests with lissom_oo_or_mpi.ty (before getting rid of the parallel weights generation code on the mpi-side) and it seems to be working fine. Anyway, the short-run plan is to fix the bug with weights, then move the pmi initialisation code away from the topographica launch script and then run some tests for correctness with lissom. Hoping to achieve that tomorrow.

Performance results for PMI-optimised version

Slightly better than it used to be.

Parallel Efficiency:
cpu_cores: 1    E = 99.5%
cpu_cores: 2    E = 96.6%
cpu_cores: 4    E = 91.7%
cpu_cores: 8    E = 79.7%
cpu_cores: 16   E = 75.3%
cpu_cores: 24   E = 67.6%

And simulation total against node activation:

Topograhica: topo.sim.run time and MPI node activation time

The gap between the total simulation time (topo.sim.run()) and activation on the nodes (MPI_CFProjection_node.activate()), as we observed in C-Topo simulations, ideally should be irrelevantly small. Proof is here:

main: Parallel Activate (the black line) is the equivalent of Simulation Total in Topographica. Everything else (the colour-filled areas), combined, is the equivalent of MPI node activation. It's clear that the activation time in C is
a) completely dominated by the dot-product computation
b) takes about as much time as the total simulation.

Not the case with Topographica. I will assume that I have this bit nailed. Another thing to notice on the first figure is that the time difference between simulation total and activation increases with the number of cores used: there is almost none on 2 cores and it becomes quite visible on 24. I think because of this the overhead is unlikely to come from the serial bits of the application. What else? The only thing that comes to my mind is communication overheads which, in turn, could be due to one of two reasons: inefficient use of MPI routines (unlikely, this was an SMP run on 1 node, there wasn't supposed to be any "real" communication involved) or .... something else (I was thinking hard what to put here, got nothing so far). Anyhow, I need to test this assumption first, which won't be too hard: I'll place my timers in PMI's invoke and call methods that manage all communication in parallel topographica and see how much time is being spent there.

Topographica: Serial vs Parallel on 1 core

  <run n='100' m='100' iterations='1000' cpu_cores='1' density='10000'>
    <sim_tot_mpi name='Simulation total'>214.378264904</sim_tot_mpi>
    <sim_tot_seq name='Simulation total'>212.18565011</sim_tot_seq>
  </run>

it doesn't look like PMI/mpi4py adds much static overhead.

Detailed timings

Some more detailed timings for Topographica:

"Simulation total" 16.2974710464
"MPI_CFP.activate()" 13.5669333935
"MPI_CFP_n.activate()" 11.1868505478
"MPI_CFP_n.activate():response_fn" 11.1458990574

MPI_CFP.activate() - the activate function of MPI_CFProjection class. This class contains serial code and through PMI calls functions of MPI_CFProjection_node
MPI_CFP_n.activate() - the activate function of MPI_CFProjection_node (parallel CFProjection)

MPI_CFP_n.activate():response_fn - response function call (response_fn mainly consists of the dot-product implementation)

Here we can see that most of the parallel activation time is spent doing dot-products (11.15 s out of 11.19). Parallel activation takes 11.19 seconds out of 13.57 seconds (or 82%) of serial-side activation call which consists only of the PMI call (the bridge between the serial and parallel implementations: it distributes input data to nodes and invokes their methods specified in the PMI call) and re-combining the data. Here's the source code for this function:

    
def activate(self, input_activity):
        activity_list = pmi.invoke(self.pmiobj,'activate',input_activity)

        self.activity = numpy.array([])
        for activity_row in activity_list:
            self.activity = numpy.append(self.activity, activity_row)
        self.activity = self.activity.reshape(self.activity_shape[0],self.activity_shape[1])

The missing 18% are probably lost communicating doing some python stuff. In all that wouldn't be too bad if not the fact that serial-side activation takes 83% of the total Simulation time while the parallel-side activation contributes to 68% of the total topo.sim.run time. This means that about 32% of simulation time is lost doing something unknown. I don't think it's likely that this is pure Python overhead because in this post, comparing serial versions of Topographica's matchmodel and C-Topo I've shown that the overhead is fairly constant and small. For example in this run of a 100x100 density, Topographica is slower than C only by 2 seconds:
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 100

Serial: 18.9039599895
=== Sequential Time ===
Dot Product: 16.5311
-----------------------
Total: 16.5367

"Serial" is topographica's time and the one at the bottom is for C-Topo.

Here's some sample output of C-Topo with the same density, same number of processors (I left only the important entries):

  <run n='100' density='10000' m='100' iterations='1000' cpu_cores='24'>
    <main_mpi_act name='main: Parallel Activate'>10.80980301</main_mpi_act>
    <act_mpi_dp name='parallel: Dot Product'>9.99707174</act_mpi_dp>
  </run>

The total simulation time here is main_mpi_act, 10.81 seconds. 9.997 seconds of which is spend doing dot-products. That is 92%, as opposed to Topographica's 68% which more than 20% less efficient. The challenge now is to figure out where this inefficiency comes from.

C vs Python. Hector, 13 nodes, 1-52 cores

A few more graphs.

Topographica:

C-Topo

I'm not sure what's causing performance go down at 52 cores for C. Last time in a similar experiment (except with 10000 density instead of 14400) 52 cores were showing very good efficiency. This could be a one-off glitch. Anyway, the point of this post is: topographica is still worse, even when subscribing only 4 cores per node.

C vs Python. Hector, 1 node SMP, 1-24 cores

Speedups:

Topographica:

C-Topo:



Parallel Efficiencies:

                Topographica  C-Topo
cpu_cores: 1    E = 99.0%     E = 99.5%
cpu_cores: 2    E = 95.4%     E = 98.3%
cpu_cores: 4    E = 89.3%     E = 96.3%
cpu_cores: 8    E = 78.4%     E = 91.7%
cpu_cores: 16   E = 66.1%     E = 90.6%
cpu_cores: 24   E = 55.4%     E = 86.2%