Some more detailed timings for Topographica:
"Simulation total" 16.2974710464
"MPI_CFP.activate()" 13.5669333935
"MPI_CFP_n.activate()" 11.1868505478
"MPI_CFP_n.activate():response_fn" 11.1458990574
MPI_CFP.activate() - the activate function of MPI_CFProjection class. This class contains serial code and through PMI calls functions of MPI_CFProjection_node
MPI_CFP_n.activate() - the activate function of MPI_CFProjection_node (parallel CFProjection)
MPI_CFP_n.activate():response_fn - response function call (response_fn mainly consists of the dot-product implementation)
Here we can see that most of the parallel activation time is spent doing dot-products (11.15 s out of 11.19). Parallel activation takes 11.19 seconds out of 13.57 seconds (or 82%) of serial-side activation call which consists only of the PMI call (the bridge between the serial and parallel implementations: it distributes input data to nodes and invokes their methods specified in the PMI call) and re-combining the data. Here's the source code for this function:
def activate(self, input_activity):
activity_list = pmi.invoke(self.pmiobj,'activate',input_activity)
self.activity = numpy.array([])
for activity_row in activity_list:
self.activity = numpy.append(self.activity, activity_row)
self.activity = self.activity.reshape(self.activity_shape[0],self.activity_shape[1])
The missing 18% are probably lost communicating doing some python stuff. In all that wouldn't be too bad if not the fact that serial-side activation takes 83% of the total Simulation time while the parallel-side activation contributes to 68% of the total topo.sim.run time. This means that about 32% of simulation time is lost doing something unknown. I don't think it's likely that this is pure Python overhead because in
this post, comparing serial versions of Topographica's matchmodel and C-Topo I've shown that the overhead is fairly constant and small. For example in this run of a 100x100 density, Topographica is slower than C only by 2 seconds:
[jupiter3]sXXXXXXX: ./c_vs_python.sh -m 100 -i 100
Serial: 18.9039599895
=== Sequential Time ===
Dot Product: 16.5311
-----------------------
Total: 16.5367
"Serial" is topographica's time and the one at the bottom is for C-Topo.
Here's some sample output of C-Topo with the same density, same number of processors (I left only the important entries):
<run n='100' density='10000' m='100' iterations='1000' cpu_cores='24'>
<main_mpi_act name='main: Parallel Activate'>10.80980301</main_mpi_act>
<act_mpi_dp name='parallel: Dot Product'>9.99707174</act_mpi_dp>
</run>
The total simulation time here is main_mpi_act, 10.81 seconds. 9.997 seconds of which is spend doing dot-products. That is 92%, as opposed to Topographica's 68% which more than 20% less efficient. The challenge now is to figure out where this inefficiency comes from.
