The gap between the total simulation time (topo.sim.run()) and activation on the nodes (MPI_CFProjection_node.activate()), as we observed in C-Topo simulations, ideally should be irrelevantly small. Proof is here:
main: Parallel Activate (the black line) is the equivalent of Simulation Total in Topographica. Everything else (the colour-filled areas), combined, is the equivalent of MPI node activation. It's clear that the activation time in C is
a) completely dominated by the dot-product computation
b) takes about as much time as the total simulation.
Not the case with Topographica. I will assume that I have this bit nailed. Another thing to notice on the first figure is that the time difference between simulation total and activation increases with the number of cores used: there is almost none on 2 cores and it becomes quite visible on 24. I think because of this the overhead is unlikely to come from the serial bits of the application. What else? The only thing that comes to my mind is communication overheads which, in turn, could be due to one of two reasons: inefficient use of MPI routines (unlikely, this was an SMP run on 1 node, there wasn't supposed to be any "real" communication involved) or .... something else (I was thinking hard what to put here, got nothing so far). Anyhow, I need to test this assumption first, which won't be too hard: I'll place my timers in PMI's invoke and call methods that manage all communication in parallel topographica and see how much time is being spent there.
No comments:
Post a Comment