In this experiment I was using 7 MPP nodes, subscribing from 1 to 24 cores on each node. The results are quite interesting. First, the speedup:
Some quick conclusions:
- there is only so much that the code can scale (at least in MPP+SMP mode)
- there is a "peak performance" point somewhere between 112 and 168 cores after which it doesn't make anymore sense to add any more computational power since it negatively affects the performance
- best performance can be achieved if nodes are undersubscribed
- up to 4 cores used per node can give very close to linear performance. Let's look at parallel efficiency values:
cpu_cores: 1 E = 99%
cpu_cores: 7 E = 99%
cpu_cores: 14 E = 97%
cpu_cores: 28 E = 94%
cpu_cores: 56 E = 81%
cpu_cores: 112 E = 68%
cpu_cores: 168 E = 37%
I will have to verify the last point in the next experiment where I run the code on 1-60 processors, subscribing only up to 4 CPUs per node. In the current experiment 28 cores (4 per node) provide 94% efficiency which sounds fairly good to me.
Next, parallel timings:
It is clear from the graph that starting from 4 cores per node processes start under-performing and create visible load imbalance. There is no actual load imbalance: all nodes have exactly the same amount of work to do, but probably due memory accessing bottlenecks some processors wait longer to get their data from memory while others have to wait in the collective routing for the former to complete their computations. In addition to that, the increasing time spent in the MPI_Gather collective is definitely not representing the real communication overheads: if it was then the broadcasting time would be increasing as well because the amount of data broadcast by node 0 on each iteration is greater than the amount of data it has to gather by the corresponding collective routine. The only explanation for why broadcasting time is very small is that it happens after another collective - MPI_Gather, which implicitly synchronises all nodes, and the actual communication time for this routine is very small.
Interesting results, it looks like you code is not significantly impacted by network bandwidth (in other words its performance is not really limited by the MPI) as you have perfect speedup going from 1 node to 7 nodes. It looks like memory bandwidth is more important (the amount of memory bandwidth a core can access will go down as you populate the nodes). Given there are 4 sockets per node, having 4 processes per node will give you access to the most memory resources for the best performance so it is not surprising that this gives you the best scaling here.
ReplyDeleteThese results are "strong scaling" (i.e. keeping the problem size fixed and increasing the computational resources). It would be interesting to also see some "weak scaling" results if that is possible. This would be increase the problem size in relation to the increase in computational resources.
I guess the next thing to do would be to add more topographica functionality to see if you can match the performance issues you see with parallel topographica, is that correct?
That is absolutely correct! The three things I'm working on at the moment are: getting results from my 15-node run (with 4 cores used on each one) - that's done, just need to post the graphs. Next, I'm placing timers in topographica and preparing it to run on Hector. Then we can compare the performance of the two versions. If it's roughly the same I would have to increase the complexity of the topographica simulation script and my C code. If it's significantly different I will first (hopefully) improve topographica and then proceed to increasing complexity. Another thing I'm currently working out is how to plot two speed-up lines (for topo and c-topo) on the same graph. It's very non-trivial with my script...
ReplyDeletealso, i don't think i fully understand the concept of "weak scaling". could you roughly sketch an experiment for me? (i.e. #of CPU's and problem size increments)
ReplyDeleteIn weak scaling you would double the problem size when you double the number of processes. So generally you can start with a small problem size for 1 core and then double it for 2, double it for 4, etc...
ReplyDeleteIs this feasible with topographica?
I see. Won't be a problem: both problem size and, obviously, the number of cores/nodes are passed as command line arguments.
ReplyDelete