Overall timing capped by the wallclock timer (/usr/bin/time)
Timings for activation only:
Finally, the speed-up (parallel activation against serial):
Next, results obtained with a 200x200 simulation on 4 nodes, using up to 96 cores:
It seems to be achieving only sub-linear speed-ups, although the scaling is pretty good. I need to try a smaller density with a larger number of processors to see what can really be achieved in terms of performance. However, it seems that over a 100 times the serial is quite realistic, given there's enough CPUs in use.
Any chance you can put an "ideal" speedup line on the graph (i.e. a line that shows perfect speedup)?
ReplyDeletethanks
adrianj
How are you taking your parallel timings? Are you just taking the time for process 0 or taking the time of all processes and averaging them? Or something different?
ReplyDeleteThe difference between wall-clock and actual time could be due to startup time for the mpi job launchers.
I assume you are doing something like this:
time aprun -n ... ./myprogram
If this is true then time is timing both your program and aprun (or mpirun if you're on eddie) so there is additional overhead in the time data which means that you may not be able to get the times from your program to be equal to the time that time returns
adrianj
Yeah, I also thought of adding a red "Target Speedup" line. "Ideal" also sounds fine. Will do it shortly.
ReplyDeleteThe timings are only taken on node 0. It used to be that each node was creating an independent file, but I later realised that it was redundant: activation is very synchronised due to the collectives it performs on each iteration. Besides, the graphs show that simulation time is dominated by computation and I'm happy with that. Guess it would've made sense to read individual timings for each node if I was doing something like a task farm. However, I could do this - reduce individual timers to node 0 and divide by mpi size, might take and hour or two. Do you think it's worth it?
As for the wallclock times - that's precisely how I'm doing it. I think the main point of this is to show that the difference between the "real" time and the timings from the "inside" decreases for higher densities (to be honest I'm not sure why I'm doing external timings, but it's not very hard to do and Jim wanted to see these, so it's there). Another thing is that there's no timer set around the I/O (printing the timings to file) - that could be adding some static overhead
kk
I don't think it is hugely necessary to have a separate time for each process, it's just if there were big difference between wallclock and your program time it could be load balance (i.e. process 0 sitting waiting for another process to finish). I don't think your program should have load balance problems, but if we suspect it does then individual timers per process would be interesting. For the moment don't worry about it.
ReplyDeletecheers
adrianj
Do you have any idea why hector resets my password after I've used it for some time? For example, now I can't log in with the same password I used yesterday, and it's the second time it's happening. I filed a helpdesk query and also requested a new password through SAFE, which I will only receive tomorrow. It's ok, just slightly irritating. Is there anything I should know to prevent my password resetting (or whatever it is that happens so that I can't log in)?
ReplyDeletecheers
kk
This comment has been removed by the author.
ReplyDeleteThis all looks good, with very understandable plots, and it's great to see the actual times for everything.
ReplyDeleteWhere you say "size" you should probably say "cores"; size can mean many things.
Why does the difference between wallclock and your measured computation times increase with the number of cores (in the first plot)? Does launching an MPI process have startup time dependent on the number of processors? The difference between wallclock and computation time has become quite significant by 24 processors, which makes it difficult to interpret the rest of the results. Can you run separate tests to find out (a) how much the overhead is for simply launching a process, without doing any computation, and why that would scale with the number of processors (assuming you've already distributed the weights generation), and (b) whether the difference between wallclock and computation time decreases with number of iterations?
Basically, whenever you have a difference between wallclock and your measured computation time that approaches the magnitude of your computation time, as it does for 16 and 24 processors in the first plot, then you have to investigate why and demonstrate what sort of overhead this is, and how we expect it to scale with things like problem size and problem duration. You have to clearly demonstrate the conditions under which your measurements will actually reflect the wallclock time experience of the user, so that users will know when they can expect good performance and when it will be dominated by overheads.
I think for the most part it's best to use the timings on the main core, because that should be when the whole job starts and finishes, but it would be useful to know whether there is any variation by core. I'd put that as a separate plot somewhere, as a different point than the ones made here (which are about how fast the user can expect his parallel job to complete).
It is plausible that there is an increase in the wallclock time reported as more cores are used. What is actually being timed in this case is not topographica directly but the job launcher that runs topographica in parallel. This has to run the topographica executable on all the cores available to it. If this is more than one node (i.e. over 24 cores) then it involves network communication. Even if it is on a single node starting an executable on 24 cores does cost something. It is more than possible that the 1 ish second difference you see could be attributed to this. It may be worth running a longer job (something that takes say 60 seconds on 24 cores) to see what the difference is there.
ReplyDeleteIf this is a big problem it may be worth moving way from the ./time command and getting wall clock by putting in a call to the system clock as the first and last entries in your program to get the wall clock time.
adrianj