#real - the overall runtime (taken from /usr/bin/time)
main_mpi_act - MPI Activation time
main_seq_act - same for the serial
main_create_cfs - initialising connection fields
main_utils - irrelevant stuff
First observations:
- I should give more meaningful names to labels (i.e. "Parallel Activation time" instead of main_mpi_act)
- Parallel time going up with the number of processors??? Really? However, the reason for this will be clear from the next graph
act_mpi_tot - total time spent inthe parallel activation code
act_mpi_of - output functions (the hysteresis function in my case)
act_mpi_gather - gathering output activities
act_mpi_dp - computing dot-product
act_mpi_bc - broadcasting the input matrix
act_mpi_distr_weights - distributing the initial sets of weights
act_mpi_utils - shizzle
By just looking at this I can tell that optimising initial weights distribution could be a lot more important than we thought. Of course it's a one-off operation and it's contribution to the overall time is going to be smaller for longer simulations (mine were only 20 iterations long), but it's still massive! Anyway, here's the kind of graph we're probably most interested in (same as above, but without weights distribution):
Dot product seems to scale linearly, but with the addition of MPI calls the overall activation time (without weights distribution) scales somewhat sub-linearly. Anyway, I won't make any conclusions about scaling until I get my results from ECDF. Besides, these results cannot be trusted since someone was using half of Jupiter cores while I was running my experiments.
As a side note: it would probably make sense to optimise the weights distribution process in my C code. It shouldn't be hard and will save a lot of CPU time on ECDF.
[update] Implemented local weights initialisation: now each node creates it's own set of weights instead of doing it on the master node and then broadcasting this massive set of data to all nodes. Suddenly the graphs look a lot nicer (also, no one was using jupiter except me this time). Same as before, 150x150 problem size, but this time 50 iterations instead of 20.
Overall run statistics:
MPI only:
In the 1st and 4th graphs, I'm not sure I understand what main_seq_act and main_mpi_act are.
ReplyDelete"
main_mpi_act - MPI Activation time
main_seq_act - same for the serial
"
Is main_mpi_act the time spent running MPI code (hopefully mainly the dot product but also MPI overheads), and main_seq_act the time spent running code that's not MPI-related? If so, what is that, mainly?
It looks like you have also distributed the activity output function (hysteresis). Have you?
Minor issue: would be a little easier if quantities were always the same color between different graphs (e.g. say act_mpi_dp is always blue).
Chris
P. S. Is your code in a source code repository somewhere that I could look at if I wanted to?
In my C code you have an option to run both Serial and Parallel versions of activation (controlled by defined constants). Serial does not use any MPI and computes activities the same way topographica does. If MPI is turned on, the application will run the second version of activation, parallel and separate from the serial one. After both runs the results are compared to ensure correctness. Thus, main_seq_act represents the difference between the two timer calls wrapping the serial activation function. Similarly, main_mpi_act is the difference between the wall clock time before the parallel activation call and after. act_mpi_tot act_seq_tot represent the same thing (total time spent inside parallel or serial activation) except that these two are located inside the activation functions (at the very beginning and at the very end) and main_seq_act and main_mpi_act are placed outside, rigth before and right after the activation calls.
ReplyDeletemain(){
ReplyDelete...
t0 = MPI_Wtime();
mpiActivate();
main_mpi_act = MPI_Wtime() - t0;
...
}
mpiActivate(){
t0=MPI_Wtime();
...
mpi_act_tot = MPI_Wtime - t0;
}
hope this helps
Ok, I was confused because I didn't see why main_mpi_act and main_seq_act would be stacked in the graphs - I didn't realize they came from the same run.
ReplyDeleteChris