loc_arr = beta(5,20,size=(ceil(N/100),N))Initially it was a square NxN matrix but I divided the number of elements along the first dimension by a hundred to reduce the overall runtime (I just needed some computations there, not necessarily too heavy): fewer elements in first dimension => fewer dot-products => less time spent doing them. While changing the value of the divisor I noticed something interesting: computation time was changing, as expected, but communication time was changing as well! Furthermore, the communication time differed between application runs. I couldn't get my head around this for some time until I realised: of course, Early Reduce (well, scatter/... in my case) problem! Since I'm measuring time only on the master node, it could arrive to the communication routine while some other node hasn't finished computing! To check this I put barriers before each communication call, and - here we are, comm times went down, and became stable.
Now this made me think of the following: we were always assuming that data distribution in Topographica is load-balanced. Frankly, I wasn't even thinking about it until now. And suddenly, it seems I found the source of the problem with Topographica's massive communication overheads: poor load balance! I really have to look into the distribution patterns, and test this assumption, but I have a strong reason to believe it.
Apart from that, why the hell did my test code un-balance?! All nodes have exactly the same amount of data to compute, plus the computation time taken by all nodes was nearly the same! Some OS-related issues? I don't think so: in my benchmarking code in C there are no barriers before communication routines, and the timings are always stable... I should probably start using ECDF/HECToR to exclude any OS-related overheads anyway.
hacking on...
Often balance problems are introduced through the start up of MPI codes. Depending on the library/tools used the master often starts first, then starts all the other processes and then after everything has setup it starts computing itself, or at least processes on different cores are not all guaranteed to be launched at the same time so MPI codes can often be naturally unbalanced at the start. For a large code this isn't an issue, the difference in times is small compared to the overall run time, however for a benchmark code this isn't necessarily true, it depends on how long your benchmark is and how much work it does.
ReplyDeleteHowever, this type of initial unbalance should disappear once communications, especially collective communication, have been undertaken it should (to a large part) disappear, so if you are seeing re-current load balance problems then there may be other issues. How many cores are you using on each node, how many nodes are you using? It could be that balance problems are occuring because there is fast on-node communications and slow off-node communications so there are two communication costs which can introduce load balance.
I used 3 cores on the same machine (SMP) and 3 boxes, 1 core on each (MPP). That's the thing... I thought it would be useful to do some mpi profiling to see what's happening there and then for topographica as well. Think I'll try to use Vampir trial for now, until I start using ECDF/HECToR
ReplyDeletewas that 3 cores on one box and 1 core on 3 boxes at the same time or two different tests (one smp test and one mpp test)?
ReplyDeleteTwo different tests. I don't think I could rely on the results of a simultaneous run...
ReplyDelete