As far as I remember, communications were the biggest slowdown factor with the distributed model. Perhaps if I could hide them away, I could get a substantial increase in performance. At the moment I'm thinking of the following trick: instead of gathering the entire projection matrix after each activation (which is the main cause of the communication bottle-neck), why don't I make each node send smaller chunks while processing the rest of the matrix. What I mean by this is: say each time process A computes X number of neuron activities, where X is some small portion of A's chunk, it will broadcast (in a non-blocking way) these X entries to all other nodes, until it reaches the end of its chunk. Hopefully, this will help to hide away the communication overheads. My main concerns, however, are:
- How do we figure out what this X should be? Pick it too small, and broadcast mechanism, whatever it is, won't be able to control the flood. Pick it too big, and the whole point of this optimisation tweak is lost.
- Synchronising this will be a massive pain in the neck: there's no such thing as non-blocking broadcast in MPI-2. As for now, I'll go with bSend or iSend, which, tbh, looks a bit more of a temporary hack to me rather than a long-term solution. I'll come back to it later, probably will have to research this a bit further.
- Lastly, we'll most definitely have to use OpenMP for doing computations on each shared-memory system, since this will help to reduce the number of MPI nodes, and as a result decrease the amount of communication: simply speaking, there will be less packages flying around, because I can really see how the whole thing could get very messy otherwise.
At the moment I got Topographica up and running again, MPI code seems to be working fine and I aslo created a new repository: http://gitorious.org/parallel-topographica/parallel-topographica .
on to hacking...
No comments:
Post a Comment