Implementation

Tried testing performance with the single-run test_script-short test file, everything ok. Now need to find out where the communication happens.

wondering if I should move all the MPI code to a new directory topo/mpi. Good idea, but I'll save this for later.

Yes, it is a very good idea, because I already spent about 15 mins just trying to find bits of my parallel code, especially the PMI part. Where the hell did I put it?!!!......

ok, found everything now.

Will have to add new response functions, capable of exchanging data between nodes iteratively. This might mean that I'll have to do some MPI programming in C for the inline functions.



So, to summarise what I found out today: parallelising communication with dot-product computations at the C-level (i.e. modifying the Dot_Product_opt) is probably the way to go, however, this implies a lot of new changes, so I'll put that away for now. I'll need to speak to Chris or Jim about it first. What I could actually try doing for now is modify the activation function so that it breaks it's own chunk of the activation job into smaller pieces and applies the response_fn to each one, on every iteration sending the computed activity to other nodes in a non-blocking way. This is the first step in hiding away the communications, and it will hopefully help reducing the overhead of gathering the activity matrix on master node after each activation. Then we'll see how it goes. Hope it all makes at least some minimal amount of sense...