This blog isn't dead yet;)

Hi. A quick update. I finished my activation simulator a month or two ago. Surprisingly, the new approach (doing computations and communication in parallel) doesn't seem to be any faster than the old one (compute, then communicate) even on a cluster with slow TCP/IP interconnect, which is a bit disappointing. However, I also discovered that communication overheads in my simulator are significantly smaller than in Topographica, and I am sure that communication patterns in both cases are the same or very similar. This gives us some hope that we will be able to get good speed-ups with MPI on distributed systems. At the moment I'm very busy with my end of term work, so I'll probably resume my research in April and start full-time development in June. That's all I had to say.

2/3 complete now!

Had a few free hours this morning, and decided to write a bit more code for my simulation program. Very well, now the first parallel version that uses the same algorithms that's currently implemented in topographica, is finally working! So, two thirds of the task are done=))) the only thing left to work on is new parallel algorithm, hopefully faster than the current.

...still working on the testing program

At the moment stuck with creating a derived datatype for the connection field matrix. The problem is somewhere between three lines of code and I'm out of ideas. However, I'm having a Message Passing Programming practical today - think I'll ask there for help.

~250 lines of code so far=)))

Problem solved!

new approach try-out

Today I'm going to try out my ideas for hiding communications on a piece of C code that's doing something similar to what's happening inside the Topographica's DotProduct_opt. Now, because this piece of code obviously does not exist yet, I'll have to write it, so the plan is:

• Write the sequential version
• Then modify this to run in parallel, the way it does in the current parallel topographica
• Final stage would be trying to modify it further so that communication happens in parallel with computations and see if it gives any speed-ups

...The sequential version is now working. Ew, salt and vinegar crisps are awful... God I hate vinegar! Anyway, on to the next part.


FUUUUUUUUUUUUUUUUUUUUUUUUUUUCK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! accidentally destroyed half the code I've written.... so stupid=( rewriting, what else...

ok, finally recovered, so now back to where I've been. Probably I'll call it a night now, it's a bit late. Tomorrow - working on the MPI bit.

NEW ADDRESS

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Because this blog isn't that much related to mpi4py anymore, in a week's time I will be relocating it to a new address:

hyper-graphica.blogspot.com

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

actually, screw it, I'll do this now...

Change of plans

More or less done with optimising the activation function in python, just realised that I should probably roll back to the original code now. While I could potentially get some speed-ups, the whole approach seems to be a bit pointless since I will ANYWAY have to take the optimisation down to the C level and undo all the changes I will have done in Python. It sort of made sense when I though it was going to be a quick and easy hack, but turned out not to be so and any further work in that direction would be a waste of time.

Next step will be writing a program in C that works roughly the same way as DotProduch_opt, and then trying out several approaches at optimising it to pick the best one.

From home

Working on implementing the idea I had earlier today. Doesn't seem to be hard so far. I got topographica with all the MPI stuff running on my laptop as well, no clue why it's been so difficult the first time... Hope to get some results today, though I don't know how long it's going to take me... Anyway, I've still got 3/4 of a Bud crate.

Ok, braking the connection field array into parts and computing each part works now. Time to work on sending/receiving. 1/2 the crate...

well, it's not as easy as it seems, maybe I could finish it later. Figured what my problem was tho.... almost finished the crate

Got the activation to work with blocking send/receive. Time to improve it to use the non-blocking versions.

Implementation

Tried testing performance with the single-run test_script-short test file, everything ok. Now need to find out where the communication happens.

wondering if I should move all the MPI code to a new directory topo/mpi. Good idea, but I'll save this for later.

Yes, it is a very good idea, because I already spent about 15 mins just trying to find bits of my parallel code, especially the PMI part. Where the hell did I put it?!!!......

ok, found everything now.

Will have to add new response functions, capable of exchanging data between nodes iteratively. This might mean that I'll have to do some MPI programming in C for the inline functions.



So, to summarise what I found out today: parallelising communication with dot-product computations at the C-level (i.e. modifying the Dot_Product_opt) is probably the way to go, however, this implies a lot of new changes, so I'll put that away for now. I'll need to speak to Chris or Jim about it first. What I could actually try doing for now is modify the activation function so that it breaks it's own chunk of the activation job into smaller pieces and applies the response_fn to each one, on every iteration sending the computed activity to other nodes in a non-blocking way. This is the first step in hiding away the communications, and it will hopefully help reducing the overhead of gathering the activity matrix on master node after each activation. Then we'll see how it goes. Hope it all makes at least some minimal amount of sense...

Hiding communications

As far as I remember, communications were the biggest slowdown factor with the distributed model. Perhaps if I could hide them away, I could get a substantial increase in performance. At the moment I'm thinking of the following trick: instead of gathering the entire projection matrix after each activation (which is the main cause of the communication bottle-neck), why don't I make each node send smaller chunks while processing the rest of the matrix. What I mean by this is: say each time process A computes X number of neuron activities, where X is some small portion of A's chunk, it will broadcast (in a non-blocking way) these X entries to all other nodes, until it reaches the end of its chunk. Hopefully, this will help to hide away the communication overheads. My main concerns, however, are:

- How do we figure out what this X should be? Pick it too small, and broadcast mechanism, whatever it is, won't be able to control the flood. Pick it too big, and the whole point of this optimisation tweak is lost.
- Synchronising this will be a massive pain in the neck: there's no such thing as non-blocking broadcast in MPI-2. As for now, I'll go with bSend or iSend, which, tbh, looks a bit more of a temporary hack to me rather than a long-term solution. I'll come back to it later, probably will have to research this a bit further.
- Lastly, we'll most definitely have to use OpenMP for doing computations on each shared-memory system, since this will help to reduce the number of MPI nodes, and as a result decrease the amount of communication: simply speaking, there will be less packages flying around, because I can really see how the whole thing could get very messy otherwise.

At the moment I got Topographica up and running again, MPI code seems to be working fine and I aslo created a new repository: http://gitorious.org/parallel-topographica/parallel-topographica .

on to hacking...

thoughts aloud

Starting to get an urge to do some coding. Hm... what should I start with, interesting=) Probably do some work on Topographica again, especially since I got quite a few new ideas.