At the moment stuck with creating a derived datatype for the connection field matrix. The problem is somewhere between three lines of code and I'm out of ideas. However, I'm having a Message Passing Programming practical today - think I'll ask there for help.
~250 lines of code so far=)))
Problem solved!
Thursday, 28 October 2010
Tuesday, 26 October 2010
new approach try-out
Today I'm going to try out my ideas for hiding communications on a piece of C code that's doing something similar to what's happening inside the Topographica's DotProduct_opt. Now, because this piece of code obviously does not exist yet, I'll have to write it, so the plan is:
...The sequential version is now working. Ew, salt and vinegar crisps are awful... God I hate vinegar! Anyway, on to the next part.
FUUUUUUUUUUUUUUUUUUUUUUUUUUUCK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! accidentally destroyed half the code I've written.... so stupid=( rewriting, what else...
ok, finally recovered, so now back to where I've been. Probably I'll call it a night now, it's a bit late. Tomorrow - working on the MPI bit.
- Write the sequential version
- Then modify this to run in parallel, the way it does in the current parallel topographica
- Final stage would be trying to modify it further so that communication happens in parallel with computations and see if it gives any speed-ups
...The sequential version is now working. Ew, salt and vinegar crisps are awful... God I hate vinegar! Anyway, on to the next part.
FUUUUUUUUUUUUUUUUUUUUUUUUUUUCK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! accidentally destroyed half the code I've written.... so stupid=( rewriting, what else...
ok, finally recovered, so now back to where I've been. Probably I'll call it a night now, it's a bit late. Tomorrow - working on the MPI bit.
Thursday, 21 October 2010
NEW ADDRESS
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Because this blog isn't that much related to mpi4py anymore, in a week's time I will be relocating it to a new address:
hyper-graphica.blogspot.com
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
actually, screw it, I'll do this now...
Because this blog isn't that much related to mpi4py anymore, in a week's time I will be relocating it to a new address:
hyper-graphica.blogspot.com
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
actually, screw it, I'll do this now...
Change of plans
More or less done with optimising the activation function in python, just realised that I should probably roll back to the original code now. While I could potentially get some speed-ups, the whole approach seems to be a bit pointless since I will ANYWAY have to take the optimisation down to the C level and undo all the changes I will have done in Python. It sort of made sense when I though it was going to be a quick and easy hack, but turned out not to be so and any further work in that direction would be a waste of time.
Next step will be writing a program in C that works roughly the same way as DotProduch_opt, and then trying out several approaches at optimising it to pick the best one.
Next step will be writing a program in C that works roughly the same way as DotProduch_opt, and then trying out several approaches at optimising it to pick the best one.
Tuesday, 19 October 2010
From home
Working on implementing the idea I had earlier today. Doesn't seem to be hard so far. I got topographica with all the MPI stuff running on my laptop as well, no clue why it's been so difficult the first time... Hope to get some results today, though I don't know how long it's going to take me... Anyway, I've still got 3/4 of a Bud crate.
Ok, braking the connection field array into parts and computing each part works now. Time to work on sending/receiving. 1/2 the crate...
well, it's not as easy as it seems, maybe I could finish it later. Figured what my problem was tho.... almost finished the crate
Got the activation to work with blocking send/receive. Time to improve it to use the non-blocking versions.
Ok, braking the connection field array into parts and computing each part works now. Time to work on sending/receiving. 1/2 the crate...
well, it's not as easy as it seems, maybe I could finish it later. Figured what my problem was tho.... almost finished the crate
Got the activation to work with blocking send/receive. Time to improve it to use the non-blocking versions.
Implementation
Tried testing performance with the single-run test_script-short test file, everything ok. Now need to find out where the communication happens.
wondering if I should move all the MPI code to a new directory topo/mpi. Good idea, but I'll save this for later.
Yes, it is a very good idea, because I already spent about 15 mins just trying to find bits of my parallel code, especially the PMI part. Where the hell did I put it?!!!......
ok, found everything now.
Will have to add new response functions, capable of exchanging data between nodes iteratively. This might mean that I'll have to do some MPI programming in C for the inline functions.
So, to summarise what I found out today: parallelising communication with dot-product computations at the C-level (i.e. modifying the Dot_Product_opt) is probably the way to go, however, this implies a lot of new changes, so I'll put that away for now. I'll need to speak to Chris or Jim about it first. What I could actually try doing for now is modify the activation function so that it breaks it's own chunk of the activation job into smaller pieces and applies the response_fn to each one, on every iteration sending the computed activity to other nodes in a non-blocking way. This is the first step in hiding away the communications, and it will hopefully help reducing the overhead of gathering the activity matrix on master node after each activation. Then we'll see how it goes. Hope it all makes at least some minimal amount of sense...
wondering if I should move all the MPI code to a new directory topo/mpi. Good idea, but I'll save this for later.
Yes, it is a very good idea, because I already spent about 15 mins just trying to find bits of my parallel code, especially the PMI part. Where the hell did I put it?!!!......
ok, found everything now.
Will have to add new response functions, capable of exchanging data between nodes iteratively. This might mean that I'll have to do some MPI programming in C for the inline functions.
So, to summarise what I found out today: parallelising communication with dot-product computations at the C-level (i.e. modifying the Dot_Product_opt) is probably the way to go, however, this implies a lot of new changes, so I'll put that away for now. I'll need to speak to Chris or Jim about it first. What I could actually try doing for now is modify the activation function so that it breaks it's own chunk of the activation job into smaller pieces and applies the response_fn to each one, on every iteration sending the computed activity to other nodes in a non-blocking way. This is the first step in hiding away the communications, and it will hopefully help reducing the overhead of gathering the activity matrix on master node after each activation. Then we'll see how it goes. Hope it all makes at least some minimal amount of sense...
Hiding communications
As far as I remember, communications were the biggest slowdown factor with the distributed model. Perhaps if I could hide them away, I could get a substantial increase in performance. At the moment I'm thinking of the following trick: instead of gathering the entire projection matrix after each activation (which is the main cause of the communication bottle-neck), why don't I make each node send smaller chunks while processing the rest of the matrix. What I mean by this is: say each time process A computes X number of neuron activities, where X is some small portion of A's chunk, it will broadcast (in a non-blocking way) these X entries to all other nodes, until it reaches the end of its chunk. Hopefully, this will help to hide away the communication overheads. My main concerns, however, are:
- How do we figure out what this X should be? Pick it too small, and broadcast mechanism, whatever it is, won't be able to control the flood. Pick it too big, and the whole point of this optimisation tweak is lost.
- Synchronising this will be a massive pain in the neck: there's no such thing as non-blocking broadcast in MPI-2. As for now, I'll go with bSend or iSend, which, tbh, looks a bit more of a temporary hack to me rather than a long-term solution. I'll come back to it later, probably will have to research this a bit further.
- Lastly, we'll most definitely have to use OpenMP for doing computations on each shared-memory system, since this will help to reduce the number of MPI nodes, and as a result decrease the amount of communication: simply speaking, there will be less packages flying around, because I can really see how the whole thing could get very messy otherwise.
At the moment I got Topographica up and running again, MPI code seems to be working fine and I aslo created a new repository: http://gitorious.org/parallel-topographica/parallel-topographica .
on to hacking...
- How do we figure out what this X should be? Pick it too small, and broadcast mechanism, whatever it is, won't be able to control the flood. Pick it too big, and the whole point of this optimisation tweak is lost.
- Synchronising this will be a massive pain in the neck: there's no such thing as non-blocking broadcast in MPI-2. As for now, I'll go with bSend or iSend, which, tbh, looks a bit more of a temporary hack to me rather than a long-term solution. I'll come back to it later, probably will have to research this a bit further.
- Lastly, we'll most definitely have to use OpenMP for doing computations on each shared-memory system, since this will help to reduce the number of MPI nodes, and as a result decrease the amount of communication: simply speaking, there will be less packages flying around, because I can really see how the whole thing could get very messy otherwise.
At the moment I got Topographica up and running again, MPI code seems to be working fine and I aslo created a new repository: http://gitorious.org/parallel-topographica/parallel-topographica .
on to hacking...
Monday, 18 October 2010
thoughts aloud
Starting to get an urge to do some coding. Hm... what should I start with, interesting=) Probably do some work on Topographica again, especially since I got quite a few new ideas.
Friday, 2 April 2010
Special Thanks to..
Special thanks to:
- James Bednar, professor at Edinburgh University, for a cool project proposal
- Chris, PhD student at Edunburgh University CSNG group for all the help he offered from the very beginning of this project till, literally, last minutes.
- mpi4py developers for mpi4py and letting keep the address name for this blog
- Olaf Lenz for PMI - this module saved me a lot of time!
- Abe for thesis proof-reading and commenting (at 6am on Wednesday morning!)
- That guy in Chris's office who had a thesis cover page template
- As always, family and friends for support
P.S. One funny (-ish) fact is that actually, until september 2009 I was going to work on a completely different (all projects are allocated in May), and then suddenly changed my mind, came to Jim and asked if he had anything I could work on, and he offered me to optimise Topographica with MPI.
- James Bednar, professor at Edinburgh University, for a cool project proposal
- Chris, PhD student at Edunburgh University CSNG group for all the help he offered from the very beginning of this project till, literally, last minutes.
- mpi4py developers for mpi4py and letting keep the address name for this blog
- Olaf Lenz for PMI - this module saved me a lot of time!
- Abe for thesis proof-reading and commenting (at 6am on Wednesday morning!)
- That guy in Chris's office who had a thesis cover page template
- As always, family and friends for support
P.S. One funny (-ish) fact is that actually, until september 2009 I was going to work on a completely different (all projects are allocated in May), and then suddenly changed my mind, came to Jim and asked if he had anything I could work on, and he offered me to optimise Topographica with MPI.
Thursday, 1 April 2010
Day 124
Day 124: Debriefing
Well, as I have finally submitted my thesis (Have to admit, this was the first paper of such kind that I have ever written in my life and, bloody hell, that was a nightmare!) the project is officially over now, and it's time to summarise the results. Perhaps, it would be useful to reiterate the objectives first (I'll just copy bits from my thesis):
Purpose of the project:
Optimising the Open-Source Neural Map Simulator Topographica by making it capable of distributing its heaviest computations over CPU cores of one machine or over a network of independent machines would allow users of Topographica to run experiments on a bigger scale than it was previously possible. This is expected to help neuroscientists have a better understanding of brain activity at the cortical level. Such understanding, apart from expanding the pool of human knowledge, could potentially assist in the creation of medicines to combat and, hopefully, cure mental diseases, and, possibly benefit the human race in other, as yet undiscovered ways.
Primary Objective
Improve the performance of Topographica simulations by using the High Performance Computing approach, allowing Topographica to take advantage of the computational power of both Multiprocessing and Cluster architectures.
Other Objectives
- Research and summarise how distributing computations with the Message Passing Interface can benefit Topographica
- Research what MPI software packages are currently available and can be utilised for optimising Topographica simulations
- Create a strong base for future development and optimisation
Was it successful with regards to that? Can't tell, it's really not for me to decide, let the markers do their job=) Am I satisfied? Yes. Anyway, a quick summary of achievements:
- We proved that it is possible to reach high performance with MPI in Topographica simulations
- x3.2 maximum speed-up factor, relative to the original single-processed implementation, has been recorded at realistic simulations with Lissom model at 142 cortex density, 24 LGN and retina densities (values used in most of examples for the CMVC book)
-We analysed the current range of available MPI tools and found the most suitable up to this date modules and packages
- Pinpointed the slow-down factors of both shared-memory and distributed models, on which further development efforts should be concentrated
- Created a strong base for further optimisations and development work
I remember once Jim (my supervisor) has mentioned that he had posted a proposal for the next year's Honours Projects - "High Performance Cortical Simulator", basically taking this project to the next level. He also said that someone was very interested and there was a fair chance this work would be picked from up from the point where I finished. Well, all I can say about this as someone who knows the current state of things with Topographica and MPI: it's not going to be easy in any way. However, having said this, I hope that the person won't change his or her mind and will continue my work. I wish that person all the best of luck and I would be very excited to hear about their progress. Maybe this blog will even get a second author, who knows!=)
What about myself? Well, in the short run, I still have some data from the experiments that I want to post here in the next few days, so stay tuned. In the long-run, I'm applying to EPCC (Edinburgh Parallel Computing Centre), for the MSC in High Performance Computing and to some other universities that do similar courses. If I get a place (fingers crossed), I will have more stuff to write about.
Well, perhaps that's all I wanted to say for now.
Yours,
Konstantin
Well, as I have finally submitted my thesis (Have to admit, this was the first paper of such kind that I have ever written in my life and, bloody hell, that was a nightmare!) the project is officially over now, and it's time to summarise the results. Perhaps, it would be useful to reiterate the objectives first (I'll just copy bits from my thesis):
Purpose of the project:
Optimising the Open-Source Neural Map Simulator Topographica by making it capable of distributing its heaviest computations over CPU cores of one machine or over a network of independent machines would allow users of Topographica to run experiments on a bigger scale than it was previously possible. This is expected to help neuroscientists have a better understanding of brain activity at the cortical level. Such understanding, apart from expanding the pool of human knowledge, could potentially assist in the creation of medicines to combat and, hopefully, cure mental diseases, and, possibly benefit the human race in other, as yet undiscovered ways.
Primary Objective
Improve the performance of Topographica simulations by using the High Performance Computing approach, allowing Topographica to take advantage of the computational power of both Multiprocessing and Cluster architectures.
Other Objectives
- Research and summarise how distributing computations with the Message Passing Interface can benefit Topographica
- Research what MPI software packages are currently available and can be utilised for optimising Topographica simulations
- Create a strong base for future development and optimisation
Was it successful with regards to that? Can't tell, it's really not for me to decide, let the markers do their job=) Am I satisfied? Yes. Anyway, a quick summary of achievements:
- We proved that it is possible to reach high performance with MPI in Topographica simulations
- x3.2 maximum speed-up factor, relative to the original single-processed implementation, has been recorded at realistic simulations with Lissom model at 142 cortex density, 24 LGN and retina densities (values used in most of examples for the CMVC book)
-We analysed the current range of available MPI tools and found the most suitable up to this date modules and packages
- Pinpointed the slow-down factors of both shared-memory and distributed models, on which further development efforts should be concentrated
- Created a strong base for further optimisations and development work
I remember once Jim (my supervisor) has mentioned that he had posted a proposal for the next year's Honours Projects - "High Performance Cortical Simulator", basically taking this project to the next level. He also said that someone was very interested and there was a fair chance this work would be picked from up from the point where I finished. Well, all I can say about this as someone who knows the current state of things with Topographica and MPI: it's not going to be easy in any way. However, having said this, I hope that the person won't change his or her mind and will continue my work. I wish that person all the best of luck and I would be very excited to hear about their progress. Maybe this blog will even get a second author, who knows!=)
What about myself? Well, in the short run, I still have some data from the experiments that I want to post here in the next few days, so stay tuned. In the long-run, I'm applying to EPCC (Edinburgh Parallel Computing Centre), for the MSC in High Performance Computing and to some other universities that do similar courses. If I get a place (fingers crossed), I will have more stuff to write about.
Well, perhaps that's all I wanted to say for now.
Yours,
Konstantin
Subscribe to:
Posts (Atom)