Day 57

Day 57: Simulations. Part I

Hi! I remember myself promising to post some info on simulations here and perhaps give a better idea of what I'm working on. I think, I'll break it into several parts: the first one will be just a brief intro, a very top-level description. In part II we'll go through the simplest simulation script: tiny.ty. Maybe there will be even part III and IV, but let's not go that far yet.

Also, it's kinda funny that I do that. I don't believe there is a single person in the world who reads this apart from my project supervisor Jim (hopefully) and his Ph.D. student Chris, who's offered me an incredible amount of help on this project, and neither of these two people need to be explained how their code works (especially since I'm not entirely sure my explanation is going to be correct). Anyway, there you go:


This is what a sample Topographica vision system model might look like:







However, the simplest model would only include an image from the retina of an eye connected to V1, the primary visual cortex. In Topographica's implementation an image from the retina is represented as the Input Activity matrix where each element corresponds to the brightness value of each pixel of the image. This matrix could be generated from an image file or randomly, using various distributions.

V1, or the primary visual cortex is a matrix of Connection Fields, which are sets of weights on one input Sheet. Each Connection Field contributes to the activity of one unit on the output sheet, and is normally used as part of a Projection including many other Connection Fields.

The simplest simulation would involve generating an output activity matrix using an input activity matrix and Projection, which can be described as a connection between two sheets of activities: input and output. More specifically, a Projection is a Connection that can produce an Activity matrix when given an input Activity matrix, which will typically be used by the destination Sheet when it computes its activation. ^[2]

Computing the activation requires applying a response function to the input matrix and weights on the Connection Fields. In our example this function would simply dot-product slices of the input activity matrix with the corresponding weight matrices.

Day 56

Day 56

I'm in the process of writing interim report at the moment. God, this is so boring and useless... Well, actually there is one good thing about it: after creating a more or less detailed timeline I realised that for the next couple of months my personal life has seized to exist. Reminds last year's System Design Project a little bit, except for there's no team this time, just me. I loved SDP! There's a saying that wise people learn from their own mistakes and smart people learn from others'. Well, looking from that perspective I'm definitely a wise person. I think that all the f*** ups we had made as a team and each one of us personally during this project were very well worth getting the relatively low mark (~60%). I mean, I don't like getting low marks, but this one brought me a hell of a lot of experience. And it counts for much. Also, I think I've done a good job overall, made new friends and learned a lot, not only from mistakes.

Well, anyway I guess this is where the fun part begins... except for this report isn't much fun, to be honest. Allright, time to get back to it.

Day 55

Hi. Here's my to-do list for the first implementation. Everything in green is successfully implemented (accomplished), yellow - not required any more or not my responsibility and red - failures.

1. Modify the topographica script so that it runs with mpiexec

2. Make a new subclass of CFProjection, e.g. MultiProcessorCFProjection , that will be a plug-in replacement for CFProjection but with parallel capabilities. At first, MultiProcessorCFProjection does exactly the same thing as CFProjection. Make your test .ty file(s) use this class instead of CFProjection.

3. To make the rest of the steps easier, turn off optimization (remove _opt from the various components in your copy of tiny.ty or other testing file), so that you don't have to worry about the C code's implicit assumptions. Also turn off learning, for now. Also turn off randomness in the initial weights -- use Gaussian patterns for now.

4. Find out how to instantiate CFProjection

5. Once that works, change the constructor or other initialization routines, if any, for MultiProcessorCFProjection to divide the work between compute nodes from the start, so that each node only has a copy of the weights relevant to the processing that it has to do. At first, it's probably easiest if the master node (rank 0) has a copy of all the weights, while the others have only a subset. While in this stage, we can debug everything using the usual plotting and analysis mechanisms that use the data from the master node, without having to reimplement all of that just yet. Implementation plan:

1. [Chris] Refactor CFProjection to replace the list of lists of cfs (self._cfs) with a flattened list (self.flatcfs?), to remove the assumption that all CFs are stored in 2D arrays. This will require changing all of the C code, which is a pain, but it should actually significantly simplify it.

2. [Chris] Make the MaskedCFiter use self.flatcfs instead of the self.cfs numpy object array. Again, this should simplify the implementation of MaskedCFIter , while making it simpler to keep everything coherent. Actually, we should probably just make the mask update the flatcfs list whenever it changes, so that only unmasked cfs are in the list; otherwise we'll have a problem relating the 2D mask to the 1D list. Up to this point, none of the changes should involve multiprocessing at all.

3. When multiprocessing (or on a single processor just for testing), split up the list and distribute it to multiple Python processes, so that any operation invoked on MultiprocessorCFProjection is actually distributed across all processors and run on MultiprocessorCFProjection running on that processor, using its local list of cfs. It's probably best just to assert numpy.alltrue(mask) (or whatever is syntactically correct), and not worry about masks at all.

   1. Find out what exactly CFSheet consists of, and how to distribute it

      1. Figure out what topo.sim['V1'].activity is

      2. Read through the rest of the class

      3. Read through the ProjectionSheet class

   2. Figure out how simulation.connect and simulation.run work

   3. Find out how exactly CFProjection is dotproducting cfs

   4. Figure out what input_sheet_slice.submatrix() is

   5. Figure out how CFIter works and how to iterate from and to a certain point in matrix

   6. Create the s/r mechanism

   7. Using the s/r mechanism, send cfs to slave nodes and store. Check if sending and receiving work

   8. Create new activate() method in MPI_CFProjection

      1. At each call broadcast the input_activity matrix

         1. Create according method in MPI_io

         2. Create according method in MPI_client

         3. Add broadcasting to MPI_CFProjection.activate()

      2. Send an appropriate portion of CF's to each node

         1. Create according method in MPI_io

         2. Create according method in MPI_client

         3. Add broadcasting to MPI_CFProjection.activate()

      3. Gather resulting activities into one matrix

         1. Compute activity on all nodes

         2. Create methods (io,client?) for sending back resulting activities

         3. Gather activities into new matrix at MPI_CFProjection

            1. Transform the non-zero part into a list

            2. Send

            3. Gather at master node and put into the activity array, calculating appropriate position for gathered activity values

   9. Commit to GIT

   10. Debug

   11. Commit again

   12. Test, test, test...

   13. Talk to Jim and Chris

Day 54

Day 54: a historical moment


Ok, now I can officially announce that a few days ago I've reached a major milestone: the slowest ever and probably buggiest as well, however distributed implementation of topographica has been created. I've reached the bottom, and now it's time to start climbing to the top, creating the fastest (and coolest!) version of topographica the World has seen. It's a bit too early to dring champaigne or whatever, but now I have no doubts: this project will be delivered.

Thank you. Questions?

P.S. I have a mountain of work to do on this weekend, so I really don't know whether I will be able to write about simulations, but I haven't forgotten my promise, so stay tuned!

Day 51

Day 51

Good God! First success!!!!

[update] Either I don't get something, or there is indeed something wrong with mpi4py's non-blocking communication. If non-blocking communication of arbitrary objects was introduced in version 1.2 then why does Irecv() still not work the same way as recv(). True, Irecv does not require it's buffer to be tupled with MPI basic types, which allows pickling. But why not get rid of buffer parameter completely? Anyway, I've switched back to blocking communication already, so it's fine.

[update] Beware of KKALERT's, hehe!

Day ...

Day whatever it is.

Wrong results again... Still trying to bring the chaos I've created in my new implementation back to order. I would love to share my ideas and code with you, but at the moment that would be absolutely pointless, and, besides, embarrassing. Going to bed now, maybe tomorrow will be more productive. Think my next informative post will be about looking closer at Topographica's simulations - .ty files, simulation.py etc. In my opinion, it's quite a sophisticated mechanism, that I still don't have a full understanding of, but regard as a very fine piece of Python code. If everything goes well, I'll write about it on the weekend.

Day 44

Day 44

Hi! I'm back in Edinburgh and finally (almost) done dealing with all the small to medium issues that stacked up while I was away, so I think I'll resume writing pretty soon. In the meanwhile, want to say that I'm glad to be back. I've got about 2.5 month left till the deadline for this project and I'm not quite where I want to be with it, so it's about time to accelerate. Watch me ;)

Day 35

Day 35

I'm in Singapore till Thursday. Don't think I'll do much work till I get back to Edinburgh.

Day 34

Day 34: A story.

A brilliant post from a brilliant blog: http://sethgodin.typepad.com/seths_blog/2009/12/how-far-away-is-your-future.html .

Good point.

A few years ago, when I was a first year student at Edinburgh University (I'm 4th now) I had an Opportunity. Two of my friends were dealing Georgian Wine in Edinburgh (God, that was some wonderful wine!). They'd been doing it for a few years by then already, and were only selling wine to local shops in bulk. Once I asked one of them why they were not dealing wine on the Web, and he said they didn't know how. Then I offered my help and they agreed to let me into their business. However, I never managed to proceed any further than that. Given the amount of free time I had in my first year at uni, I definitely could've made it work out, if not for profit (It was a really small business and both guys left UK a year ago anyway), then at least for the extremely valuable entrepreneurial experience.

What was my excuse then? Very simple - I thought I had too much time. Have you ever had anything similar?

On the positive side, back then I've learned a lesson: grab it as soon as you see it, which, interestingly enough, in real life works for me more like start caring about it a moment before it's too late. Anyway, whatever gets the job done.

How much of the pre-paid "too much" time do you think you still have in your life? What is your allowance for f*** ups like the one I made 3 years ago?

If you ask me, I've ran out of mine.

Day 32

Day 32

Oops, it's 2010 already...