Day 10
Two words that make me want to puke, sleep and bang my head against a wall - all at the same time: Contract Management.
Friday, 11 December 2009
Wednesday, 9 December 2009
Day 9
Day 9
Hi. As promised, I'm going to post some code here today. I'll give a relatively simple example of MPI Dynamic Process Management which is good for two reasons:
a) It shows in a very basic way how to spawn processes and how mpi4py communicators work
b) It reproduces the CPU issue I described earlier
[update: blogger editor removes empty spaces and tabs from the beginning of a line and it screwed up my Python indentation entirely, so be careful]
master_test.py :
Hi. As promised, I'm going to post some code here today. I'll give a relatively simple example of MPI Dynamic Process Management which is good for two reasons:
a) It shows in a very basic way how to spawn processes and how mpi4py communicators work
b) It reproduces the CPU issue I described earlier
[update: blogger editor removes empty spaces and tabs from the beginning of a line and it screwed up my Python indentation entirely, so be careful]
master_test.py :
"""The following directives are not compulsory. However, my Python interpreter was complaining about encoding so I included these"""slave_test.py :
#!/usr/bin/python
# -*- coding: iso-8859-15 -*-
from mpi4py import MPI
import sys
"""The number of processes we want to spawn"""
numprocs = 5
"""Objects comm1 and comm2 will serve for communicating with two independent of each other groups of processes. The following commands spawn these groups with 5 processes in the first one and 6 in the second one."""
comm1 = MPI.COMM_SELF.Spawn(sys.executable,args=['slave_test.py'], maxprocs=numprocs)
comm2 = MPI.COMM_SELF.Spawn(sys.executable,args=['slave_test.py'], maxprocs=numprocs+1)
print "Type some text and press Return please: "
"""Receiving user input"""
uin = raw_input()
uin1 = "FROM MASTER TO G1: " + uin
uin2 = "FROM MASTER TO G2: " + uin
"""Sending data to all processes in both groups. First parameter of the send() function is data that we want to send, second - rank of a target process, third - tag. Tags can be used for managing sequential messanges, although this can be totally ignored in the example"""
for i in range(0,numprocs):
comm1.send(uin1, i,0)
comm2.send(uin2, i,0)
#endfor
"""Since there's one more process in the second group, we need to send data there as well"""
comm2.send(uin2,numprocs,0)
"""Receiving input from the first group process with rank 4"""
data = comm1.recv(source=4,tag=1)
print "Master: ", data
#!/usr/bin/pythonNow, if you run master_test.py, but before typing in text, go to a different terminal window (that's for UNIX-like OS users. For Windows users - get yourself a Linux distro, install it, rejoice and go to a different terminal window), type "top", and see what I've been talking about earlier - 11 python processes sitting on your CPU and pretending to do a lot of work! However, you might not have this problem if you're using something different from OpenMPI (or, perhaps a newer version of OpenMPI?).
# -*- coding: iso-8859-15 -*-
from mpi4py import MPI
"""This object will serve for communicating with Master process"""
comm = MPI.Comm.Get_parent()
"""Getting the rank of the process (a number between 0 and the number of processes spawned in this group)"""
rank = comm.Get_rank()
data = ''
"""Receiving data from process with rank 0. Here it is very important to understand that comm object (initialised as MPI.Comm.Get_parent) links to the parent group where there is only one process - Master, and it has rank 0. In other words, there are 3 separate groups of processes in our system - Master (with 1 node - Master itself), first group of slaves (with 5 nodes and ranks from 0 to 4) and second group of slaves (with 6 nodes and ranks from 0 to 5)"""
data = comm.recv(source=0,tag=0)
print "Slave", comm.rank, "/", comm.size-1, ": ","["+ data + "]"
"""The following bit of code illustrates how different processes can communicate within one group and with the Master process. A message will be sent from node 0 of group 1 to node 4 of group 1 and then from node 4 - to Master"""
"""Getting a communication object for inter-group communicating"""
group_comm = MPI.COMM_WORLD
if group_comm.Get_rank()==0 and comm.size==5:
data = "FROM SLAVE 0 TO SLAVE 4: " + "[" + data + "]"
"""Sending data to process 4"""
group_comm.send(data,4,1)
#endif
if group_comm.Get_rank()==4 and comm.size==5:
data = group_comm.recv(source=0,tag=1)
print "Slave", group_comm.rank, "/", group_comm.size-1, ": ","[" + data + "]"
data = "FROM SLAVE 4 TO MASTER: " + "[" + data + "]"
"""Notice: using comm, not group_comm, since now sending data to Master. Calling the same send command on group_comm would send data to slave process 0, back to where it came from."""
comm.send(data,0,1)
#endif
Day8
Day 8
Hi.
I won't post any code here today, sorry. I spent entire day revising for EPM exam and covered one of the biggest topics of that course: Cost Estimation. To illustrate how much I am excited about that I will give an example of what I've learned today (taken from notes):
Doesn't it look like the most important thing I've learned over the last 3.5 years of studying informatics?
Hi.
I won't post any code here today, sorry. I spent entire day revising for EPM exam and covered one of the biggest topics of that course: Cost Estimation. To illustrate how much I am excited about that I will give an example of what I've learned today (taken from notes):
Doesn't it look like the most important thing I've learned over the last 3.5 years of studying informatics?
Monday, 7 December 2009
Day 7
Day 7
07/12/2009 04:03
Sorry for a very short note, but it's 4am and I really want to go sleep. Another day full of revision. However, I managed to find time to write some code for topographica, and I finally implemented CFProxy. It works now! The entire amount of code I wrote today would not exceed 10 lines, but that's not the point - I know how it all works now and ....... now I need to find a different way to make it work since blocking receive still eats up a hell of a lot of CPU, bastard. I'll show what I mean exactly by posting my code tomorrow, but now I need a rest.
I'm meeting Jim (my supervisor) tomorrow at noon and don't really want to be in a zombie state then, so ... yeah, bed time.
[by the way, by "tomorrow" I actually mean "today". I usually refer to "tomorrow" as to the time after I wake up, even if it's on the same day. Just in case, wanted to clarify]
07/12/2009 04:03
Sorry for a very short note, but it's 4am and I really want to go sleep. Another day full of revision. However, I managed to find time to write some code for topographica, and I finally implemented CFProxy. It works now! The entire amount of code I wrote today would not exceed 10 lines, but that's not the point - I know how it all works now and ....... now I need to find a different way to make it work since blocking receive still eats up a hell of a lot of CPU, bastard. I'll show what I mean exactly by posting my code tomorrow, but now I need a rest.
I'm meeting Jim (my supervisor) tomorrow at noon and don't really want to be in a zombie state then, so ... yeah, bed time.
[by the way, by "tomorrow" I actually mean "today". I usually refer to "tomorrow" as to the time after I wake up, even if it's on the same day. Just in case, wanted to clarify]
Day 6
Day 6
06/12/2009 23:50
Revising for the exam. No work done for the project today, regretfully. I hate Engineering Project Management.
06/12/2009 23:50
Revising for the exam. No work done for the project today, regretfully. I hate Engineering Project Management.
Saturday, 5 December 2009
Day 5
Day 5
05/12/2009 01:40
I've figured out what I need to do and how to do it, so creating a connection field proxy won't be too hard now. However, I encountered a new problem:
Until this point I was working only with blocking communication methods of mpi4py, ignoring the non-blocking ones. From the mpi4py manual:
Blocking Communications
MPI provides basic send and receive functions that are blocking. These functions block the caller until the data buffers involved in the communication can be safely reused by the application program.
In MPI for Python, the Send(), Recv() and Sendrecv() methods of communicator objects provide support
for blocking point-to-point communications within Intracomm and Intercomm instances. These methods can communicate memory buffers. The variants send(), recv() and sendrecv() can communicate general
Python objects.
Nonblocking Communications
On many systems, performance can be significantly increased by overlapping communication and computation. This is particularly true on systems where communication can be executed autonomously by an intelligent, dedi-cated communication controller.
MPI provides nonblocking send and receive functions. They allow the possible overlap of communication and computation. Non-blocking communication always come in two parts: posting functions, which begin the re-quested operation; and test-for-completion functions, which allow to discover whether the requested operation has completed.
In MPI for Python, the Isend() and Irecv() methods of the Comm class initiate a send and receive oper-
ation respectively. These methods return a Request instance, uniquely identifying the started operation. Its
completion can be managed using the Test(), Wait(), and Cancel() methods of the Request class. The
management of Request objects and associated memory buffers involved in communication requires a careful, rather low-level coordination. Users must ensure that objects exposing their memory buffers are not accessed at the Python level while they are involved in nonblocking message-passing operations.
Often a communication with the same argument list is repeatedly executed within an inner loop. In such cases, communication can be further optimized by using persistent communication, a particular case of nonblocking communication allowing the reduction of the overhead between processes and communication controllers. Furthermore , this kind of optimization can also alleviate the extra call overheads associated to interpreted, dynamic languages like Python.
In MPI for Python, the Send_init() and Recv_init() methods of the Comm class create a persistent request
for a send and receive operation respectively. These methods return an instance of the Prequest class, a subclass of the Request class. The actual communication can be effectively started using the Start() method, and its completion can be managed as previously described.
That's all fine, but the problem with recv() (blocking receive) is that if it has been called but nothing is sent to it yet (for example, another process spends some time computing the data it's about to send), the waiting process will eat up as much CPU resources as it can until it receives the data, which is a very bad thing if you're thinking optimisation.
Seems like now I have to look closer at non-blocking communication or something else, which is fine except for there are no tutorials for that provided with the mpi4py manual, API contains almost no useful info at all and googling for examples gave 0 results. Seems like there isn't a single person in the whole wide world is using mpi4py apart from me!!! Damn it.
However, it feels nice to be pioneering it=))
05/12/2009 01:40
I've figured out what I need to do and how to do it, so creating a connection field proxy won't be too hard now. However, I encountered a new problem:
Until this point I was working only with blocking communication methods of mpi4py, ignoring the non-blocking ones. From the mpi4py manual:
Blocking Communications
MPI provides basic send and receive functions that are blocking. These functions block the caller until the data buffers involved in the communication can be safely reused by the application program.
In MPI for Python, the Send(), Recv() and Sendrecv() methods of communicator objects provide support
for blocking point-to-point communications within Intracomm and Intercomm instances. These methods can communicate memory buffers. The variants send(), recv() and sendrecv() can communicate general
Python objects.
Nonblocking Communications
On many systems, performance can be significantly increased by overlapping communication and computation. This is particularly true on systems where communication can be executed autonomously by an intelligent, dedi-cated communication controller.
MPI provides nonblocking send and receive functions. They allow the possible overlap of communication and computation. Non-blocking communication always come in two parts: posting functions, which begin the re-quested operation; and test-for-completion functions, which allow to discover whether the requested operation has completed.
In MPI for Python, the Isend() and Irecv() methods of the Comm class initiate a send and receive oper-
ation respectively. These methods return a Request instance, uniquely identifying the started operation. Its
completion can be managed using the Test(), Wait(), and Cancel() methods of the Request class. The
management of Request objects and associated memory buffers involved in communication requires a careful, rather low-level coordination. Users must ensure that objects exposing their memory buffers are not accessed at the Python level while they are involved in nonblocking message-passing operations.
Often a communication with the same argument list is repeatedly executed within an inner loop. In such cases, communication can be further optimized by using persistent communication, a particular case of nonblocking communication allowing the reduction of the overhead between processes and communication controllers. Furthermore , this kind of optimization can also alleviate the extra call overheads associated to interpreted, dynamic languages like Python.
In MPI for Python, the Send_init() and Recv_init() methods of the Comm class create a persistent request
for a send and receive operation respectively. These methods return an instance of the Prequest class, a subclass of the Request class. The actual communication can be effectively started using the Start() method, and its completion can be managed as previously described.
That's all fine, but the problem with recv() (blocking receive) is that if it has been called but nothing is sent to it yet (for example, another process spends some time computing the data it's about to send), the waiting process will eat up as much CPU resources as it can until it receives the data, which is a very bad thing if you're thinking optimisation.
Seems like now I have to look closer at non-blocking communication or something else, which is fine except for there are no tutorials for that provided with the mpi4py manual, API contains almost no useful info at all and googling for examples gave 0 results. Seems like there isn't a single person in the whole wide world is using mpi4py apart from me!!! Damn it.
However, it feels nice to be pioneering it=))
Day 4
Day 4
04/12/2009 21:09
Today I created an implementation plan for ConnectionField proxy.
In the nutshell, ConnectionField objects are where Topographica stores matrices of neural weights and some other stuff. I don't want to explain what they are exactly, because that would take up to a couple of pages of text and wouldn't be quite related to what I was about to write in this post. Maybe I'll do that some other day, but for now treat them as wrappers around 2d arrays of float values, because that's what they are, essentially.
I started my proxy implementation plan from trying to manually instantiate a ConnectionField object from Topographica prompt (basically, a modified Python prompt):
That was supposed to create a ConnectionField c holding a 3x3 matrix, but instead gave an error, complaining about some masks (err... maybe voodoo masks???). A week or two ago Jim (my Project Supervisor) said he had the same kind of problem, then Chris fixed this. I decided to merge trunk into my branch on the svn to get this fix, however after updating my local copy from the svn branch, I noticed that some of topographica's files which I never touched before got modified. That was weird. Tried instantiating ConnectionField again and ... it worked!!! Either someone has merged trunk into my branch for me or this all is a complete mistery. Anyway, I wasn't sure how to merge svn branches so I'm glad I didn't have to spend time on this.
04/12/2009 21:09
Today I created an implementation plan for ConnectionField proxy.
In the nutshell, ConnectionField objects are where Topographica stores matrices of neural weights and some other stuff. I don't want to explain what they are exactly, because that would take up to a couple of pages of text and wouldn't be quite related to what I was about to write in this post. Maybe I'll do that some other day, but for now treat them as wrappers around 2d arrays of float values, because that's what they are, essentially.
I started my proxy implementation plan from trying to manually instantiate a ConnectionField object from Topographica prompt (basically, a modified Python prompt):
topo_t000000.00_c1>>> from topo.base.cf import ConnectionField, CFSheet
topo_t000000.00_c2>>> s = CFSheet()
topo_t000000.00_c3>>> c = ConnectionField(s)
That was supposed to create a ConnectionField c holding a 3x3 matrix, but instead gave an error, complaining about some masks (err... maybe voodoo masks???). A week or two ago Jim (my Project Supervisor) said he had the same kind of problem, then Chris fixed this. I decided to merge trunk into my branch on the svn to get this fix, however after updating my local copy from the svn branch, I noticed that some of topographica's files which I never touched before got modified. That was weird. Tried instantiating ConnectionField again and ... it worked!!! Either someone has merged trunk into my branch for me or this all is a complete mistery. Anyway, I wasn't sure how to merge svn branches so I'm glad I didn't have to spend time on this.
Friday, 4 December 2009
Day 3
Day3
03/12/2009 16:48
Hi.
There was a problem with mpi4py on Jupiter (not on my laptop though) that was bothering me for quite some time. When trying to do "from mpi4py import MPI" from Python prompt, the system gave:
/topographica/bin/python: symbol lookup error: topographica/lib/openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int
(sometimes it was complaining about other .so files). Chris (a PhD student that helps me with this project) has found a solution for that: before executing the topographica startup script cast a bash voodoo spell
(solution found at http://code.google.com/p/petsc4py/issues/detail?id=14)
Today I spent quite a few hours trying to add this export statement to topographica's startup script:
and failed miserably. Doing
(or environ instead of setenv) didn't give any results. I'll leave that for now, can't spend any more time on this issue.
03/12/2009 16:48
Hi.
There was a problem with mpi4py on Jupiter (not on my laptop though) that was bothering me for quite some time. When trying to do "from mpi4py import MPI" from Python prompt, the system gave:
/topographica/bin/python: symbol lookup error: topographica/lib/openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int
(sometimes it was complaining about other .so files). Chris (a PhD student that helps me with this project) has found a solution for that: before executing the topographica startup script cast a bash voodoo spell
export LD_PRELOAD=topographica/lib/libmpi.so
(solution found at http://code.google.com/p/petsc4py/issues/detail?id=14)
Today I spent quite a few hours trying to add this export statement to topographica's startup script:
#!/disk/scratch/fast/s********/topographica/bin/python
# Startup script for Topographica
import topo
topo.release='0.9.6'
topo.version='10827'
# Process the command-line arguments
from sys import argv
from topo.misc.commandline import process_argv
process_argv(argv[1:])
and failed miserably. Doing
import os;
os.setenv["LD_PRELOAD"] = "topographica/lib/libmpi.so"
(or environ instead of setenv) didn't give any results. I'll leave that for now, can't spend any more time on this issue.
Thursday, 3 December 2009
Day 2
Day 2
02/12/2009 16:18
Spent doing all sorts of wonderful things: going to Christmas Fair with friends (by the way, this was awesome!!!), then having a pint in pub which is, apparently, Best Scottish Pub 2009 (God knows why. It's nice and everything, but I know at least a dozen of pubs in Edinburgh that are not an ounce worse than this one), playing video games.... Not working though. Great evening and multiple positive emotions. Hello Topo!
02/12/2009 16:18
Spent doing all sorts of wonderful things: going to Christmas Fair with friends (by the way, this was awesome!!!), then having a pint in pub which is, apparently, Best Scottish Pub 2009 (God knows why. It's nice and everything, but I know at least a dozen of pubs in Edinburgh that are not an ounce worse than this one), playing video games.... Not working though. Great evening and multiple positive emotions. Hello Topo!
Wednesday, 2 December 2009
Day 1
Intro Note
Hi,
A couple of hours ago I discovered that Kubuntu came with a nice Outlook-like tool called Kontact, so I decided to have a closer look at it. One of all thepotentially (but not really) useful things it had was Journal, and I thought "well, keeping a journal is something I haven't done for a good while, so it might be fun", and decided to write this in it. Chances are, I'll get bored with writing notes very quickly and since Journal is an off-line tool (actually, I might be wrong), nobody will ever see these notes. However, if you're reading this it means three things:
1. I didn't get bored.
2. Keeping a Journal is fun, apparently!
3. I decided to publish my journal
S-s-o. Currently I am working on a project of optimising a massive chunk of open-source Python/C code called Topographica through parallelisation with OpenMPI and mpi4py, as part of my Honours studies. This journal is going to be primarily about that, so if the previous sentence didn't make any kind of sense to you, DON'T READ!!! Otherwise the information you receive from this journal will be misinterpreted by your Lateral Geniculate Nucleus which might result in static electricity being generated in some parts of your brain, which, in turn, might interfere with normal magnetic flows resulting in neural malfunctioning, examples of which could be nausea, fever, blindness, internal bleeding, brain tumor, heart attack or even lack of appetite! However, usually you only get constipation which is not a very nice thing, but at least not fatal.
I warned you!
Also, I'm not an English native speaker. On top of that, there's no spellcheck in Kontact's Journal [update: however, there is a spellcheck in my browser, so there will be less errors in my blog], so if you spot some grammatical errors / typos / sentences that don't make any kind of sense,I don't give a f... do apologise.
A few words about the project: I started it a couple of months ago, and because of Topographica being a massive project and many other reasons, all that time I was mostly busy setting things up, reading stuff, figuring out how and what I am supposed to do and so on. Only now I came to the point when I am actually able to write some code that is supposed to do some useful (-ish) work for Topo. Plus, I have no idea how many days exactly have passed since I started working on this project (and even if I knew calling this day a "day 67" without having "days 1..66" would've been weird). Therefore today will be the Day 1.
Ok, That's probably all I wanted to say in my introduction, so ... if you're still reading (and weren't terrified by the Lateral Geniculate Thing), then thanks for reading. Hope someone finds this useful.
Day 1
Ok, It's December 1st now, semester has finished, and apart from the Project Management exam on the 14th, I don't have much to do for about a month or so. It seems like a perfect time to get some work done for the Project.
What I have:
A. Topographica installed and working on:
1. My laptop
2, My university account (further related to as DICE)
3. Cluster network (further related to as Jupiter 1, 2 or 3)
B. Small mpi4py implementation of a master-slave application that I've written a few weeks ago that does something similar to what I want Topo to do.
C. Some knowledge of how Topo's code works, however still weak. With all respect, God, I hate Python syntax!
What I want to have:
Implement a proxy-like model of ConnectionField, which will be slower than the surrent version but might give me some ideas what to do next and will be a good exercise in any case.
Strategy and approach:
1. Tomorrow, the first thing to do: create a detailed plan of implementation.
2. Try not to forget about the Management exam. This shouldn't really be a problem, but it's a good idea to start taking care of such things as early as possible.
3. Write some code ffs!!!
Ok. First note. Over.
Hi,
A couple of hours ago I discovered that Kubuntu came with a nice Outlook-like tool called Kontact, so I decided to have a closer look at it. One of all the
1. I didn't get bored.
2. Keeping a Journal is fun, apparently!
3. I decided to publish my journal
S-s-o. Currently I am working on a project of optimising a massive chunk of open-source Python/C code called Topographica through parallelisation with OpenMPI and mpi4py, as part of my Honours studies. This journal is going to be primarily about that, so if the previous sentence didn't make any kind of sense to you, DON'T READ!!! Otherwise the information you receive from this journal will be misinterpreted by your Lateral Geniculate Nucleus which might result in static electricity being generated in some parts of your brain, which, in turn, might interfere with normal magnetic flows resulting in neural malfunctioning, examples of which could be nausea, fever, blindness, internal bleeding, brain tumor, heart attack or even lack of appetite! However, usually you only get constipation which is not a very nice thing, but at least not fatal.
I warned you!
Also, I'm not an English native speaker. On top of that, there's no spellcheck in Kontact's Journal [update: however, there is a spellcheck in my browser, so there will be less errors in my blog], so if you spot some grammatical errors / typos / sentences that don't make any kind of sense,
A few words about the project: I started it a couple of months ago, and because of Topographica being a massive project and many other reasons, all that time I was mostly busy setting things up, reading stuff, figuring out how and what I am supposed to do and so on. Only now I came to the point when I am actually able to write some code that is supposed to do some useful (-ish) work for Topo. Plus, I have no idea how many days exactly have passed since I started working on this project (and even if I knew calling this day a "day 67" without having "days 1..66" would've been weird). Therefore today will be the Day 1.
Ok, That's probably all I wanted to say in my introduction, so ... if you're still reading (and weren't terrified by the Lateral Geniculate Thing), then thanks for reading. Hope someone finds this useful.
Day 1
Ok, It's December 1st now, semester has finished, and apart from the Project Management exam on the 14th, I don't have much to do for about a month or so. It seems like a perfect time to get some work done for the Project.
What I have:
A. Topographica installed and working on:
1. My laptop
2, My university account (further related to as DICE)
3. Cluster network (further related to as Jupiter 1, 2 or 3)
B. Small mpi4py implementation of a master-slave application that I've written a few weeks ago that does something similar to what I want Topo to do.
C. Some knowledge of how Topo's code works, however still weak. With all respect, God, I hate Python syntax!
What I want to have:
Implement a proxy-like model of ConnectionField, which will be slower than the surrent version but might give me some ideas what to do next and will be a good exercise in any case.
Strategy and approach:
1. Tomorrow, the first thing to do: create a detailed plan of implementation.
2. Try not to forget about the Management exam. This shouldn't really be a problem, but it's a good idea to start taking care of such things as early as possible.
3. Write some code ffs!!!
Ok. First note. Over.
Subscribe to:
Posts (Atom)