Day 113
Oh God, I did it! I figured out how to run topographica on a cluster, and it wasn't that easy. A very good guide that helped me was: Setting up a Beowulf Cluster Using Open MPI on Linux. Running MPI jobs points 1-4 are also useful. Just to summarise, in order to run an MPI job on a cluster, you need:
- Hostfile with hostnames of machines in the cluster
- OpenMPI installed on each machine in the cluster, in the same directory
- system PATH needs to know where OpenMPI's bin/ directory is
- system LD_LIBRARY_PATH needs to know where OpenMPI's lib/ directory is
- All common libraries that are going to be used during the runtime need to be installed in the same directory (i.e. path to them should be the same on all nodes). In my case, all topographica libraries are local and therefore I had to have it installed on each node in the same directory, e.g. ~/CODE/topographica
One note: if using an mpirun command with an absolute path to it (e.g. /usr/bin/openmpi/mpirun), then OpenMPI bin/ and lib/ directories will be added automatically to PATH and LD_LIBRARY_PATH respectively on all nodes.
Showing posts with label openmpi. Show all posts
Showing posts with label openmpi. Show all posts
Sunday, 21 March 2010
Tuesday, 9 March 2010
Day 101
Day 101: a note on PMI communications
There are a few ways for the Controller to communicate with the Workers. In most cases I'm using pmi.invoke. Invoke call takes function name to be executed on workers (as a string value), a list of arguments to be passed to workers and returns a list of results returned from workers (if any). For example, let's say we have a function "get_data" of "SomeClass" on workers that takes "data" as an argument and returns "modified_data". First we need to initialise SomeClass on all workers and get reference to it. pmi.create does the job. Let's say the reference is stored in "pmiobj" object. Now, calling
pmi.invoke(pmiobj,"get_data",data)
Would return (assume we have 4 nodes, and all run the same code) a list:
[modified_data,modified_data,modified_data,modified_data]
How that works: pmi.invoke simply passes data to the worker on node 0, which saves it (by reference!) and broadcasts to all other nodes, which involves pickling it on node 0 and unpickling on receiving by other nodes. If this data object does not implement Single Segment Buffer Interface (like numpy arrays, for instance) it sent, generally, 10 times slower than if it does. Ok, nothing new, I guess
However, the communication can be organised in different, slightly lower-level, fashion. Let's say, now instead of passing data and processing it on the nodes, we want workers to store the reference to "some_object" and then use data stored in that object later in some computations. This could be done by issuing pmi.invoke again. However, what if data in "some_object" gets modified on controller? All workers apart from the one on node 0 would have out-dated (which in many situations would mean useless) copy of "some_object", and would need to somehow get a new one, e.g. from the worker on node 0. In order to avoid redundant pickling/unpickling and network overheads we can use pmi localcall (or was it local_call?), that just passes data to the Worker on node 0. Node 0 can now store the reference and when needed by all nodes, broadcast data from the object using MPI bcast. This would ensure that all nodes will work with up to date copy of the data. Simple trick, but very useful, especially since sometimes pickling objects can have unpredictable results,(like for example, in my case, bloody dest..... spent hours debugging this!).
There are a few ways for the Controller to communicate with the Workers. In most cases I'm using pmi.invoke. Invoke call takes function name to be executed on workers (as a string value), a list of arguments to be passed to workers and returns a list of results returned from workers (if any). For example, let's say we have a function "get_data" of "SomeClass" on workers that takes "data" as an argument and returns "modified_data". First we need to initialise SomeClass on all workers and get reference to it. pmi.create does the job. Let's say the reference is stored in "pmiobj" object. Now, calling
pmi.invoke(pmiobj,"get_data",data)
Would return (assume we have 4 nodes, and all run the same code) a list:
[modified_data,modified_data,modified_data,modified_data]
How that works: pmi.invoke simply passes data to the worker on node 0, which saves it (by reference!) and broadcasts to all other nodes, which involves pickling it on node 0 and unpickling on receiving by other nodes. If this data object does not implement Single Segment Buffer Interface (like numpy arrays, for instance) it sent, generally, 10 times slower than if it does. Ok, nothing new, I guess
However, the communication can be organised in different, slightly lower-level, fashion. Let's say, now instead of passing data and processing it on the nodes, we want workers to store the reference to "some_object" and then use data stored in that object later in some computations. This could be done by issuing pmi.invoke again. However, what if data in "some_object" gets modified on controller? All workers apart from the one on node 0 would have out-dated (which in many situations would mean useless) copy of "some_object", and would need to somehow get a new one, e.g. from the worker on node 0. In order to avoid redundant pickling/unpickling and network overheads we can use pmi localcall (or was it local_call?), that just passes data to the Worker on node 0. Node 0 can now store the reference and when needed by all nodes, broadcast data from the object using MPI bcast. This would ensure that all nodes will work with up to date copy of the data. Simple trick, but very useful, especially since sometimes pickling objects can have unpredictable results,(like for example, in my case, bloody dest..... spent hours debugging this!).
Day 101
Day 101
A side note on PMI terminology I'm using: "Serial" can be also referred to as "the Controller" as well a "Parallel" or "Nodes" can be referred to as "Workers". Tonight while debugging my code I had to, for the first time, have a look at PMI source code. That actually helped... Ok, sleep now. I have to take care of other uni work, so probably I'll resume the Project not sooner than Friday. I really wish I will finish it in one go then...
A side note on PMI terminology I'm using: "Serial" can be also referred to as "the Controller" as well a "Parallel" or "Nodes" can be referred to as "Workers". Tonight while debugging my code I had to, for the first time, have a look at PMI source code. That actually helped... Ok, sleep now. I have to take care of other uni work, so probably I'll resume the Project not sooner than Friday. I really wish I will finish it in one go then...
Monday, 8 March 2010
Day 100
Day 100: PMI. Interesting observation #2
In this example I will demonstrate how changing numpy printoptions threshold affects the communication speed between Serial and Parallel. Honestly, I have no idea what printoptions threshold has to do with MPI or PMI communication speed and I am really tempted to investigate this issue further, but sadly don't have time for this at the moment. Anyway, output first (using mpirun with 2 nodes), then source code:
The numbers speak for themselves. The source code:
In this example I will demonstrate how changing numpy printoptions threshold affects the communication speed between Serial and Parallel. Honestly, I have no idea what printoptions threshold has to do with MPI or PMI communication speed and I am really tempted to investigate this issue further, but sadly don't have time for this at the moment. Anyway, output first (using mpirun with 2 nodes), then source code:
numpy_printoptions threshold set to 0
sending 200x200 numpy matrix with random elements
Sending x 10 times took 1.40654301643
numpy_printoptions threshold set to 201*201 (greater than the size of communicated matrix)
sending 200x200 numpy matrix with random elements
Sending x 10 times took 12.252920866
The numbers speak for themselves. The source code:
if __name__ != 'pmi':
##################################################
## Serial code
##################################################
import pmi
from time import time
from numpy.random import beta
from numpy import array, set_printoptions
pmi.setup()
pmi.execfile_(__file__)
x = beta(a=1,b=1000,size=(200,200))
# create a frontend class
class Hello(object):
def __init__(self, name):
self.pmiobj = pmi.create('HelloLocal', name)
def __call__(self):
return pmi.invoke(self.pmiobj, '__call__')
def time_send(self):
t0 = time()
for i in range(0,10):
pmi.invoke(self.pmiobj,'recv',x)
print "Sending x 10 times took",(time() - t0)
def time_recv(self):
t0 = time()
for i in range(0,10):
pmi.invoke(self.pmiobj,'send')
print "Receiving x 10 times took",(time() - t0)
# use the class
hello = Hello('Konstantin')
print "numpy_printoptions threshold set to 0"
set_printoptions(threshold=0)
print "sending 200x200 numpy matrix with random elements"
hello.time_send()
set_printoptions(threshold=201*201)
print "numpy_printoptions threshold set to 201*201 (greater than the size of communicated matrix)"
print "sending 200x200 numpy matrix with random elements"
hello.time_send()
else:
##################################################
## Parallel code
##################################################
from mpi4py import MPI
from numpy.random import beta
y = beta(a=1,b=1000,size=(100,100))
class HelloLocal(object):
def __init__(self, name):
self.name = name
def __call__(self):
return 'Hello %s, this is MPI task %d!' % (self.name, MPI.COMM_WORLD.rank)
def recv(self,x):
"do nothing"
def send(self):
return y
Day 100
Day 100: PMI. Interesting observation #1
In this example I will be testing if PMI passes objects from Serial into Parallel by reference or by value. First I will give the output of my little test script, and then the source code.
and the source code:
I think, the reason for this is very simple: PMI seems to be broadcasting the data to all nodes, when pmi.invoke is issued, (perhaps, using MPI bcast()) from node 0 and keep the broadcast data on node 0 as it is, i.e. reference to the object passed into pmi.invoke, and not the copy of this object.
In this example I will be testing if PMI passes objects from Serial into Parallel by reference or by value. First I will give the output of my little test script, and then the source code.
############ TEST 1: PASSING FROM SERIAL INTO PARALLEL ############
Testing PMI pass by reference: an array is created in Serial and then passed into Parallel
array before mpi.invoke: [1 2 3]
If the array was passed by _value_ then after modifying it in Parallel it should remain
unchanged in Serial
array after mpi.invoke: [1000 2 3]
As you can see, the array has been changed in serial as well, which proves that objects
are passed by reference in pmi.
############ TEST 2: PASSING FROM PARALLEL INTO SERIAL ############
Now we'll check if modifying the same array in Serial will make it change in Parallel
The array: [1000 2 3]
Changed the first element to be 999: [999 2 3]
Now calling pmi.invoke that will make Parallel print out the array (it has been stored
as a class atribute during the previous experiment)
Node 0 array: [999 2 3]
Node 1 array: [1000 2 3]
Oops! It seems like objects are passed by reference only between Serial and node 0 of parallel.
Let's verify that: the next pmi.invoke will call a method in Parallel that will attempt to set
the second array element to be 1000 on node 0 (and only on node 0) and the third element to be
1000 on node 1 (and node 1 only)
############ TEST 3: CHANGING DIFFERENT ELEMENTS ON DIFFERENT NODES ############
Once again, array before pmi.invoke: [999 2 3]
Array after pmi.invoke: [ 999 1000 3]
Ok, that proves the case that objects are indeed passed by reference in pmi between Serial
and Parallel,but only for the first node. For all other nodes they are passed by reference.
This is a bit tricky, but might be useful.
and the source code:
if __name__ != 'pmi':
##################################################
## Serial code
##################################################
import pmi
from time import time
from numpy import array
pmi.setup()
pmi.execfile_(__file__)
# create a frontend class
class Hello(object):
def __init__(self, name):
self.pmiobj = pmi.create('HelloLocal', name)
def __call__(self):
return pmi.invoke(self.pmiobj, '__call__')
def check_pass_by_ref1(self,arr):
pmi.invoke(self.pmiobj,"modify_array",arr)
def check_pass_by_ref2(self):
pmi.invoke(self.pmiobj,"print_array")
def check_pass_by_ref3(self):
pmi.invoke(self.pmiobj,"modify_array2")
# use the class
hello = Hello('Konstantin')
a = array([1,2,3])
print "############ TEST 1: PASSING FROM SERIAL INTO PARALLEL ############"
print "Testing PMI pass by reference: an array is created in Serial and then passed into Parallel"
print "array before mpi.invoke:",a
hello.check_pass_by_ref1(a)
print "If the array was passed by _value_ then after modifying it in Parallel it should remain"
print "unchanged in Serial"
print "array after mpi.invoke:",a
print "As you can see, the array has been changed in serial as well, which proves that objects"
print "are passed by reference in pmi."
# test 2
print ""
print "############ TEST 2: PASSING FROM PARALLEL INTO SERIAL ############"
print "Now we'll check if modifying the same array in Serial will make it change in Parallel"
print "The array:", a
a[0] = 999
print "Changed the first element to be 999:", a
print "Now calling pmi.invoke that will make Parallel print out the array (it has been stored"
print "as a class atribute during the previous experiment)"
hello.check_pass_by_ref2()
print "Oops! It seems like objects are passed by reference only between Serial and node 0 of parallel."
print "Let's verify that: the next pmi.invoke will call a method in Parallel that will attempt to set"
print "the second array element to be 1000 on node 0 (and only on node 0) and the third element to be"
print " 1000 on node 1 (and node 1 only)"
print ""
print "############ TEST 3: CHANGING DIFFERENT ELEMENTS ON DIFFERENT NODES ############"
print "Once again, array before pmi.invoke:",a
hello.check_pass_by_ref3()
print "Array after pmi.invoke:",a
print ""
print "Ok, that proves the case that objects are indeed passed by reference in pmi between Serial"
print "and Parallel,but only for the first node. For all other nodes they are passed by reference."
print " This is a bit tricky, but might be useful."
else:
##################################################
## Parallel code
##################################################
from mpi4py import MPI
from numpy.random import beta
y = beta(a=1,b=1000,size=(210,210))
class HelloLocal(object):
def __init__(self, name):
self.name = name
self.rank = MPI.COMM_WORLD.Get_rank()
def __call__(self):
return 'Hello %s, this is MPI task %d!' % (self.name, MPI.COMM_WORLD.rank)
def modify_array(self,arr):
self.arr = arr
self.arr[0] = 1000
def modify_array2(self):
if self.rank== 0:
self.arr[1] = 1000
if self.rank== 1:
self.arr[2] = 1000
def print_array(self):
print "Node",self.rank,"array:",self.arr
I think, the reason for this is very simple: PMI seems to be broadcasting the data to all nodes, when pmi.invoke is issued, (perhaps, using MPI bcast()) from node 0 and keep the broadcast data on node 0 as it is, i.e. reference to the object passed into pmi.invoke, and not the copy of this object.
Wednesday, 24 February 2010
Day 88
Day 88: The moment of truth
Benchmarking. The time has come. This evening I will test the performance of my implementation and update this post as I get results with each test. The machine I'm going to use is a shared-memory multicore with 8 Intel(R) Xeon(R) X5450 CPU's, 3GHz, 6MB cache, 4 cores each. This little monster is called Jupiter III and together with Jupiters I and II forms a 3-piece cluster.
So, at the moment I am testing the test-bench and soon will be ready to launch. Don't switch the channel=)
[Update]
Phew! Preparation for testing took me way more time than I expected... However, look at this beautiful script! (./topographica line's were created by Chris, everything else is mine)
Now to do testing you just need to run this script with 3 parameters: -c for cortex density, i for the number of iterations and -n for the number of processes to use. I already tried this out on my machine and so far the results are not that inspiring:
./test_script -c 25 -n 2 -i 100
Serial: 5.36015510559
MPI: 6.15542793274
On the bright side, the results computed in MPI are still correct. Now I'm going to try this out on Jupiter.
[update 2] Ok, found a bug in the script: it ran tiny.ty instead of tiny_mpi.ty. How the hell did it get correct results? Anyway, new output (my laptop):
./test_script -c 25 -n 2 -i 100
Not using MPI
Not using MPI
Serial: 1.02229189873
Using MPI
NODE 0 initialized
NODE 1 initialized
MPI: 3.37540221214
[update 3] Ok, I need to make a break now. For some reason Jupiter won't let me run the tests, so I'll have to deal with that somehow. But later.
Benchmarking. The time has come. This evening I will test the performance of my implementation and update this post as I get results with each test. The machine I'm going to use is a shared-memory multicore with 8 Intel(R) Xeon(R) X5450 CPU's, 3GHz, 6MB cache, 4 cores each. This little monster is called Jupiter III and together with Jupiters I and II forms a 3-piece cluster.
So, at the moment I am testing the test-bench and soon will be ready to launch. Don't switch the channel=)
[Update]
Phew! Preparation for testing took me way more time than I expected... However, look at this beautiful script! (./topographica line's were created by Chris, everything else is mine)
#!/bin/bash
# Argument = -c cortex density -n number of cpu's -i number of iterations
usage()
{
cat << EOF
usage: $0 options
This script tests the performance of parallel Topographica implementation
OPTIONS:
-c cortex density
-n number of cpu's
-i number of iterations
EOF
}
C_DENSITY=
N_CPU=
ITERATIONS=
while getopts “c:n:i:” OPTION
do
case $OPTION in
c)
C_DENSITY=$OPTARG
;;
n)
N_CPU=$OPTARG
;;
i)
ITERATIONS=$OPTARG
;;
?)
usage
exit
;;
esac
done
if [[ -z $C_DENSITY ]] || [[ -z $N_CPU ]] || [[ -z $ITERATIONS ]]
then
usage
exit 1
fi
# Mock run to make sure that all C code has compiled before proceeding to actual testing
./topographica -p cortex_density=1 examples/tiny.ty -c "topo.sim.run(1)"
# Serial run
./topographica -p cortex_density=$C_DENSITY examples/tiny.ty -c "import timeit; print 'Serial:', timeit.Timer('topo.sim.run($ITERATIONS)','import topo').timeit(number=1)" -c 'import pickle; pickle.dump(topo.sim["V1"].activity,open("results.pickle","wb"))'
# Parallel run
mpirun -n $N_CPU ./topographica -p mpi=True -p cortex_density=$C_DENSITY examples/tiny.ty -c "import timeit; print 'MPI:', timeit.Timer('topo.sim.run($ITERATIONS)','import topo').timeit(number=1)" -c 'import pickle; from numpy.testing import assert_array_equal; previous_result=pickle.load(open("results.pickle","r")); assert_array_equal(topo.sim["V1"].activity,previous_result)'
Now to do testing you just need to run this script with 3 parameters: -c for cortex density, i for the number of iterations and -n for the number of processes to use. I already tried this out on my machine and so far the results are not that inspiring:
./test_script -c 25 -n 2 -i 100
Serial: 5.36015510559
MPI: 6.15542793274
On the bright side, the results computed in MPI are still correct. Now I'm going to try this out on Jupiter.
[update 2] Ok, found a bug in the script: it ran tiny.ty instead of tiny_mpi.ty. How the hell did it get correct results? Anyway, new output (my laptop):
./test_script -c 25 -n 2 -i 100
Not using MPI
Not using MPI
Serial: 1.02229189873
Using MPI
NODE 0 initialized
NODE 1 initialized
MPI: 3.37540221214
[update 3] Ok, I need to make a break now. For some reason Jupiter won't let me run the tests, so I'll have to deal with that somehow. But later.
Tuesday, 2 February 2010
Day 65
Day 65
I just thought I haven't posted for a while, so this is a quick update on what's currently going on.
First, this Monday my laptop has refused to switch on and now seems to be pretty much dead, but hopefully this is just coma and anyways my warranty hasn't expired yet, so this Wednesday it's going to be picked up and hopefully returned in a week, fully functional. Not a big issue, just a bit annoying.
Second, some time ago Chris suggested I have a look at PMI - "a pure python module that allows libraries to provide functions that are parallelized using MPI but that can nonetheless be called from serial Python scripts." I'll write about it if I find the way to utilize this thing, but at the moment it seems to be quite a handy module. However, here's what I want to get rid of in my implementation by using PMI:
This class allows master node to communicate with slave nodes. Potentially, PMI will eliminate all need to have such class.
[update] No, it will not. What PMI allows you to do is hide the parallel implementation, so that your programme looks like it is a serial (single-processed) code. However, it does not enhance the underlying MPI implementation in any way, so it still needs to be implicit. What does that mean in my case? Well, if I was to introduce PMI into my code as it is now, then there would have been a class - something like CFProjection_proxy, linked to the real (i.e. parallel) implementation of CFProjection_slave using PMI mechanism. CFPRojection_slave's would still have to communicate with the master node CFProjection using MPI_io (the code listed above) because PMI does not influence or change the way different nodes of an MPI application communicate. In simple words, it just makes your code look like it's not using MPI, when it actually does.
Well, so what's so bad about it? Why not use PMI anyway, since from all that it looks like it's actually doing a good job? Actually, there's nothing wrong with it and it might be a good idea to use it. No, it won't add any new functionality, but it will definitelly make things look a lot nicer. However, I prefer dealing with things one at a time, and also finishing what I started, so I think I'll concentrate on looking for the solution to my initial problem.
To summarise what I'm looking for, have a look at this: Java RMI . Without going into too much detail, RMI is a mechanism that allows Java objects calling each other's methods accross several different JVM's (which can be run on different hosts!) and it stands for Remote Method Invocation. Combined with serialization (something similar to Python's pickling), it allows to hide all explicit communication-level code away from the user, which is great! Thus, my assumption is that if something exists in Java, it has a chance to exist in Python. As simple as that.
Searching goes on.
I just thought I haven't posted for a while, so this is a quick update on what's currently going on.
First, this Monday my laptop has refused to switch on and now seems to be pretty much dead, but hopefully this is just coma and anyways my warranty hasn't expired yet, so this Wednesday it's going to be picked up and hopefully returned in a week, fully functional. Not a big issue, just a bit annoying.
Second, some time ago Chris suggested I have a look at PMI - "a pure python module that allows libraries to provide functions that are parallelized using MPI but that can nonetheless be called from serial Python scripts." I'll write about it if I find the way to utilize this thing, but at the moment it seems to be quite a handy module. However, here's what I want to get rid of in my implementation by using PMI:
'''
Created on 20 Jan 2010
@author: megatelevizor
'''
from mpi4py import MPI
import time
from MPI_client import client
""" Vocabulary
a = assign flatcfs
c = compute activity
i = Initialize
i_a = Get a copy of input_activity matrix
k = Kill
p = Print
r = Return activity
s = Set strength
"""
class communicator(object):
def listen(self):
self.comm = MPI.COMM_WORLD
self.rank = self.comm.Get_rank()
self.client_node = client()
while True:
while not self.comm.Iprobe(source=0, tag=0):
time.sleep(0.1)
command_prefix = self.comm.recv(source=0, tag=0)
""" Print """
if command_prefix == "p":
i_command_body = self.comm.recv(source=0,tag=1)
self.client_node.print_command(command = i_command_body)
""" Kill """
if command_prefix == "k":
exit()
""" Initialize MPI_CFProjection_slave"""
if command_prefix == "i":
self.client_node.init_mpi_cfprojection_slave(node_id=self.rank)
""" Assign flatcfs"""
if command_prefix == "a":
new_flatcfs = self.comm.recv(source=0,tag=1)
print "Node",self.rank,": received", len(new_flatcfs),"flatcfs"
self.client_node.assign_cfs(new_flatcfs)
""" Set a copy of input activity matrix """
if command_prefix == "i_a":
input_activity = self.comm.recv(source=0,tag=1)
self.client_node.set_input_activity(input_activity)
""" Compute activity """
if command_prefix == "c":
self.client_node.compute_activity()
""" Return activity """
if command_prefix == "r":
activity = self.client_node.get_activity()
print "Node",self.rank,"activity:",activity
self.comm.send(obj=activity, dest=0, tag=2)
""" Set strength"""
if command_prefix == "s":
strength = self.comm.recv(source=0,tag=1)
self.client_node.set_strength(strength)
""" TAGS:
0 - command prefix
1 - input data
2 - output data
"""
This class allows master node to communicate with slave nodes. Potentially, PMI will eliminate all need to have such class.
[update] No, it will not. What PMI allows you to do is hide the parallel implementation, so that your programme looks like it is a serial (single-processed) code. However, it does not enhance the underlying MPI implementation in any way, so it still needs to be implicit. What does that mean in my case? Well, if I was to introduce PMI into my code as it is now, then there would have been a class - something like CFProjection_proxy, linked to the real (i.e. parallel) implementation of CFProjection_slave using PMI mechanism. CFPRojection_slave's would still have to communicate with the master node CFProjection using MPI_io (the code listed above) because PMI does not influence or change the way different nodes of an MPI application communicate. In simple words, it just makes your code look like it's not using MPI, when it actually does.
Well, so what's so bad about it? Why not use PMI anyway, since from all that it looks like it's actually doing a good job? Actually, there's nothing wrong with it and it might be a good idea to use it. No, it won't add any new functionality, but it will definitelly make things look a lot nicer. However, I prefer dealing with things one at a time, and also finishing what I started, so I think I'll concentrate on looking for the solution to my initial problem.
To summarise what I'm looking for, have a look at this: Java RMI . Without going into too much detail, RMI is a mechanism that allows Java objects calling each other's methods accross several different JVM's (which can be run on different hosts!) and it stands for Remote Method Invocation. Combined with serialization (something similar to Python's pickling), it allows to hide all explicit communication-level code away from the user, which is great! Thus, my assumption is that if something exists in Java, it has a chance to exist in Python. As simple as that.
Searching goes on.
Monday, 7 December 2009
Day 7
Day 7
07/12/2009 04:03
Sorry for a very short note, but it's 4am and I really want to go sleep. Another day full of revision. However, I managed to find time to write some code for topographica, and I finally implemented CFProxy. It works now! The entire amount of code I wrote today would not exceed 10 lines, but that's not the point - I know how it all works now and ....... now I need to find a different way to make it work since blocking receive still eats up a hell of a lot of CPU, bastard. I'll show what I mean exactly by posting my code tomorrow, but now I need a rest.
I'm meeting Jim (my supervisor) tomorrow at noon and don't really want to be in a zombie state then, so ... yeah, bed time.
[by the way, by "tomorrow" I actually mean "today". I usually refer to "tomorrow" as to the time after I wake up, even if it's on the same day. Just in case, wanted to clarify]
07/12/2009 04:03
Sorry for a very short note, but it's 4am and I really want to go sleep. Another day full of revision. However, I managed to find time to write some code for topographica, and I finally implemented CFProxy. It works now! The entire amount of code I wrote today would not exceed 10 lines, but that's not the point - I know how it all works now and ....... now I need to find a different way to make it work since blocking receive still eats up a hell of a lot of CPU, bastard. I'll show what I mean exactly by posting my code tomorrow, but now I need a rest.
I'm meeting Jim (my supervisor) tomorrow at noon and don't really want to be in a zombie state then, so ... yeah, bed time.
[by the way, by "tomorrow" I actually mean "today". I usually refer to "tomorrow" as to the time after I wake up, even if it's on the same day. Just in case, wanted to clarify]
Day 6
Day 6
06/12/2009 23:50
Revising for the exam. No work done for the project today, regretfully. I hate Engineering Project Management.
06/12/2009 23:50
Revising for the exam. No work done for the project today, regretfully. I hate Engineering Project Management.
Saturday, 5 December 2009
Day 5
Day 5
05/12/2009 01:40
I've figured out what I need to do and how to do it, so creating a connection field proxy won't be too hard now. However, I encountered a new problem:
Until this point I was working only with blocking communication methods of mpi4py, ignoring the non-blocking ones. From the mpi4py manual:
Blocking Communications
MPI provides basic send and receive functions that are blocking. These functions block the caller until the data buffers involved in the communication can be safely reused by the application program.
In MPI for Python, the Send(), Recv() and Sendrecv() methods of communicator objects provide support
for blocking point-to-point communications within Intracomm and Intercomm instances. These methods can communicate memory buffers. The variants send(), recv() and sendrecv() can communicate general
Python objects.
Nonblocking Communications
On many systems, performance can be significantly increased by overlapping communication and computation. This is particularly true on systems where communication can be executed autonomously by an intelligent, dedi-cated communication controller.
MPI provides nonblocking send and receive functions. They allow the possible overlap of communication and computation. Non-blocking communication always come in two parts: posting functions, which begin the re-quested operation; and test-for-completion functions, which allow to discover whether the requested operation has completed.
In MPI for Python, the Isend() and Irecv() methods of the Comm class initiate a send and receive oper-
ation respectively. These methods return a Request instance, uniquely identifying the started operation. Its
completion can be managed using the Test(), Wait(), and Cancel() methods of the Request class. The
management of Request objects and associated memory buffers involved in communication requires a careful, rather low-level coordination. Users must ensure that objects exposing their memory buffers are not accessed at the Python level while they are involved in nonblocking message-passing operations.
Often a communication with the same argument list is repeatedly executed within an inner loop. In such cases, communication can be further optimized by using persistent communication, a particular case of nonblocking communication allowing the reduction of the overhead between processes and communication controllers. Furthermore , this kind of optimization can also alleviate the extra call overheads associated to interpreted, dynamic languages like Python.
In MPI for Python, the Send_init() and Recv_init() methods of the Comm class create a persistent request
for a send and receive operation respectively. These methods return an instance of the Prequest class, a subclass of the Request class. The actual communication can be effectively started using the Start() method, and its completion can be managed as previously described.
That's all fine, but the problem with recv() (blocking receive) is that if it has been called but nothing is sent to it yet (for example, another process spends some time computing the data it's about to send), the waiting process will eat up as much CPU resources as it can until it receives the data, which is a very bad thing if you're thinking optimisation.
Seems like now I have to look closer at non-blocking communication or something else, which is fine except for there are no tutorials for that provided with the mpi4py manual, API contains almost no useful info at all and googling for examples gave 0 results. Seems like there isn't a single person in the whole wide world is using mpi4py apart from me!!! Damn it.
However, it feels nice to be pioneering it=))
05/12/2009 01:40
I've figured out what I need to do and how to do it, so creating a connection field proxy won't be too hard now. However, I encountered a new problem:
Until this point I was working only with blocking communication methods of mpi4py, ignoring the non-blocking ones. From the mpi4py manual:
Blocking Communications
MPI provides basic send and receive functions that are blocking. These functions block the caller until the data buffers involved in the communication can be safely reused by the application program.
In MPI for Python, the Send(), Recv() and Sendrecv() methods of communicator objects provide support
for blocking point-to-point communications within Intracomm and Intercomm instances. These methods can communicate memory buffers. The variants send(), recv() and sendrecv() can communicate general
Python objects.
Nonblocking Communications
On many systems, performance can be significantly increased by overlapping communication and computation. This is particularly true on systems where communication can be executed autonomously by an intelligent, dedi-cated communication controller.
MPI provides nonblocking send and receive functions. They allow the possible overlap of communication and computation. Non-blocking communication always come in two parts: posting functions, which begin the re-quested operation; and test-for-completion functions, which allow to discover whether the requested operation has completed.
In MPI for Python, the Isend() and Irecv() methods of the Comm class initiate a send and receive oper-
ation respectively. These methods return a Request instance, uniquely identifying the started operation. Its
completion can be managed using the Test(), Wait(), and Cancel() methods of the Request class. The
management of Request objects and associated memory buffers involved in communication requires a careful, rather low-level coordination. Users must ensure that objects exposing their memory buffers are not accessed at the Python level while they are involved in nonblocking message-passing operations.
Often a communication with the same argument list is repeatedly executed within an inner loop. In such cases, communication can be further optimized by using persistent communication, a particular case of nonblocking communication allowing the reduction of the overhead between processes and communication controllers. Furthermore , this kind of optimization can also alleviate the extra call overheads associated to interpreted, dynamic languages like Python.
In MPI for Python, the Send_init() and Recv_init() methods of the Comm class create a persistent request
for a send and receive operation respectively. These methods return an instance of the Prequest class, a subclass of the Request class. The actual communication can be effectively started using the Start() method, and its completion can be managed as previously described.
That's all fine, but the problem with recv() (blocking receive) is that if it has been called but nothing is sent to it yet (for example, another process spends some time computing the data it's about to send), the waiting process will eat up as much CPU resources as it can until it receives the data, which is a very bad thing if you're thinking optimisation.
Seems like now I have to look closer at non-blocking communication or something else, which is fine except for there are no tutorials for that provided with the mpi4py manual, API contains almost no useful info at all and googling for examples gave 0 results. Seems like there isn't a single person in the whole wide world is using mpi4py apart from me!!! Damn it.
However, it feels nice to be pioneering it=))
Day 4
Day 4
04/12/2009 21:09
Today I created an implementation plan for ConnectionField proxy.
In the nutshell, ConnectionField objects are where Topographica stores matrices of neural weights and some other stuff. I don't want to explain what they are exactly, because that would take up to a couple of pages of text and wouldn't be quite related to what I was about to write in this post. Maybe I'll do that some other day, but for now treat them as wrappers around 2d arrays of float values, because that's what they are, essentially.
I started my proxy implementation plan from trying to manually instantiate a ConnectionField object from Topographica prompt (basically, a modified Python prompt):
That was supposed to create a ConnectionField c holding a 3x3 matrix, but instead gave an error, complaining about some masks (err... maybe voodoo masks???). A week or two ago Jim (my Project Supervisor) said he had the same kind of problem, then Chris fixed this. I decided to merge trunk into my branch on the svn to get this fix, however after updating my local copy from the svn branch, I noticed that some of topographica's files which I never touched before got modified. That was weird. Tried instantiating ConnectionField again and ... it worked!!! Either someone has merged trunk into my branch for me or this all is a complete mistery. Anyway, I wasn't sure how to merge svn branches so I'm glad I didn't have to spend time on this.
04/12/2009 21:09
Today I created an implementation plan for ConnectionField proxy.
In the nutshell, ConnectionField objects are where Topographica stores matrices of neural weights and some other stuff. I don't want to explain what they are exactly, because that would take up to a couple of pages of text and wouldn't be quite related to what I was about to write in this post. Maybe I'll do that some other day, but for now treat them as wrappers around 2d arrays of float values, because that's what they are, essentially.
I started my proxy implementation plan from trying to manually instantiate a ConnectionField object from Topographica prompt (basically, a modified Python prompt):
topo_t000000.00_c1>>> from topo.base.cf import ConnectionField, CFSheet
topo_t000000.00_c2>>> s = CFSheet()
topo_t000000.00_c3>>> c = ConnectionField(s)
That was supposed to create a ConnectionField c holding a 3x3 matrix, but instead gave an error, complaining about some masks (err... maybe voodoo masks???). A week or two ago Jim (my Project Supervisor) said he had the same kind of problem, then Chris fixed this. I decided to merge trunk into my branch on the svn to get this fix, however after updating my local copy from the svn branch, I noticed that some of topographica's files which I never touched before got modified. That was weird. Tried instantiating ConnectionField again and ... it worked!!! Either someone has merged trunk into my branch for me or this all is a complete mistery. Anyway, I wasn't sure how to merge svn branches so I'm glad I didn't have to spend time on this.
Friday, 4 December 2009
Day 3
Day3
03/12/2009 16:48
Hi.
There was a problem with mpi4py on Jupiter (not on my laptop though) that was bothering me for quite some time. When trying to do "from mpi4py import MPI" from Python prompt, the system gave:
/topographica/bin/python: symbol lookup error: topographica/lib/openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int
(sometimes it was complaining about other .so files). Chris (a PhD student that helps me with this project) has found a solution for that: before executing the topographica startup script cast a bash voodoo spell
(solution found at http://code.google.com/p/petsc4py/issues/detail?id=14)
Today I spent quite a few hours trying to add this export statement to topographica's startup script:
and failed miserably. Doing
(or environ instead of setenv) didn't give any results. I'll leave that for now, can't spend any more time on this issue.
03/12/2009 16:48
Hi.
There was a problem with mpi4py on Jupiter (not on my laptop though) that was bothering me for quite some time. When trying to do "from mpi4py import MPI" from Python prompt, the system gave:
/topographica/bin/python: symbol lookup error: topographica/lib/openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int
(sometimes it was complaining about other .so files). Chris (a PhD student that helps me with this project) has found a solution for that: before executing the topographica startup script cast a bash voodoo spell
export LD_PRELOAD=topographica/lib/libmpi.so
(solution found at http://code.google.com/p/petsc4py/issues/detail?id=14)
Today I spent quite a few hours trying to add this export statement to topographica's startup script:
#!/disk/scratch/fast/s********/topographica/bin/python
# Startup script for Topographica
import topo
topo.release='0.9.6'
topo.version='10827'
# Process the command-line arguments
from sys import argv
from topo.misc.commandline import process_argv
process_argv(argv[1:])
and failed miserably. Doing
import os;
os.setenv["LD_PRELOAD"] = "topographica/lib/libmpi.so"
(or environ instead of setenv) didn't give any results. I'll leave that for now, can't spend any more time on this issue.
Wednesday, 2 December 2009
Day 1
Intro Note
Hi,
A couple of hours ago I discovered that Kubuntu came with a nice Outlook-like tool called Kontact, so I decided to have a closer look at it. One of all thepotentially (but not really) useful things it had was Journal, and I thought "well, keeping a journal is something I haven't done for a good while, so it might be fun", and decided to write this in it. Chances are, I'll get bored with writing notes very quickly and since Journal is an off-line tool (actually, I might be wrong), nobody will ever see these notes. However, if you're reading this it means three things:
1. I didn't get bored.
2. Keeping a Journal is fun, apparently!
3. I decided to publish my journal
S-s-o. Currently I am working on a project of optimising a massive chunk of open-source Python/C code called Topographica through parallelisation with OpenMPI and mpi4py, as part of my Honours studies. This journal is going to be primarily about that, so if the previous sentence didn't make any kind of sense to you, DON'T READ!!! Otherwise the information you receive from this journal will be misinterpreted by your Lateral Geniculate Nucleus which might result in static electricity being generated in some parts of your brain, which, in turn, might interfere with normal magnetic flows resulting in neural malfunctioning, examples of which could be nausea, fever, blindness, internal bleeding, brain tumor, heart attack or even lack of appetite! However, usually you only get constipation which is not a very nice thing, but at least not fatal.
I warned you!
Also, I'm not an English native speaker. On top of that, there's no spellcheck in Kontact's Journal [update: however, there is a spellcheck in my browser, so there will be less errors in my blog], so if you spot some grammatical errors / typos / sentences that don't make any kind of sense,I don't give a f... do apologise.
A few words about the project: I started it a couple of months ago, and because of Topographica being a massive project and many other reasons, all that time I was mostly busy setting things up, reading stuff, figuring out how and what I am supposed to do and so on. Only now I came to the point when I am actually able to write some code that is supposed to do some useful (-ish) work for Topo. Plus, I have no idea how many days exactly have passed since I started working on this project (and even if I knew calling this day a "day 67" without having "days 1..66" would've been weird). Therefore today will be the Day 1.
Ok, That's probably all I wanted to say in my introduction, so ... if you're still reading (and weren't terrified by the Lateral Geniculate Thing), then thanks for reading. Hope someone finds this useful.
Day 1
Ok, It's December 1st now, semester has finished, and apart from the Project Management exam on the 14th, I don't have much to do for about a month or so. It seems like a perfect time to get some work done for the Project.
What I have:
A. Topographica installed and working on:
1. My laptop
2, My university account (further related to as DICE)
3. Cluster network (further related to as Jupiter 1, 2 or 3)
B. Small mpi4py implementation of a master-slave application that I've written a few weeks ago that does something similar to what I want Topo to do.
C. Some knowledge of how Topo's code works, however still weak. With all respect, God, I hate Python syntax!
What I want to have:
Implement a proxy-like model of ConnectionField, which will be slower than the surrent version but might give me some ideas what to do next and will be a good exercise in any case.
Strategy and approach:
1. Tomorrow, the first thing to do: create a detailed plan of implementation.
2. Try not to forget about the Management exam. This shouldn't really be a problem, but it's a good idea to start taking care of such things as early as possible.
3. Write some code ffs!!!
Ok. First note. Over.
Hi,
A couple of hours ago I discovered that Kubuntu came with a nice Outlook-like tool called Kontact, so I decided to have a closer look at it. One of all the
1. I didn't get bored.
2. Keeping a Journal is fun, apparently!
3. I decided to publish my journal
S-s-o. Currently I am working on a project of optimising a massive chunk of open-source Python/C code called Topographica through parallelisation with OpenMPI and mpi4py, as part of my Honours studies. This journal is going to be primarily about that, so if the previous sentence didn't make any kind of sense to you, DON'T READ!!! Otherwise the information you receive from this journal will be misinterpreted by your Lateral Geniculate Nucleus which might result in static electricity being generated in some parts of your brain, which, in turn, might interfere with normal magnetic flows resulting in neural malfunctioning, examples of which could be nausea, fever, blindness, internal bleeding, brain tumor, heart attack or even lack of appetite! However, usually you only get constipation which is not a very nice thing, but at least not fatal.
I warned you!
Also, I'm not an English native speaker. On top of that, there's no spellcheck in Kontact's Journal [update: however, there is a spellcheck in my browser, so there will be less errors in my blog], so if you spot some grammatical errors / typos / sentences that don't make any kind of sense,
A few words about the project: I started it a couple of months ago, and because of Topographica being a massive project and many other reasons, all that time I was mostly busy setting things up, reading stuff, figuring out how and what I am supposed to do and so on. Only now I came to the point when I am actually able to write some code that is supposed to do some useful (-ish) work for Topo. Plus, I have no idea how many days exactly have passed since I started working on this project (and even if I knew calling this day a "day 67" without having "days 1..66" would've been weird). Therefore today will be the Day 1.
Ok, That's probably all I wanted to say in my introduction, so ... if you're still reading (and weren't terrified by the Lateral Geniculate Thing), then thanks for reading. Hope someone finds this useful.
Day 1
Ok, It's December 1st now, semester has finished, and apart from the Project Management exam on the 14th, I don't have much to do for about a month or so. It seems like a perfect time to get some work done for the Project.
What I have:
A. Topographica installed and working on:
1. My laptop
2, My university account (further related to as DICE)
3. Cluster network (further related to as Jupiter 1, 2 or 3)
B. Small mpi4py implementation of a master-slave application that I've written a few weeks ago that does something similar to what I want Topo to do.
C. Some knowledge of how Topo's code works, however still weak. With all respect, God, I hate Python syntax!
What I want to have:
Implement a proxy-like model of ConnectionField, which will be slower than the surrent version but might give me some ideas what to do next and will be a good exercise in any case.
Strategy and approach:
1. Tomorrow, the first thing to do: create a detailed plan of implementation.
2. Try not to forget about the Management exam. This shouldn't really be a problem, but it's a good idea to start taking care of such things as early as possible.
3. Write some code ffs!!!
Ok. First note. Over.
Subscribe to:
Posts (Atom)