C vs Python. Hector, 13 nodes, 1-52 cores

A few more graphs.

Topographica:

C-Topo

I'm not sure what's causing performance go down at 52 cores for C. Last time in a similar experiment (except with 10000 density instead of 14400) 52 cores were showing very good efficiency. This could be a one-off glitch. Anyway, the point of this post is: topographica is still worse, even when subscribing only 4 cores per node.

C vs Python. Hector, 1 node SMP, 1-24 cores

Speedups:

Topographica:

C-Topo:



Parallel Efficiencies:

                Topographica  C-Topo
cpu_cores: 1    E = 99.0%     E = 99.5%
cpu_cores: 2    E = 95.4%     E = 98.3%
cpu_cores: 4    E = 89.3%     E = 96.3%
cpu_cores: 8    E = 78.4%     E = 91.7%
cpu_cores: 16   E = 66.1%     E = 90.6%
cpu_cores: 24   E = 55.4%     E = 86.2%

script output

luckily missing module warnings disappeared after I removed module swaps from .bashrc

TOPOGRAPHICA RUNS

### Topographica ###
Application 1013295 resources: utime ~1199s, stime ~63s real 83.19 user 0.00 sys 0.00

### C-Topo ###
Application 1013303 resources: utime ~301s, stime ~3s real 14.70 user 0.18 sys 0.01

### Comparing the results of c-topo and topographica ###
Results matched

Graphs will shortly follow, first for single-node run.

update

Good news and somewhat confusing news. The good news is: it finally runs and produces timings!!! I couldn't be happier about it. The confusing news is this: I have no idea why it runs, and the output it produces is even more confusing (script output,not the timings; timings are fine). When I first tried running topographica from my /work directory it ran but probably froze, because it only completed when it ran out of allocated time (10 minutes). Here's what I found in the output file:

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

Of course they are not loaded! I swapped PrgEnv-pgi for PrgEnv-gnu and xt-mpt for xt-mpich2 (mpi4py wouldn't build otherwise). Then I tried swapping the modules back, without recompiling mpi4py. And it worked! Why did it not segfault? Apparently, this means that you can build mpi4py with one set of MPI libraries and run with another?.. However, that's not it: despite the correct execution, the new output file still contained

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-pgi' is currently not loaded
ModuleCmd_Switch.c(172):ERROR:152: Module 'xt-mpt' is currently not loaded

huh? Ok, I'm just happy it runs, at last. I have to admit, making topographica run on Hector was a royal pain in the...

the full error output

messages between these ">>>>>" "<<<<<" are printed exceptions that break execution, I added the print statements into weave/catalog.py:

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011' <<<<< >>>> [Errno 2] No such file or directory: '/home/d04/d04' <<<<< >>>> [Errno 17] File exists: '/home/d04' <<<<< Traceback (most recent call last): File "/var/spool/alps/1009053/topographica_mpi", line 15, in <module> process_argv(argv[1:]) File "/esfs2/d04/d04/s0676011/topographica/topo/misc/commandline.py", line 474, in process_argv execfile(filename,__main__.__dict__) File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ cp.pbs.o321605 create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/basic.py", line 31, in <module> from topo.base.patterngenerator import PatternGenerator,Constant File "/esfs2/d04/d04/s0676011/topographica/topo/base/patterngenerator.py", line 238, in <module> from topo.misc.inlinec import inline,c_header File "/esfs2/d04/d04/s0676011/topographica/topo/misc/inlinec.py", line 59, in <module> import weave File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/__init__.py", line 9, in <module> from blitz_tools import blitz File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/blitz_tools.py", line 11, in <module> import inline_tools File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/inline_tools.py", line 15, in <module> function_catalog = catalog.catalog() File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 351, in __init__ sys.path.append(default_dir()) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 202, in default_dir create_dir(path) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 128, in create_dir create_dir(base) File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 131, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Weave

>>>> [Errno 2] No such file or directory: '/home/d04/d04/s0676011/.python26_compiled' <<<<<

The directory exists (I mkdir'ed it), obviously. I even chmod'ed 777 permissions for it. Seems like the root of the problem is the same as with matplotlib: Hector's backend can't access files and directories on the frontend. Except for unlike with matplotlib, I can't tell topographica to ignore Weave

Are there any shared directories I could make use of? Otherwise, I'm really puzzled...

One thing I don't understand: my C app creates an xml file (timings), which I'm quite sure it does on the backend (the file is created from withing the main body of the code) and that does not cause any trouble with permissions. However, for python libraries it does - could the problem be somewhere else?

Weird error message

Not sure what's going on there (from somewhere in the middle of the same error output as the last message in the previous post was taken from):

File "examples/matchmodel.ty", line 23, in <module> from topo import transferfn,pattern,projection,responsefn,learningfn,sheet,numbergen File "/esfs2/d04/d04/s0676011/topographica/topo/transferfn/__init__.py", line 25, in <module> from basic import a1.out bin but c_activity.out ChangeLog.txt compare_arrays.py contrib COPYING.txt cp.pbs cp.pbs~ create_topographica_script.py c_timings.xml c_vs_python.sh debian demo doc draw_graph.py etc examples experiment.py external filename hostfile images include info ld_preload lib Makefile #Makefile# man matplotlib-bug.o mpi_experiments mpitest.py param parsetime.py README.txt results.pickle setup_app.py setup_exe.py setup_ez.py _setup.py setup.py share test_scripts timed_topographica.sh topo topographica topographica_mpi useful_info

What is it importing from basic?.. That's the contents of topographica's root directory

Topographica on Hector

Today Adrian helped me out with the libgfortran bug: the library I was looking for was inside one of the subdirectories of gcc, namely /opt/gcc/4.5.2/snos/lib. I have no clue why find /opt/ -name 'libgfortran.so.3' wasn't picking it up.


The next issue - topographica not being able to find it's source code (the topo module). This was fixed by adding  the following line into my .pbs script:

export PYTHONPATH=$PBS_O_WORKDIR/topo:$PYTHONPATH

The next issue was matplotlib unable to create .matplotlib directory anywhere:

RuntimeError: Failed to create /home/d04/d04/s0676011/.matplotlib; 
consider setting MPLCONFIGDIR to a writable directory for matplotlib configuration data

I tried $HOME, $PBS_O_WORKDIR, work and topographica directories and even ./  . Nothing helped, I'm still not sure why. Neither did any solutions I found on the web. What fixed that problem was tracing the import and adding exception to the file that was importing matplotlib (topo/misc/commandline.py):

try:
    # Import matplotlib, if available, and use a non-GUI backend by default
    import matplotlib
    matplotlib_imported=True
    from matplotlib import rcParams
    rcParams['backend']='Agg'
except ImportError:
    print "Warning: Could not import matplotlib; pylab plots will not work."
except RuntimeError as e: #KKALERT: added this to be able to run on hector                   
    print "Warning: pylab plots will not work:",e

Thing I'm stuck with at the moment:

File "/esfs2/d04/d04/s0676011/topographica/lib/python2.6/site-packages/weave/catalog.py", line 129, in create_dir os.mkdir(p) OSError: [Errno 17] File exists: '/home/d04'

Starting to think that launching Topographica on Hector is a project on it's own...

mpi4py and hector

Trying to compile mpi4py with gcc didn't get me anywhere, even after following what I was suggested at the mpi4py google group. I did manage after all to compile it with gcc after swapping several libraries. Now however, I'm stuck with this:

Traceback (most recent call last):
  File "mpitest.py", line 1, in <module>
    from mpi4py import MPI
ImportError: libgfortran.so.3: cannot open shared object file: No such file or directory

I've spent a whole day trying fix this and didn't get any further this error message. The library just isn't there and I'm not 100% sure that this is what mpi4py is trying to find. Why does it even need fortran's libraries? i'm stuck...

mpi4py and Cray's compiler

They don't seem to work together. Basically, the problem is that mpi4py is by default compiled with gcc, however I need to compile it with cc instead to be able to run parallel jobs with aprun. Now, cc gives a bunch of unrecognised compiler flag options (it doesn't understand gcc's flags) when trying to build mpi4py with it. I think I tried pretty much everything I could with what I found on the web to make it work, but no luck so far... I think the options now are:

1) get mpi4py to compile with cc
2) use MPI that is based on gcc (OpenMPI? MPICH2?) if that's possible on Hector.

I posted about this on mpi4py's google group board, seems like they already had to deal with this issue before, so hoping to get some help from there.

Also, am I getting it right: cc actually uses Portland's compiler (pgcc) under the hood? It appears so from the error messages I'm getting