Linux on Power Open Source Documentation

Recompiling the GM Myrinet driver

We distribute RPM packages for the GM Myrinet driver here. These RPMs were built with a precise kernel level that might be different from the one you are running. If you wish to recompile the GM driver to match you kernel, here are the steps to follow :

Check your prerequisites

You must have the kernel source installed, as well as the gcc compiler and the cross-compilers (cross-ppc64-*) installed. It's also very important to configure the source tree correctly. Here are the commands to use :

# cd /lib/modules/$(uname -r)/build/include
# rm asm
# ln -s asm-ppc64 asm
# cd linux
# cp /boot/vmlinuz-2.4.xxx.autoconf.h ./autoconf.h (replace xxx by the current version number)
# cp /boot/vmlinuz-2.4.xxx.version.h ./version.h (replace xxx by the current version number)

If you are building as a regular userid (and you should), you may have to create a ~/.rpmmacros file. This file contains the top directory of your rpm building space. You can use something like :

$ cat ~/.rpmmacros
# Path to top of build area (without the '{' '}' )
'%_topdir {your_build_root}/build'

You must create this top building directory before using rpm.

$ cd ~/
$ mkdir build
$ cd build
$ mkdir RPMS SRPMS SOURCES BUILD SPECS

Grab the source RPM

As a regular user, download the source RPM here and install it. This will install files under the build/SOURCES and build/SPECS directories.

Build the RPM

The build process is started with :

$ rpm -ba ~/build/SPECS/gm-2.0.6-ppc64.spec > gm-build.log 2>&1

If all goes well, you should have a gm-2.0.6-1.ppc64.rpm file under the ~/build/RPMS/ppc64 directory.

Install the RPM

As root, you can now proceed to the installation of the RPM package

# rpm -i ~fthomas/build/RPMS/ppc64/gm-2.0.6-1.ppc64.rpm
....
# gm_install_drivers
# /etc/init.d/gm start

SSH setup

To run MPI applications across nodes, you need to setup either ssh or rsh so that you are not asked a password when running a remote command. The current default setup for SuSE SLES8 is to disable rsh so we will focus on ssh first.

Suppose you are working on a SLES8 cluster with a NFS mounted home directory. On a node that will serve as a home node, you need to do :

$ ssh-keygen -t rsa -N '' (two single quotes)

$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

The very first time you ssh to a remote node, you are asked to confirm that you wish to connect to this node. The message will look like :

The authenticity of host 'lp02 (9.100.202.69)' can't be established.

RSA key fingerprint is 51:e6:7e:6e:d6:57:a0:8a:7a:26:5d:64:00:b6:f9:67.

Are you sure you want to continue connecting (yes/no)?

To avoid this happening when running a MPI application, we recommand that you ssh once to all the nodes that you may wish to use for parallel applications. This may involve something like :

$ for i in node1 node2 node3 node4

ssh $i date

done

Minimum RPMs to install to compile and run MPI applications

If you have a Myrinet interconnect, you need to install gm, mpich-gm, mpich and mpi-wrappers. Optionally, you may wish to try the LAM implementation. The mpi-wrappers install will require ignoring the dependencies with the --nodeps flag to rpm. If you only have a TCP/IP interconnect, then gm and mpich-gm are not necessary.

Using the mpi-wrappers scripts

There are many variants for compiling and running MPI applications under Linux. The main variables are the bitness (32|64), the protocol used (gm|tcp|shmem), the remote execution command and the compilers. Choosing the right invocation is usually a matter of setting up the correct PATH. But changing the PATH every time we need to compile or run a MPI application is not practical and we thought of writing a few wrapper scripts that would hide this complexity.

These scripts belong to the mpi-wrappers rpm and install in /usr/bin so that they are picked up first in the user's PATH. (You should check that /usr/bin is at the beginning of your PATH.)


variable name	possible choices	notes
MP_IMPL	anl\|lam	Choice for the implementation to use. The possible choices are anl (Argonne National Lab) aka MPICH and LAM (Local Area Multicomputer). They are the 2 most popular MPI implementations and are very similar in function.
OBJECT_MODE	32\|64	Choice for the bitness.
MP_EUILIB	gm\|tcp\|shmem	Choice for the underlying communication protocol. gm is for the use of Myrinet interconnect and provides a priori the best performance. Other choices are tcp for TCP/IP networks and shmem for use of the shared memory single node implementation.
MP_SHLIB	yes\|no	Wheter or not to link the applications with shared libraries.
MP_CC	xlc\|gcc	Choice for the C compiler to use. Possible choices are xlc for using the IBM Visualage compiler or gcc for using the GNU compiler.
MP_CXX	xlC\|g++	Choice for the C++ compiler to use. Possible choices are xlC for using the IBM Visualage compiler or g++ for using the GNU compiler.
MP_FC	xlf\|g77	Choice for the Fortran compiler to use. Possible choices are xlf for using the IBM XL Fortran compiler or g77 for using the GNU compiler.
MP_F90	xlf90\|g90	Choice for the Fortran90 compiler to use. Possible choices are xlf90 for using the IBM XL Fortran compiler or g90 for using the GNU compiler. The latter does not exist yet... and this may last for a while...
MP_RSH	ssh\|rsh	Choice for the remote execution command. This is relevant only when you choose a multi node protocol like gm or tcp. For shmem, no remote execution command is invoked and this choice is therefore harmless.

Now, let's see which are the wrapper scripts and the defaults for each of these variables.


wrapper	function	default settings
mpicc	compile and link C applications	MP_IMPL=anl,OBJECT_MODE=32,MP_EUILIB=gm,MP_CC=xlc,MP_RSH=ssh MP_SHLIB=yes (valid for MP_IMPL=anl only)
mpiCC	compile and link C++ applications	MP_IMPL=anl,OBJECT_MODE=32,MP_EUILIB=gm,MP_CXX=xlC,MP_RSH=ssh MP_SHLIB=yes (valid for MP_IMPL=anl only)
mpif77	compile and link F77 applications	MP_IMPL=anl,OBJECT_MODE=32,MP_EUILIB=gm,MP_FC=xlf,MP_RSH=ssh MP_SHLIB=yes (valid for MP_IMPL=anl only)
mpif90	compile and link F90 applications	MP_IMPL=anl,OBJECT_MODE=32,MP_EUILIB=gm,MP_F90=xlf90,MP_RSH=ssh MP_SHLIB=yes (valid for MP_IMPL=anl only)
mpirun	Run MPI applications	MP_IMPL=anl,OBJECT_MODE=32,MP_EUILIB=gm,MP_RSH=ssh