LANL-Trace is a tracing mechanism that integrates with MPI and uses either strace or ltrace to capture the I/O of a parallel app
Version: 1.0.0LANL-Trace is a tracing mechanism that integrates with MPI and uses either strace or ltrace to capture the I/O of a parallel application. In addition, LANL-Trace runs a simple MPI program both before and after the traced program in order to provide information about the possible skew and drift of the distributed clocks in order to allow researchers to meaningfully interpret the timestamps in each of the trace files of which one is produced by each process in the parallel application.
License: BSD License
Operating System: Linux
Homepage: institutes.lanl.gov
Developed by:
INSTALLATION AND RUNNING
The code should basically almost work without a large amount of modification.
Everything is driven by scripts/mpirun. You can use modules/mpitrace to put scripts/mpirun in your path. Then once scripts/mpirun is in your path, you just exec your normal mpirun [whatever] command and trace output should be produced.
However, the trace code attempts to build an executable in order to measure time skew and time drift before and after running your mpi program. Sometimes this build fails. If it does, try to see the command that failed and execute it by hand. Then when you try mpirun again, it will just use the executable you just built and won't try again to rebuild.
Also, you will need to changes paths in scripts/mpirun and modules/mpitrace.
There are various other files in this directory. Just ignore them. They are left-over aborted attempts. This above mechanism should work with Open-MPI.
For mpich, it's basically the same except you'll want to use the standard mpich dbg=foo method and copy the mpirun_dbg.ltrace file to your mpich bin.
OUTPUT AND ANALYSIS
When the trace mechanism runs, it produces a fair number of files. There is a timing file which attempts to capture the drift and skew of the distributed clocks as described above. There is a dirinfo file which measures the free space of the storage system both before and after the trace as well as attempting to query various other aspects of the storage system. There is a SUMMARY file which just shows the command run along with its arguments.
Then for each and every process there are three files produced.
machinename.pid.trace contains the raw trace data,
machinename.pid.out contains the standard output of that process,
machinename.pid.summary contains some simple summary and profiling info. Much of this info is available in the raw trace data but some of it is not. This summary file also contains summary data for system calls that weren't
captured. The idea is that this tool is for I/O only so we attempt to construct a filtering regex to only capture I/O calls. The summary files list
system calls that were captured by the underlying strace mechanism but which weren't logged by LANL-Trace. In the event that you are interested in any of these, you can edit the filter regex accordingly.