MPI and Password Cracking
Author: Jason R. Davis
Site: TMTO[dot]ORG
Date: May 28th, 2009
Table of Contents:
Introduction
Directory Structure Methodology
Hardware and Software
Installing OpenMPI
Installing John the Ripper MPI
Utilizing CPU Cores on a Standalone PC
Outro
Introduction
After sauntering through the web for many hours searching for GOOD documentation on this
topic, I realized that it simply does not exist. I’ve been using MPI, both OpenMPI and
MPICH2, for a little over two years and have become familiar with implementing it
effectively in production environments. I also have experience with setting up scalable
password cracking utilities that operate through MPI. With that said, I couldn’t sit on
this knowledge anymore, and went through the laborious task of documenting it for public
release. I will be updating this documentation in the future to include instructions on
how to scale this to a full cluster. For now, it’ll let you utilize all the cores on a
single PC for cracking, instead of just one core.
The goals:
Setup a cluster like directory structure that is scalable
Install OpenMPI
Install John the Ripper MPI
Run a simple test and crack a MD5 hash
Directory Structure Methodology
This is nearly the most important part of building a system that is scalable and won’t
break if you upgrade to a newer version of an application. I am going to quickly outline
how I build a directory structure for a single machine. That structure is then scalable
to a cluster.
/apps/
/apps/myapp1/
/apps/myapp1/v1.x/
/apps/myapp1/v1.x/install/
/apps/myapp1/v1.x/x86_32/
/apps/myapp1/v1.x/x86_64/
/apps/myapp1/v2.x/
/apps/myapp1/v2.x/install/
/apps/myapp1/v2.x/x86_32/
/apps/myapp1/v2.x/x86_64/
/apps/myapp1/latest ? v2.x/
/apps/myapp1/stable ? v1.x/
/apps/myapp1/env/
/apps/
This is the base directory for all our cluster applications.
/apps/myapp1/
For each application we create a subdirectory of our base directory, in this case "myapp1"
/apps/myapp1/v1.x/
The stable version of the application will go below this directory structure. This is
where the production application will reside. You will use the actual version number of
the application for the directory name. You will not use "v1.x".
/apps/myapp1/v1.x/install/
The source code for this specific version will be downloaded, unpacked, and compiled from
this directory.
/apps/myapp1/v1.x/x86_32/
If the application was compiled on a 32-bit processor, then this is where the application
will be installed and ran from.
/apps/myapp1/v1.x/x86_64/
If the application was compiled on a 64-bit processor, then this is where the application
will be installed and ran from.
/apps/myapp1/v2.x/
The latest version of the application will go below this directory structure. This is where
the alpha, beta, or unstable release will reside. This includes applications that need to
be tested and are not ready for production. You will use the actual version number of the
application for the directory name. You will not use "v2.x".
/apps/myapp1/v2.x/install/
The source code for this specific version will be downloaded, unpacked, and compiled from
this directory.
/apps/myapp1/v2.x/x86_32/
If the application was compiled on a 32-bit processor, then this is where the application
will be installed and ran from.
/apps/myapp1/v2.x/x86_64/
If the application was compiled on a 64-bit processor, then this is where the application
will be installed and ran from.
/apps/myapp1/latest ? v2.x
This is a symbolic link "latest" pointing to the directory named after the unstable
installed version.
/apps/myapp1/stable ? v1.x
This is a symbolic link "stable" pointing to the directory named after the stable
installed version
These symbolic links will allow you to upgrade software and not need to rebuild the
environment to accommodate the change. Simply modify the symbolic link to point to the
newer version, still under the name of latest. If I have "/apps/myapp1/v1.x/bin/" in
the $PATH variable for all users .bashrc, then I would have to update all the user
.bashrc files if I wanted to upgrade everyone to "/apps/myapp1/v2.x/bin/". By using
the "stable" and "latest" symbolic links, you only need to put "/apps/myapp1/stable/bin/"
in each users .bashrc file. And when you want to migrate all users to "/apps/myapp1/v2.x/bin/"
simply modify the "stable" symbolic link to point to it. All the users will automatically
start using the newest version on a simple change to the symbolic links. This is very
important for scalability and allows you to test new releases without affecting the other
users by putting "/apps/myapp1/latest/bin/" into your $PATH.
/apps/myapp1/env/
This is where scripts and files that modify the user environment will be placed
Hardware and Software
For this process I’ve chosen Gentoo as the GNU/Linux distribution. I’ve installed it
onto a server containing a single "Intel(R) Xeon(R) E5405 @ 2.00GHz" which happens
to be a Quad Core processor. Which is why I use "-np 4" in the mpirun commands. Also
note, I am using a 64-bit OS and software. Pay attention to your version and make the
minor changes to the commands to accommodate that.
Installing OpenMPI
OpenMPI v1.3.2 (Latest) / OpenMPI v1.3.1 (Stable)
~ # mkdir /apps
~ # mkdir /apps/openmpi
Install v1.3.2:
~ # mkdir /apps/openmpi/v1.3.2
~ # ln -s /apps/openmpi/v1.3.2 /apps/openmpi/latest
~ # mkdir /apps/openmpi/v1.3.2/install
~ # mkdir /apps/openmpi/v1.3.2/x86_32
~ # mkdir /apps/openmpi/v1.3.2/x86_64
~ # cd /apps/openmpi/v1.3.2/install
~ # wget http://www.open-mpi.org/software/ompi/v1.3/downloads/openmpi-1.3.2.tar.gz
~ # tar -xf openmpi-1.3.2.tar.gz
~ # cd openmpi-1.3.2
For 32-bit systems:
~ # ./configure --prefix= /apps/openmpi/v1.3.2/x86_32
For 64-bit systems:
~ # ./configure --prefix=/apps/openmpi/v1.3.2/x86_64
~ # make
~ # make install
Install v1.3.1:
~ # mkdir /apps/openmpi/v1.3.1
~ # ln -s /apps/openmpi/v1.3.1 /apps/openmpi/stable
~ # mkdir /apps/openmpi/v1.3.1/install
~ # mkdir /apps/openmpi/v1.3.1/x86_32
~ # mkdir /apps/openmpi/v1.3.1/x86_64
~ # cd /apps/openmpi/v1.3.1/install
~ # wget http://www.open-mpi.org/software/ompi/v1.3/downloads/openmpi-1.3.1.tar.gz
~ # tar -xf openmpi-1.3.1.tar.gz
~ # cd openmpi-1.3.1
For 32-bit systems:
~ # ./configure --prefix= /apps/openmpi/v1.3.1/x86_32
For 64-bit systems:
~ # ./configure --prefix=/apps/openmpi/v1.3.1/x86_64
~ # make
~ # make install
~ # mkdir /apps/openmpi/env
~ # cd /apps/openmpi/env
~ # echo ‘export PATH=${PATH}:/apps/openmpi/latest/x86_64/bin’ > prepenv_latest
~ # echo ‘export PATH=${PATH}:/apps/openmpi/stable/x86_64/bin’ > prepenv_stable
Edit /etc/env.d/00basic and modify the LDPATH to include the OpenMPI libraries:
LDPATH="/usr/local/lib:/apps/openmpi/latest/x86_64/lib:/apps/openmpi/stable/x86_64/lib"
~ # env-update && source /etc/profile
Running a OpenMPI Sanity Check:
~ # source /apps/openmpi/env/prepenv_latest
~ # mpirun -np 4 hostname
The last command above will print the computers hostname four times. Not less, not more,
four times. If it does, then your "latest" OpenMPI version is working.
~ # env-update && source /etc/profile
~ # source /apps/openmpi/env/prepenv_stable
~ # mpirun -np 4 hostname
The last command above will print the computers hostname four times. Not less, not more,
four times. If it does, then your "stable" OpenMPI version is working.
At this point you should have a functioning OpenMPI install. Two versions, latest and
stable, are available based upon which file you source. The reason I force users to
login and "source" the "prepenv_xxxxxx" file is because I don’t want to add bloat to
user profiles. I despise $PATH variables that are 30+ directories long, etc. Less is
more in this case. I don’t add any information to users .bashrc in this case. Some
users actually put the "source /apps/openmpi/env/prepenv_stable" command in their
.bashrc, which makes OpenMPI available at the time they login by setting the $PATH.
Installing John the Ripper MPI
Note: You cannot install John the Ripper MPI unless you have a functioning MPI environment.
So do not continue unless you have successfully installed OpenMPI and completed the sanity
check.
John the Ripper MPI v1.7.3.1 (Latest)
~ # mkdir /apps/jtr
Install v1.7.3.1:
~ # mkdir /apps/jtr/v1.7.3.1
~ # ln -s /apps/jtr/v1.7.3.1 /apps/jtr/latest
~ # mkdir /apps/jtr/v1.7.3.1/install
~ # mkdir /apps/jtr/v1.7.3.1/x86_32
~ # mkdir /apps/jtr/v1.7.3.1/x86_64
~ # cd /apps/jtr/v1.7.3.1/install
~ # wget http://www.bindshell.net/tools/johntheripper/john-1.7.3.1-all-2-mpi8.tar.gz
~ # tar -xf john-1.7.3.1-all-2-mpi8.tar.gz
~ # cd john-1.7.3.1-all-2-mpi8/src
For 32-bit systems:
~ # make linux-x86-sse2
For 64-bit systems:
~ # make linux-x86-64
~ # cd /apps/jtr/v1.7.3.1/install/john-1.7.3.1-all-2-mpi8/run
~ # cp * /apps/jtr/v1.7.3.1/x86_64/
~ # mkdir /apps/jtr/env
~ # cd /apps/jtr/env
~ # echo ‘export PATH=${PATH}:/apps/jtr/latest/x86_64’ > prepenv_latest
~ # echo ‘source /apps/openmpi/env/prepenv_latest’ >> prepenv_latest
~ # echo ‘cd /apps/jtr/latest/x86_64’ >> prepenv_latest
Running a John the Ripper Sanity Check:
~ # env-update && source /etc/profile
~ # source /apps/jtr/env/prepenv_latest
~ # mpirun -np 4 ./john -test
The last command above will self test John the Ripper MPI and run a benchmark to see how
fast your computer can run it.
Utilizing CPU Cores on a Standalone PC
At this point you should have a functioning MPI environment and a functioning version of
John the Ripper. So now that the foundation is laid, let us move on to the fun part:
Cracking Passwords!
The first task we need to complete is to create a cleanup script for John the Ripper,
because it creates a lot of files that don’t get cleaned up after its runs. Only run
this if you no longer need the john_log and john.pot files, because it removes them
as well.
~ # cd /apps/jtr/env
~ # echo "#!/bin/bash" > cleanup_latest.sh
~ # echo "rm /apps/jtr/latest/x86_64/john_rec.rec*" >> cleanup_latest.sh
~ # echo "rm /apps/jtr/latest/x86_64/john_log" >> cleanup_latest.sh
~ # echo "rm /apps/jtr/latest/x86_64/john.pot" >> cleanup_latest.sh
~ # chmod + x cleanup_latest.sh
The entire platform is setup now. Next we will begin a test run, with a simple MD5 hash.
~ # env-update && source /etc/profile
~ # source /apps/jtr/env/prepenv_latest
~ # echo "root:e80b5017098950fc58aad83c8c14978e" >> md5s.txt
~ # mpirun -np 4 ./john -format=raw-md5 -incremental:alpha ./md5s.txt
Loaded 1 password hash (Raw MD5 [raw-md5])
Loaded 1 password hash (Raw MD5 [raw-md5])
Loaded 1 password hash (Raw MD5 [raw-md5])
Loaded 1 password hash (Raw MD5 [raw-md5])
abcdef (root)
Process 0 completed loop.
thread: 0 guesses: 1 time: 0:00:00:00 c/s: 1582K trying: abaflm - abcdch
The last command above initiated John the Ripper MPI utilizing all 4 cores and
successfully broke the password in under a second. The password was "abcdef".
If you have questions about the syntax for running John the Ripper, please read the
related docs.
After each run, I usually copy the "john.pot" file to safe spot, and then cleanup:
~ # /apps/jtr/env/cleanup_latest.sh
Outro
As mentioned in at the beginning of this document, I will be amending updates to show users
how to expand to a cluster, spanning many PCs and many CPU cores. If you have trouble,
utilize Google as much as possible; every error I came across had an answer and a quick
Google search brought it right to me.
Jason R. Davis
TMTO[dot]ORG
# milw0rm.com [2009-06-08]
