How to crack password using Nvidia GPU

While this is not my primary area of expertise, I have been using John the Ripper more frequently lately, so I began to wonder how to take advantage of the powerful Nvidia GPU...

Prerequisites

I have used the following software during this brief tutorial.

  • Ubuntu 15.04 Vivid Vervet
  • John the Ripper 1.8.0-jumbo-1 source code

I have tested it on Lenovo Z50-70 notebook equipped with Intel i7-4510U @ 2.00GHz and Nvidia GeForce 840M.

You can use live USB image with persistent storage to create system specialized for this purpose.

Installation

Enable multiverse repository as it will contain packages required to use OpenCL and CUDA.

$ sudo apt-add-repository multiverse

Update the package index files.

$ sudo apt-get update

Install Nvidia driver and CUDA toolkit.

$ sudo apt-get install nvidia-340 nvidia-cuda-toolkit
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  bbswitch-dkms ca-certificates-java default-jre default-jre-headless dkms
  fonts-dejavu-extra java-common lib32gcc1 libatk-wrapper-java
  libatk-wrapper-java-jni libbonobo2-0 libbonobo2-common libc6-i386
  libcublas6.5 libcuda1-340 libcudart6.5 libcufft6.5 libcufftw6.5
  libcuinj64-6.5 libcurand6.5 libcusparse6.5 libdrm-dev libgconf2-4 libgif4
  libgl1-mesa-dev libgl1-mesa-glx libglapi-mesa libgnome2-0 libgnome2-bin
  libgnome2-common libgnomevfs2-0 libgnomevfs2-common libjansson4 libnppc6.5
  libnppi6.5 libnpps6.5 libnvblas6.5 libnvtoolsext1 libnvvm2 liborbit-2-0
  libpthread-stubs0-dev libthrust-dev libvdpau-dev libvdpau1 libx11-dev
  libx11-doc libx11-xcb-dev libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev
  libxcb-glx0-dev libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev
  libxcb-shape0-dev libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev
  libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev libxnvctrl0
  libxshmfence-dev libxxf86vm-dev mesa-common-dev nvidia-cuda-dev
  nvidia-cuda-doc nvidia-cuda-gdb nvidia-opencl-dev nvidia-opencl-icd-340
  nvidia-prime nvidia-profiler nvidia-settings nvidia-visual-profiler
  ocl-icd-libopencl1 opencl-headers openjdk-7-jre openjdk-7-jre-headless
  screen-resolution-extra tzdata tzdata-java x11proto-core-dev
  x11proto-damage-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-gl-dev
  x11proto-input-dev x11proto-kb-dev x11proto-xext-dev
  x11proto-xf86vidmode-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
  bumblebee equivs libbonobo2-bin desktop-base libgnomevfs2-bin
  libgnomevfs2-extra gamin fam gnome-mime-data libvdpau-doc
  nvidia-vdpau-driver vdpau-driver libxcb-doc libxext-doc libcupti-dev
  icedtea-7-plugin icedtea-7-jre-jamvm sun-java6-fonts fonts-ipafont-gothic
  fonts-ipafont-mincho ttf-wqy-microhei ttf-wqy-zenhei fonts-indic
Recommended packages:
  libcuda1 libcuda-6.5-1 libnvcuvid1 nvidia-opencl-icd
The following NEW packages will be installed:
  bbswitch-dkms ca-certificates-java default-jre default-jre-headless dkms
  fonts-dejavu-extra java-common lib32gcc1 libatk-wrapper-java
  libatk-wrapper-java-jni libbonobo2-0 libbonobo2-common libc6-i386
  libcublas6.5 libcuda1-340 libcudart6.5 libcufft6.5 libcufftw6.5
  libcuinj64-6.5 libcurand6.5 libcusparse6.5 libdrm-dev libgconf2-4 libgif4
  libgl1-mesa-dev libgnome2-0 libgnome2-bin libgnome2-common libgnomevfs2-0
  libgnomevfs2-common libjansson4 libnppc6.5 libnppi6.5 libnpps6.5
  libnvblas6.5 libnvtoolsext1 libnvvm2 liborbit-2-0 libpthread-stubs0-dev
  libthrust-dev libvdpau-dev libvdpau1 libx11-dev libx11-doc libx11-xcb-dev
  libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev
  libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev libxcb-shape0-dev
  libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev libxdamage-dev libxdmcp-dev
  libxext-dev libxfixes-dev libxnvctrl0 libxshmfence-dev libxxf86vm-dev
  mesa-common-dev nvidia-340 nvidia-cuda-dev nvidia-cuda-doc nvidia-cuda-gdb
  nvidia-cuda-toolkit nvidia-opencl-dev nvidia-opencl-icd-340 nvidia-prime
  nvidia-profiler nvidia-settings nvidia-visual-profiler ocl-icd-libopencl1
  opencl-headers openjdk-7-jre openjdk-7-jre-headless screen-resolution-extra
  tzdata-java x11proto-core-dev x11proto-damage-dev x11proto-dri2-dev
  x11proto-fixes-dev x11proto-gl-dev x11proto-input-dev x11proto-kb-dev
  x11proto-xext-dev x11proto-xf86vidmode-dev xorg-sgml-doctools xtrans-dev
The following packages will be upgraded:
  libgl1-mesa-glx libglapi-mesa tzdata
3 upgraded, 92 newly installed, 0 to remove and 243 not upgraded.
Need to get 606 MB/606 MB of archives.
After this operation, 1,395 MB of additional disk space will be used.
[...]

Reboot system to use installed driver.

$ sudo reboot

Verify installed driver.

$ nvidia-smi
Sat Sep  5 21:30:30 2015       
+------------------------------------------------------+                       
| NVIDIA-SMI 340.76     Driver Version: 340.76         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 840M        Off  | 0000:03:00.0     N/A |                  N/A |
| N/A   44C    P8    N/A /  N/A |    292MiB /  4095MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
+-----------------------------------------------------------------------------+

Install packages required to build an application from source code.

$ sudo apt-get install build-essential libssl-dev
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version.
The following extra packages will be installed:
  libssl-doc libssl1.0.0 zlib1g-dev
The following NEW packages will be installed:
  libssl-dev libssl-doc zlib1g-dev
The following packages will be upgraded:
  libssl1.0.0
1 upgraded, 3 newly installed, 0 to remove and 242 not upgraded.
Need to get 3,074 kB of archives.
After this operation, 8,417 kB of additional disk space will be used.
[...]

Download community-enhanced version.

$ wget http://www.openwall.com/john/j/john-1.8.0-jumbo-1.tar.gz

Extract downloaded archive.

$ tar xfz john-1.8.0-jumbo-1.tar.gz

Change working directory.

$ cd john-1.8.0-jumbo-1/src/

Execute configuration script.

$ ./configure

Verify OpenCL and CUDA support.

[...]

Configured for building John the Ripper 1.8.0-jumbo-1:

Target CPU .................................. x86_64 AVX2, 64-bit LE
AES-NI support .............................. depends on OpenSSL
Target OS ................................... linux-gnu
Cross compiling ............................. no
Legacy arch header .......................... x86-64.h
OpenMPI support (default disabled) .......... no
Fork support ................................ yes
OpenMP support .............................. yes
OpenCL support .............................. yes
CUDA support ................................ yes
Generic crypt(3) format ..................... yes

Optional libraries found:
Rexgen (extra cracking mode) ................ no
GMP (performance for SRP formats) ........... no
PCAP (vncpcap2john and SIPdump) ............. no
BZ2 (gpg2john extra decompression logic) .... yes

Development options (these may hurt performance when enabled):
Memdbg memory debugging settings ............ disabled
AddressSanitizer ("ASAN") ................... disabled

Install missing libraries to get any needed features that were omitted.

Configure finished.  Now 'make clean && make -s' to compile.

Compile source code.

$ make -s -j 4

Compiled software is available in the run directory.

$ cd ../run

You can execute commands directly from that directory.

Usage

List OpenCL devices.

$ sudo ./john --list=opencl-devices
Platform #0 name: NVIDIA CUDA
Platform version: OpenCL 1.1 CUDA 6.5.45
	Device #0 (0) name:	GeForce 840M
	Device vendor:		NVIDIA Corporation
	Device type:		GPU (LE)
	Device version:		OpenCL 1.1 CUDA
	Driver version:		340.76
	Native vector widths:	char 1, short 1, int 1, long 1
	Preferred vector width:	char 1, short 1, int 1, long 1
	Global Memory:		3.0 GB
	Global Memory Cache:	48.0 KB
	Local Memory:		47.0 KB (Local)
	Max memory alloc. size:	1023.10 MB
	Max clock (MHz):	1124
	Profiling timer res.:	1000 ns
	Max Work Group Size:	1024
	Parallel compute cores:	3
	Stream processors:	384  (3 x 128)
	Warp size:		32
	Max. GPRs/work-group:	65536
	Kernel exec. timeout:	yes
	PCI device topology:	03:00.0
	NVML id:		0
	Temperature:		45°C
	Utilization:		n/a

List OpenCL devices.

$ sudo ./john --list=cuda-devices
1 CUDA device found:

CUDA Device #0
	Name:                          GeForce 840M
	Type:                          discrete
	Compute capability:            5.0 (sm_50)
	Number of stream processors:   384 (3 x 128)
	Clock rate:                    1124 Mhz
	Memory clock rate (peak)       900 Mhz
	Memory bus width               64 bits
	Peak memory bandwidth:         14 GB/s
	Total global memory:           3.0 GB
	Total shared memory per block: 48.0 KB
	Total constant memory:         64.0 KB
	L2 cache size                  1024.0 KB
	Kernel execution timeout:      Yes
	Concurrent copy and execution: One direction
	Concurrent kernels support:    Yes
	Warp size:                     32
	Max. GPRs/thread block         65536
	Max. threads per block         1024
	Max. resident threads per MP   2048
	PCI device topology:           03:00.0
	NVML id:                       0
	Fan speed:                     n/a
	GPU temp:                      45°C
	Utilization:                   n/a
$ sudo ./john --format=sha512crypt-opencl shadow

List OpenCL supported formats.

$ sudo ./john --list=formats --format=opencl
sha1crypt-opencl, oldoffice-opencl, PBKDF2-HMAC-SHA1-opencl, rar-opencl, 
RAR5-opencl, lotus5-opencl, agilekeychain-opencl, bcrypt-opencl, 
blockchain-opencl, md5crypt-opencl, sha256crypt-opencl, sha512crypt-opencl, 
descrypt-opencl, dmg-opencl, encfs-opencl, gpg-opencl, keychain-opencl, 
keyring-opencl, krb5pa-md5-opencl, krb5pa-sha1-opencl, mscash2-opencl, 
mysql-sha1-opencl, ssha-opencl, nt-opencl, ntlmv2-opencl, o5logon-opencl, 
ODF-opencl, ODF-AES-opencl, office2007-opencl, office2010-opencl, 
office2013-opencl, PBKDF2-HMAC-SHA256-opencl, pbkdf2-hmac-sha512-opencl, 
phpass-opencl, pwsafe-opencl, RAKP-opencl, Raw-MD4-opencl, Raw-MD5-opencl, 
Raw-SHA1-opencl, Raw-SHA256-opencl, Raw-SHA512-opencl, 7z-opencl, 
strip-opencl, sxc-opencl, wpapsk-opencl, XSHA512-opencl, zip-opencl

List CUDA supported formats.

$ sudo ./john --list=formats --format=cuda
md5crypt-cuda, sha256crypt-cuda, sha512crypt-cuda, mscash-cuda, mscash2-cuda, 
phpass-cuda, pwsafe-cuda, Raw-SHA512-cuda, wpapsk-cuda, xsha512-cuda, 
Raw-SHA224-cuda, Raw-SHA256-cuda

Decrypt password using OpenMP.

$ sudo ./john ~/shadow
Warning: detected hash type "sha512crypt", but the string is also recognized as "sha512crypt-opencl"
Use the "--format=sha512crypt-opencl" option to force loading these as that type instead
Warning: detected hash type "sha512crypt", but the string is also recognized as "sha512crypt-cuda"
Use the "--format=sha512crypt-cuda" option to force loading these as that type instead
Warning: detected hash type "sha512crypt", but the string is also recognized as "crypt"
Use the "--format=crypt" option to force loading these as that type instead
Loaded 1 password hash (sha512crypt, crypt(3) $6$ [SHA512 64/64 OpenSSL])
Will run 4 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:05:09  3/3 0g/s 742.9p/s 742.9c/s 742.9C/s buro15..bitaka
[...]	

Decrypt password using OpenCL.

$ sudo ./john --format=sha512crypt-opencl ~/shadow
Device 0: GeForce 840M
Local worksize (LWS) 32, global worksize (GWS) 2304
Loaded 1 password hash (sha512crypt-opencl, crypt(3) $6$ [SHA512 OpenCL])
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:08:44  3/3 0g/s 5005p/s 5005c/s 5005C/s GPU:67°C ljm234..devvey
[...]	

Decrypt password using CUDA.

$ sudo ./john --format=sha512crypt-cuda ~/shadow
Loaded 1 password hash (sha512crypt-cuda, crypt(3) $6$ [SHA512 CUDA (inefficient, please use sha512crypt-opencl instead)])
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:05:37 73.47% 2/3 (ETA: 22:43:44) 0g/s 348.0p/s 348.0c/s 348.0C/s GPU:59°C Morris...Macross.
[...]	

SHA512 CUDA is inefficient currently, but do not treat it as a rule.

Additional notes

You can verify installed driver using GUI application.

$ nvidia-settings


Install nvidia-opencl-dev package instead of nvidia-cuda-toolkit if you plan to use only OpenCL. It has minimal dependencies compared to the whole CUDA toolkit.

$ sudo apt-get install nvidia-340 nvidia-opencl-dev
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  bbswitch-dkms dkms lib32gcc1 libc6-i386 libcuda1-340 libdrm-dev
  libgl1-mesa-dev libgl1-mesa-glx libglapi-mesa libjansson4
  libpthread-stubs0-dev libvdpau1 libx11-dev libx11-doc libx11-xcb-dev
  libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev
  libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev libxcb-shape0-dev
  libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev libxdamage-dev libxdmcp-dev
  libxext-dev libxfixes-dev libxnvctrl0 libxshmfence-dev libxxf86vm-dev
  mesa-common-dev nvidia-opencl-icd-340 nvidia-prime nvidia-settings
  ocl-icd-libopencl1 opencl-headers screen-resolution-extra x11proto-core-dev
  x11proto-damage-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-gl-dev
  x11proto-input-dev x11proto-kb-dev x11proto-xext-dev
  x11proto-xf86vidmode-dev xorg-sgml-doctools xtrans-dev
Suggested packages:
  bumblebee nvidia-vdpau-driver vdpau-driver libxcb-doc libxext-doc
Recommended packages:
  nvidia-opencl-icd
The following NEW packages will be installed:
  bbswitch-dkms dkms lib32gcc1 libc6-i386 libcuda1-340 libdrm-dev
  libgl1-mesa-dev libjansson4 libpthread-stubs0-dev libvdpau1 libx11-dev
  libx11-doc libx11-xcb-dev libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev
  libxcb-glx0-dev libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev
  libxcb-shape0-dev libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev
  libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev libxnvctrl0
  libxshmfence-dev libxxf86vm-dev mesa-common-dev nvidia-340 nvidia-opencl-dev
  nvidia-opencl-icd-340 nvidia-prime nvidia-settings ocl-icd-libopencl1
  opencl-headers screen-resolution-extra x11proto-core-dev x11proto-damage-dev
  x11proto-dri2-dev x11proto-fixes-dev x11proto-gl-dev x11proto-input-dev
  x11proto-kb-dev x11proto-xext-dev x11proto-xf86vidmode-dev
  xorg-sgml-doctools xtrans-dev
The following packages will be upgraded:
  libgl1-mesa-glx libglapi-mesa
2 upgraded, 51 newly installed, 0 to remove and 244 not upgraded.
Need to get 77.1 MB/77.1 MB of archives.
After this operation, 365 MB of additional disk space will be used.
[...]

There is slightly more to it than that, but it is a good start.

References

John the Ripper GPU support

Milosz Galazka's Picture

About Milosz Galazka

Milosz is a Linux Foundation Certified Engineer working for a successful Polish company as a system administrator and a long time supporter of Free Software Foundation and Debian operating system. He is also open for new opportunities and challenges.

Gdansk, Poland https://sleeplessbeastie.eu