How to Set Up for Post-Mortem Debug Analysis

of IIS, ASP.net, and SharePoint Process Memory Dumps

by Chris Haun

Revised September 2011

Overview

This page is intended to give a start on analysis memory dumps of user-mode processes. This page will particularly attempt to help ASP.net developers in particular to see what their code is doing in post-mortem crash dumps and hang dumps of IIS processes (made by tools like debugdiag or adplus.vbs) using Windbg.exe and the PSSCOR extension.

Install DebugDiag 1.2

Install the latest version of DebugDiag on your workstation that has outbound internet access from http://microsoft.com/downloads. (Just search on the keyword of “debugdiag.”)

(Or version 1.2 can be downloaded from here: http://www.microsoft.com/download/en/details.aspx?id=26798 )

Debugdiag has built-in analysis scripts that use the Microsoft public symbol server to interpret the Microsoft modules loaded in the dump file.

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image001.jpg$

Collect and Analyze a Sample Dump

Create a hang dump of a w3wp.exe process on a test server, click Add Data Files, and guide the debugdiag tool to the dmp file. Click “Start Analysis” and let the tool chug away. The tool will find the symbol files it needs for the dlls in this process and will cache the symbol files at c:\symcache.

If you have some custom web applications with dlls that were compiled with symbolic output, you might consider adding the symbol files (.pdb) into the c:\symcache folder. (Or create your own private symbol server.)

Install the Debugging Tools for Windows

Browse to your preferred search engine and search for: Debugging Tools for Windows. This should bring you to an entry point for the various versions of these tools. You *might* be able to find them just by browsing this link: http://msdn.microsoft.com/en-us/windows/hardware/gg463009.

If it’s a little confusing, perhaps just look for something that looks like this:

Debugging Tools: More Information and Previous Versions

· Debugging Tools for Windows 32-bit Versions

· Debugging Tools for Windows 64-bit Versions

You might want to install one or both. Keep in mind that if you want to debug dump files made on servers running 64-bit operating systems and running 64-bit processes, you’ll definitely want to have the matching 64 bit debugging tools installed.

Launch WinDBG

Launch Windbg through the list of programs. . .

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image002.jpg$

SYMBOL PATH

Set the Symbol File path. To do this, launch windbg.exe, expand the File menu, and select Symbol File Path.

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image003.jpg$

Assuming your workstation has outbound internet access, and assuming you’ve installed DebugDiag, you’ll want to set the symbol path to:

Srv*c:\symcache*http://msdl.microsoft.com/download/symbols

When you analyze a .dmp file using Windbg.exe, this symbol path will tell windbg to look first for symbols in the local symbol cache (c:\symcache) and then look on the public symbol servers for symbols. Also, when it finds symbols on the public server, it will cache those symbols at c:\symcache.

If you’ve produced symbols (*.pdb) for your web application or have .pdb’s for third-party modules your application uses, you might just copy the .pdb files to c:\symcache. If the files are on a share in your network, you can also add something like this to your symbol path:

Srv*c:\symcache*\\sharename\foldername*http://msdl.microsoft.com/download/symbols

Add a checkmark beside RELOAD and click OK.

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image004.jpg$

LOADING THE DMP

Assuming you have a .dmp file to examine from a hang dump or crash dump, you can open it by expanding the File menu and selecting “Open Crash Dump.”

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image005.jpg$

LOADING PSSCOR EXTENSIONS

One thing that helps immensely with squeezing out asp.net related information out of an IIS process dump are the PSSCOR extension(s). Tom Christian was an escalation engineer on the ASP.net team for some years before moving to an Azure team. He wrote SOS.dll and the PSSCOR dlls. Most of the documentation around talks about sos.dll rather than psscor.dll. Anything you might read about sos.dll can be done with psscor. Think of psscor as sos.dll’s bigger brother.

One of the easiest ways to get psscor2.dll and psscor4.dll is to install Debug Diagnostics 1.2 and look in the %program files%\debugdiag\exts folder.

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image006.jpg$

Or you can download them from these links:

PSSCOR.dll (used for asp.net 1.0 and 1.1 in x86 process dumps)(this should be installed by default with the Debugging Tools for Windows)

PSSCOR2.dll (used for asp.net 2.0, 3.0, and 3.5 in x86 or x64 dumps)

PSSCOR4.dll (used for asp.net 4.0 in x64 dumps)

When you have windbg.exe open, you’ll want to load the psscor extension by typing .load {path to the dll}\psscor

For example, to load psscor2.dll, enter this into the command field of windbg:

.load c:\program files\debugdiag\exts\psscor2

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image007.jpg$

If you can’t download PSSCOR, you may be able to work with SOS.dll from these locations:

· C:\Program Files (x86)\Debugging Tools for Windows (x86)\clr10\sos.dll

· C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll

· C:\Windows\Microsoft.NET\Framework64\v2.0.50727\sos.dll

· C:\Windows\Microsoft.NET\Framework\v1.1.4322\sos.dll

If you wanted to debug an asp.net 4.0 dump and didn’t have psscor4.dll at your disposal, you could, for example load sos.dll from the 4.0 framework folder by typing in:

.load C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos

By the way, if at any time if you ever see a “Save information for workspace?” type message, select YES.

$Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\Windbg\WindbgSetup_files\image008.jpg$