A Basic Method of Troubleshooting Crashes in IIS Processes with DebugDiag 1.2

Applies to:

IIS 6.0; IIS 7.0; IIS 7.5

Summary:

If an IIS process becomes unresponsive for a time and you see any of the following types of events in the event logs. . .

Event Source: W3SVC

Event ID: 1009

Description: A process serving application pool 'xyz' terminated unexpectedly. The process id was '1234'. The process exit code was '0xc0000005'.

Event Source: W3SVC

Event ID: 1011

Description: A process serving application pool 'xyz' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1234'.

Event Source: Application Popup

Event ID: 26

Description: Application popup: inetinfo.exe - Application Error : The instruction at "0x01b2dc59" referenced memory at "0x00000008". The memory could not be "read". Click on OK to terminate the program

Source: Microsoft-Windows-WAS

Event ID: 5011

Description: A process serving application pool 'XYZ' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2780'. The data field contains the error number.

Source: .NET Runtime

Event ID: 1027

Description: Application: w3wp.exe Framework Version: v4.0.30319

Description: The process was terminated due to stack overflow.

Event ID: 1000

Description: Faulting application w3wp.exe, version …, time stamp …., faulting module xyz.dll, version …., time stamp …., exception code ….., fault offset ….., process id …., application start time …..

Source: Microsoft-Windows-WAS

Event ID: 5009

Description: A process serving application pool 'XYZ' terminated unexpectedly. The process id was '1234'. The process exit code was 'l'.

Event Source: W3SVC

Event ID: 1011

Description: A process serving application pool 'XYZ' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '1234'. The data field contains the error number.

Event ID: 1000

Description: Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919ed8, faulting module kernel32.dll, version 6.0.6001.18000, time stamp 0x4791ada5, exception code 0xe0434f4d, fault offset 0x000000000002649d, process id 0x1e90, application start time 0x01ca00762fbfb6f3.

… you should install and configure DebugDiag 1.2 to monitor the IIS processes (inetinfo.exe, all w3wp.exe’s, all dllhost.exe’s) for a crash (or access violation or second-chance exception) and dump everything in the crashing process’ memory space at the time of the crash into a .dmp file. We can then perform post-mortem analysis on the .dmp files to determine the root cause of the crash.

INSTALL DEBUGDIAG

Install Debugdiag 1.2 on the server exhibiting crashes. Download it from here:

Link http://www.microsoft.com/download/en/details.aspx?id=26798

Launch Debug Diagnostics from the list of programs. (Be sure to “Run as Administrator” to avoid this possible error: “error while attaching to process via the DbgSvc service. ReloadControlScriptFailed. Could not open handle to control script shared memory mutex.”)

When you launch debugdiag.msi it will give you an option to change the default installation path from the C:\drive to another drive.

Since dump files can be large, it may be advisable to install the tool to another drive with the most free space.

LAUNCH DEBUGDIAG

Launch DebugDiag (Start Button > Programs > Debug Diagnostics Tool 1.2)

SET UP A VERY BASIC CRASH RULE

When asked which rule type to choose, choose “Crash” and click NEXT

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image001.jpg$

For select target type you should go with “All IIS/Com+ related processes.” (However, if you know from the system event log events that it’s one specific iis application pool that is experiencing the crashes, you can instead select “A specific IIS web application pool.”)

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image002.jpg$

Leave everything set to the defaults in the Advanced Configuration for now.

(If this basic crash rule doesn’t catch the crash, we can make changes here and make the crash rule more sensitive later.)

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image003.jpg$

Continue accepting the default settings and clicking next…

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image004.jpg$

If you need the dmp files to be written to some place other than the default, select browse and give it another folder.

Click Next.

Activate the rule when you’re ready for the tool to start monitoring the IIS processes for a crash. . .

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image005.jpg$

Note how the status is set to active and the userdump count is set to 0. The userdump count should increase when a crash is detected.

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image006.jpg$

WAIT

Feel free to log off the server while waiting for the crash to occur. Since debugdiag runs as a service, you do not have to be logged into the machine.

This tool will monitor the iis processes, watching and waiting for a crash to occur. When the process begins to crash, the debugger will interrupt the process temporarily, freeze that process, write out everything in that process to a .dmp file, and then allow the process to crash.

Preliminary Analysis of the Crash Dump

After the dump file has been created, you can switch to the Advanced Analysis tab, highlight Crash/Hang Analyzers,

Click Add Data Files

Find and select the .dmp file

And click the START ANALYSIS button.

[Note: you may prefer to do this on a workstation that has access to the .dmp file and access to the internet rather than do this on the web server. It can be fairly cpu intensive.]

$Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\DebugDiag12\IIS-6-or-7-Crash-Basic_files\image007.jpg$

When the analysis script completes, Internet Explorer will launch and display the results.

Zipping the Dump

If you’d like to zip the dump file up to upload to a support team at Microsoft, here is a good way to do it.

Expand the debugdiag Tools menu, select Advanced Data Collection, select Create Full Cabinet file. This will collect and compress the event logs, the .net config files, the deadlock dump files, and more into one convenient .cab file.

$Description: Description: Description: Description: Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\TshootingASPNETDeadlockDetected2262inIIS7_files\image017.jpg$

Create a support case with the Microsoft support and upload the .cab file to the workspace your support engineer provides.

You can locate the .cab file by clicking the icon of the manila file folder.

$Description: Description: Description: Description: Description: Description: G:\CHAUN\vIISual.net\vIISual.net-PUBLISHED\Debug\TshootingASPNETDeadlockDetected2262inIIS7_files\image018.jpg$

If this method does not trigger dump files when the crash occurs, please consider following the advanced steps found here.

Additional search keywords and references:

931370 The Debug Diagnostic 1.1 tool is now available

http://support.microsoft.com/default.aspx?scid=kb;EN-US;931370

919789 How to use the Debug Diagnostics tool to troubleshoot an IIS process that stops unexpectedly

http://support.microsoft.com/default.aspx?scid=kb;EN-US;919789

921464 How to use the Debug Diagnostics 1.0 tool to analyze a memory dump file

http://support.microsoft.com/default.aspx?scid=kb;EN-US;921464