Troubleshooting the Event 2262 “Deadlock Detected” in IIS7.x

Applies To:

This page is tailored for Asp.net 2.0/3.0/3.5 running on Windows2008/IIS7 or Windows2008R2/IIS7.5  

For Windows2003/IIS6 please redirect to here.

Introduction/Overview:

These steps should be useful if your web server (or SharePoint WFE Server) becomes unresponsive to all users and you note the following error message the server’s Application Event log:

Log Name: Application

Source: Microsoft-Windows-IIS-W3SVC-WP

Event ID: 2262

Level: Warning

Description:

ISAPI 'C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

Doing a kb search for 2262 deadlock or asp.net deadlock detected shows several possible causes for the asp.net deadlock.   There is no one-size-fits-all silver-bullet solution.   Ultimately the best way to find root cause for the deadlock is to create a memory dump of the IIS/ASP.net process at the right time.  This page should help you get that memory dump and show you what to do with that dump.

Updates?

You might cautiously consider whether or not updating .net/asp.net on your server might be a good place to start. You might start by checking the version number of the aspnet_isapi.dll (found at either C:\Windows\Microsoft.NET\Framework64\v2.0.50727 or C:\Windows\Microsoft.NET\Framework\v2.0).  If it is lower than 2.0.50727.4066, for example, you might need to at least consider the hotfix (or the workaround) described in. . .

974165  FIX: Event ID 2262 is logged in the Application event log when an ASP.NET 2.0 Web application becomes unresponsive

http://support.microsoft.com/default.aspx?scid=kb;EN-US;974165

Threadpool Optimizations?

For ASP.net 1.1 (which is probably not relevant to your server) on a multi-processor server, optimizing machine.config threadpool settings would be the first thing I’d start with.   But for ASP.net 2.0 and higher, it may or may not be worth doing.   This involves making changes in your machine.config file (%WINDIR%\Microsoft.NET\Framework|Framework64\v2.0.50727\config\). 

We normally we don’t want to make these changes for Asp.net 2.0, however, because this machine.config file has by default a setting of “<processModel autoConfig="true"/>”.  That setting usually tends to work out pretty well usually.    It may be a good idea to check the C:\WINDOWS\Microsoft.NET\Framework|Framework64\v2.0.50727\config\machine.config file so you can see if autoconfig is still set to true or not.  If it is set to false, make sure that you’re following the guidelines for kb 821268.  If it is set to true, perhaps leave it alone.   If all else fails, you could consider trying to carefully set the threadpool settings manually.  

If you decide to edit the config file, please make a safe copy of your current machine.config file.   Open task manager to see how many processors Windows thinks your server has.  You’ll use the number of processors to help you configure the threadpool optimizations.   This is something that should be done in a test environment first.   A restart of IIS is required to make the changes take effect.  

Reference:

KB 821268        Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications

http://support.microsoft.com/default.aspx?scid=kb;EN-US;821268

A somewhat simplified version of this can be found here:  http://viisual.net/Configuration/ASPNET/ThreadpoolOptimizations.htm

Install and Configure DebugDiag 1.2

Install Debugdiag 1.2 on the server exhibiting deadlocks.

             Link http://www.microsoft.com/download/en/details.aspx?id=26798

Launch Debug Diagnostics from the list of programs.   

(Be sure to “Run as Administrator” to avoid this possible error: “error while attaching to process via the DbgSvc service. ReloadControlScriptFailed. Could not open handle to control script shared memory mutex.”)

Click the CANCEL button on Select Rule Type.

Expand the Tools Menu and select Options and Settings.

I’m assuming here that the server has outbound internet access.   

Please copy the text (srv*c:\symcache*http://msdl.microsoft.com/download/symbols) from the “Symbol search path for Analysis” and paste it into the field for Symbol Search Path for Debugging.    The rule we’re going to create later needs to know where to look for the symbol files.  

Click OK.

Cache the Isapi.pdb locally

There is one symbol in particular that we need to find and place into the c:\symcache folder.   It’s the symbol file (.pdb) for isapi.dll.  

Unfortunately this file may not be accessible from the public symbol path (*http://msdl.microsoft.com/download/symbols).   You’ll need to download the isapi.pdb file and either cache it locally (ideally at c:\symcache) or on a private symbol server that you create (which is beyond the scope of these steps).

Debugdiag will create the C:\symcache folder if you attempt to run one of its analysis scripts against a .dmp file.   The analysis scripts can be seen on the Advanced Analysis tab of debugdiag, as seen below. . .

Since you probably have not already run an analysis script, you probably don’t have a C:\symcache folder.    So now would be a good time to just manually create a folder named SYMCACHE (using Windows Explorer) and put it at the root of the C drive (or put it wherever you have told debugdiag to look for symbols at).

Now where can you get the isapi.pdb file?   You need to download the Windows Symbol Package that corresponds to the platform of your web server and install (unzip) it to c:\symcache.   

http://msdn.microsoft.com/en-us/windows/hardware/gg463028

Download Windows Symbol Packages

For the sake of convenience, please change the package’s installation path to c:\symcache. You should end up with an isapi.pdb file in a folder named isapi.pdb in the C:\symcache folder.

Create the Crash Rule

After the isapi.pdb file is somewhere in your c:\symcache folder, you can create the crash rule.

Click the Add Rule button.

Select CRASH from the rule type.

If you’re unsure which Application Pool to focus on or have reason to believe the problem may be occurring inside of several application pools,

select “All IIS/COM+ related processes.”   (But if you’re sure you know exactly which application pool to focus on [perhaps by matching up events in the System event log with the deadlock event in the Application event log] you might prefer to select “A specific IIS web application pool.”)

Select the Breakpoints button

Select the “Add Breakpoint” button

Carefully type isapi!SSFReportUnhealthy into the Breakpoint Expression field.

Do NOT place a checkmark beside “This is a managed (.NET) breakpoint.”  

Set action type to “Full Userdump.”

Click OK.

In the “Configure Breakpoints” window you should now see one the breakpoint expression, action type, and action limit set to:

isapi!SSFReportUnhealthy | Create Full Userdump | 1

Click Save and Close.

Click Next.

Click Next again.

If you need the dmp files to be written to some place other than the default, select browse and give it another folder.

Click Next

You should see the crash rule with a status of Active and Userdump count of 0. 

We now expect that when the deadlock occurs again, the userdump count will increase to 1.

When the userdump count increases to 1, you can recycle the IIS application pool (or restart IIS) to get some temporary relief.

If the rule fails (“rule failed”) it is probably because either it is not set up properly to find the symbols or the isapi.pdb is not where it is looking.

Preliminary Analysis of the Deadlock Dump

After the dump file has been created, you can switch to the Advanced Analysis tab, highlight Crash/Hang Analyzers,

Click Add Data Files

Find and select the .dmp file

And click the START ANALYSIS button.

[Note: you may prefer to do this on a workstation that has access to the .dmp file and access to the internet rather than do this on the web server. It can be fairly cpu intensive.]

When the analysis script completes, Internet Explorer will launch and display the results.

Zipping the Dump

If you’d like to zip the dump file up to upload to the ASP.net support team at Microsoft, here is a good way to do it.

Expand the debugdiag Tools menu, select Advanced Data Collection, select Create Full Cabinet file.   This will collect and compress the event logs, the .net config files, the deadlock dump files, and more into one convenient .cab file.

Create a support case with the Microsoft ASP.net support team and upload the .cab file to the workspace your asp.net engineer provides.   

You can locate the .cab file by clicking the icon of the manila file folder.

Additional keywords and info:

This is the error message you may see in the debugdiag log files if the isapi.pdb is not available.

[8/22/2011 7:44:16 AM] Attempting to set unmanaged breakpoint at isapi!SSFReportUnhealthy

[8/22/2011 7:44:17 AM] bpID = 0

[8/22/2011 7:44:17 AM] Current Breakpoint List(BL)

0 eu                      0001 (0001) (isapi!SSFReportUnhealthy)

*** Error due to unresolved breakpoint ***

Review the list of breakpoints below to find any unresolved breakpoints for any loaded modules.

Note unresolved breakpoints can be caused by missing/invalid symbols or by invalid breakpoint expressions.

0 eu                      0001 (0001) (isapi!SSFReportUnhealthy)

*** ERROR ***

The corresponding crash rule for this process has been put into the 'Failed' state.

Resolve the problem with the unresolved breakpoint(s) below and then re-activate the rule.

Request to detach recieved. Attempting detach...

Successfully detached from process. The target process is no longer being debugged.

This instance of Dbghost will shutdown in a few seconds.

==============IGNORE EVERYTHING BELOW=============

The following alternatives to “Plan A” are inferior to plan A and may not work at all.   I include them in this web page only to improve searchability.  

Plan B – Hang Dump with DebugDiag Immediately after Deadlock

If for some reason the three breakpoints listed in Plan A do not trigger the creation of the dump file when the deadlock occurs, the next best thing to do, as a Plan B, is to try to get a hang dump of the IIS process immediately after the deadlock occurs.   The hang dump needs to be made while the asp.net pages are still not serving.  We need to catch things in a deadlocked state.   Sometimes this method produces an adequate dump file. 

To make a hang dump, right click on your crash rule, select Dump Target Process(es), and select Create FULL Userdump.   (We do not want mini-dumps.)

If this does not create a dump file that proves to be adequate for finding root cause, it may be prudent to set the application pool (or pools) in focus to orphan their worker processes before the next deadlock and be prepared to try plan B again.   This can be done in the advanced settings of the IIS manager.   Find the “process orphaning” and “enabled” and set it to true.

Plan C - Deadlock Dumps with CDB.exe

If for some reason the debugdiag method doesn’t work, or if you don’t have outbound internet access on the server, you can also try using cdb.exe (part of the Debugging Tools for Windows) to dump out the deadlocked process.

Follow KB 828222 very carefully in conjunction with this page.  The KB is written for IIS 6.0 and this page will help adapt it to IIS 7.0.

828222        How to generate a dump file when ASP.NET deadlocks in IIS 6.0

http://support.microsoft.com/default.aspx?scid=kb;EN-US;828222

Try to match up the event 2262 (“aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'”) in the application event log with the event 1013 in the system event log (“A process serving application pool ‘WHATEVER’ exceeded time limits during shut down”).   If you can confirm that the two occur at the same time then you can know which application pool to focus your trouble shooting on.   

Download and install the Debugging Tools for Windows onto the web server suffering from the deadlock.  Download from one of these two locations. . .

	Debugging Tools for Windows 32-bit Versions
	Debugging Tools for Windows 64-bit Versions

If your server is 64 bit, it’s probably going to be the AMD 64 version that you’ll want to download.

While installing, be sure to do a custom install of the debugging tools so that you can install them to the C:\Debuggers folder rather than the default location.  

If you let it install to the default location, the steps in this kb won’t work.

In kb 828222 there is a section for “Configure the Orphan Worker Process settings.”  

In IIS7.0, you don’t need to use adsutil.vbs.  (It may work but it’s unnecessary, less efficient, more prone to error, and arguably the tool is depreciated.)

Open the IIS manager, select the application pool that is reporting the deadlock, and visit the Advanced Settings.

Set Process Orphaning : Enabled from False to True.  

Set Process Orphaning : Executable to “C:\action.cmd”

Set Process Orphaning : Executable Parameters to “%1%”

and

Carefully save the following into notepad and save as “c:\action.cmd”

@if "%_echo%"=="" echo off

setlocal

set TIMESTAMP=%DATE:~-9%_%TIME%

set TIMESTAMP=%TIMESTAMP:/=_%

set TIMESTAMP=%TIMESTAMP::=_%

set TIMESTAMP=%TIMESTAMP:.=_%

set TIMESTAMP=%TIMESTAMP: =_%

set FILENAME=c:\crash_PID_%1_%TIMESTAMP%.dmp

set LOG=c:\log.txt

set COMMAND=c:\debuggers\cdb.exe -c ".dump /o /ma %FILENAME%;q" -p %1

echo %COMMAND% > %LOG%

%COMMAND%

Endlocal

If you follow this kb (with my clarifications) this should produce a .dmp file of the “Whatever” AppPool’s w3wp.exe process and place it in a folder under c:\debuggers.

When you have that dmp file, please zip the dumpfile and upload it to a workspace so your Asp.net support engineer can perform post-mortem analysis.  

With this information we should be able to determine root cause for the deadlock and make recommendations for remediation.

Be sure to undo the changes you made to your application pool(s).

Plan D – Hang Dump

If Plan C does not work, plan D would be to run this command from a command prompt to produce a hang dump as soon as possible when the deadlock occurs next.  You’d want to try to get the hang dump immediately after the deadlock is detected and while the iis process is still deadlocked (before IIS health checking recycles it).

C:\debuggers\>  cscript.exe adplus.vbs –hang –iis –quiet

Note:  if the –iis switch doesn’t work, try –p with the process ID number (PID).

Note:  alternatively you can install DebugDiag 1.2 and use it to make a hang dump instead of using adplus.vbs.

2498330              Collecting Accurate data for 'Deadlock Detected' event from W3SVC-WP

webengine!ReportHealthProblem

webengine4!ReportHealthProblem