Troubleshooting OOM (“Out of Memory”) Exceptions in ASP.net web applications

on IIS 5.0/IIS 5.1/IIS 6.0 with Debugdiag and Perfmon

Summary:

The following steps will configure Perfmon to capture relevant data leading up to the OOM exceptions DebugDiag to automatically trigger a memory dump of the IIS processes when the System.OutOfMemoryException is observed in the browser of the client or in the application event log of the web server. If Debugdiag does not create a dump at the right time, steps for triggering a manual hang dump as a contingency plan are also included here.

Preliminaries:

Consider .Net Updates?

Consider what updates to the .net Framework you might be able to install. Generally speaking it’s a good idea to be fairly up to date on patches and updates. As an example, if your asp.net 1.1 web application is throwing OOM exceptions, it would be a good idea to AT LEAST be on SP1 for the 1.1 framework.

One possible way to see what updates are recommended for your server is to run the windows update analysis to see what is recommended (but then don’t necessarily proceed to install the recommended updates until you’ve researched them, tested then in the QA/Dev environment, and/or made your system backups.)

Debug=True?

Check all web.config files to see if Debug is set to True. In production web servers, I highly recommend that debug be set to false—especially if you’re seeing OOM exceptions. This is a common cause (but certainly not the only cause) of OOM exceptions. If Debug is set to True in any web.config file on the afflicted server, I’d recommend setting Debug to False in every web.config file followed by an iisreset at your next available server maintenance window.

Also ask web developers to consider whether dlls are known to be compiled in release mode or debug mode. It might be prudent to recompile dlls in release mode and redeploy them.

To read more on this, please consider my ASPNETDebugTrue page.

Install DebugDiag

Debugdiag may be downloaded from:

http://www.microsoft.com/downloads/results.aspx?pocId=&freetext=debugdiag&DisplayLang=en

(This is the same as going to http://microsoft.com/downloads and searching all downloads for “Debugdiag”)

Notes:

· Please download the newest version and install DebugDiag on the server exhibiting unresponsive web pages.  If you want to attach a debugger to a process on a remote machine, you’ll want to use windbg.exe.

· FREESPACE - Since dump files will generally be created in the same path, please be sure to install this on a drive that has at least one gigabyte of free space.  The default install path is C:\Program Files\IIS Resources\DebugDiag but may be changed at the beginning of the installation.  If free space is limited, you may get an idea of the minimum size of the .dmp files by opening taskmanager and adding up the amounts of virtual memory used by inetinfo.exe, dllhost.exe’s, aspnet_wp.exe, and/or w3wp.exe’s. We will have a .dmp file for each of these processes that you see in taskmanager.

· Before using debugdiag, ensure that no other debugging tools are running. For instance, if iisdump.exe was installed earlier, it needs to be removed or else its isapi filter needs to be removed from IIS. If ADplus.vbs is installed, that’s fine, but don’t attach it to the iis processes while running debugdiag.

Create and Start a Performance Monitor capture

Start Button > Run > Open: Perfmon [Enter]

Expand "Performance Logs and Alerts"

Right Click on "Counter Logs"

Choose "New Log Settings..."

Enter a descriptive name (such as OOM)

Note the log file location for later (or go to the "Log Files" tab and change the location)

Click the "Add" button

Click the "All Counters" and "All Instances" radio buttons

Select the following from the "Performance Object" dropdown, being sure to "Add" each one as you select it:

Add every Object that begins with “.NET” (such as, .NET CLR Data, .NET CLR Exceptions, .NET CLR Interop, etc.)

Add every Object that begins with ASP.NET (such as ASP.NET, ASP.NET Applications, etc.)

Memory

Process

Processor

Thread

Web Service

Click "Close"

Click "OK"

Notes:

Normally Perfmon is opened from C:\windows\system32\perfmon.exe. If however your server is a 64-bit operating system and your “aspnet” and “aspnet applications” objects are missing from Perfmon, try opening it from this alternate location: C:\windows\SysWOW64\perfmon.exe

Configure server to capture OutOfMemoryException

Open regedit and add the following registry key.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\

DWord: GCFailFastOnOOM

Value: 2

This was added to the .NET Framework so that a breakpoint exception will be thrown when and OutOfMemory condition occurs.

In theory you do not have to reboot the server or restart IIS to make this take effect. (However, it might not hurt to reboot or run iisreset if convenient.)

For further explanation on this, visit kb 820745 Failfast occurs when you experience an "out of memory" condition

Remove Recycling Triggers

Open the IIS administration console, visit the properties sheet of the application pool we’re focusing our troubleshooting on.

This will help ensure that the w3wp.exe grows and grows (either in size and/or in level of memory fragmentation) unimpeded. 

This also will help to ensure that the crash rules do not create worthless dumps by a recycle of a w3wp.exe.

Please note the settings before changing them since this can be changed back after troubleshooting has finished.

On the recycling tab, please uncheck all three of the  “recycle worker processes” options.

Also please uncheck both of the options for “memory recycling.”

On the Performance Tab please uncheck Idle timeout.

On the Health tab, please uncheck enable pinging.

Configure DebugDiag Crash Rule

Configure DebugDiag to capture the memory dump when the BreakPoint Exception is thrown and when the process stops.

Launch Debug Diagnostics 1.1 (Start > All Programs > Debug Diagnostic Tool 1.1 > Debug Diagnostics 1.1)

In the Select Rule Type window (On the Rules tab, click Add Rule) select Crash and click Next

Select "All IIS Processes" and click Next

In the Advanced Exception Configuration Window, click the [Breakpoints] button

Click Add Breakpoint

Select KERNEL32!ExitProcess

Change Action Type to Full UserDump

Click OK

Click Save and Close

Click Next through the rest of the wizard to accept defaults and activate the rule

Inject Leaktracking

Switch to the Processes tab in DebugDiag

Look in the list of processes for process(es) (aspnet_wp.exe or w3wp.exe) we’re focusing our troubleshooting upon.

Right-click the process and select "Monitor for leaks" from the gray menu.

WAIT

Wait for ASP.net to begin reporting the throwing of the OOM exceptions. 

In theory, when asp.net throws the first system.outofmemory exception, the gcfailfastonoom breakpoint will be reached and a dump will be triggered.

Assuming this works as planned, the userdump count in debugdiag will increase from 0 to 1.

If, however, the dumps are not automatically produced when the OOMs begin to be thrown, here is the contingency plan. . .

DEBUGDIAG HANG DUMP

The OOM exceptions usually begin with a trickle that may escape the notice of most and eventually, as there is less and less available memory, the ooms are thrown in greater frequency. When there is a barrage of OOMs, the process serving the problematic asp.net application begins to hang and/or become very noticeable to the client experience. If the OOMs are caused by memory fragmentation, the size of the w3wp.exe may not be very large—800mb perhaps. If the OOMs are caused by high memory usage, the w3wp.exe may reach the size of 1.3gb or even 1.6 gb. (This assumes no /3gb and a 32bit OS. With the /3gb on a 32 bit OS the maximum size of the worker process may be 2.2 gb or so. On a 64 bit OS, the size of the w3wp.exe serving the asp.net 2.0 application may be up to 16 gb in size.)

When the barrage of OOMs become noticeable, please log onto the server, launch debugdiag, and manually trigger a hang dump with debugdiag with the following steps:

a)      Launch DebugDiag

b)      Click Cancel if given the choice of making a Crash Rule or Hang Rule

c)       Expand the Tools menu

d)      Select “Create IIS/Com+ Hang Dump” and wait for dump creation to occur (may take 30 seconds or more depending on size of the w3wp.exe)

Wait for the dumps to finish (this may take a few minutes)

RECOVERY

After Debugdiag has finished making its dumps—whether automated dumps from the crash rule or from manually triggered dumps in the middle of a barrage of oom exceptions-- feel free to restart the w3wp.exe to recover from the OOM condition by (a.) recycling the application pool, (b.) running iisreset or (c.) rebooting the server.

COLLECT DATA

Perfmon

Stop the Performance Monitor log

In Performance Monitor:

1. Right click on your log that is now listed under "Counter Logs"

2. Choose "Stop Log"

3. Locate the .blg file and upload it to the workspace or ftp site

Dumps, Event logs, etc.

Advanced Data Collection

The following action will take the dump files, event logs, recent IIS logs, metabase and compress them all into one .cab file.

Expand the Tools menu

Select “Advanced Data Collection”

Select “Create Full Cabinet File”

Click OK/NEXT for any subsequent questions

After the compression process is finished, the easiest way to find the cabinet file is to click the icon of the manila folder in debugdiag.

UPLOAD FILES

Upload the files to your ASP.net engineer for analysis

1. The dumpfiles and event logs should already be compressed into one cabinet file which by default is located at:

C:\Program Files\IIS Resources\DebugDiag\Logs\<Name of Rule>

Please upload whatever .cab file exists at this location.

2. If you chose a non-default location for the dumps to be sent to, you will probably need to zip those dmp files and upload them as well. (The Advanced Data Collection > Create Full Cabinet File only compresses the folders and files found in C:\Program Files\IIS Resources\DebugDiag\Logs\

3. Blg files from perfmon need to be uploaded too.

Upload the files to the workspace provided by your support engineer.

After troubleshooting

Deactivate or Remove the crash rule in Debugdiag

Remove this registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\

DWord: GCFailFastOnOOM

Value: 2

Add any desired recycling options back to the application pool(s)