Troubleshooting High CPU in an IIS/Com+ Process in IIS 5.0

Using Perfmon and DebugDiag 1.1

Debug=True?

If you’re using asp.net at all, please ensure that all web.config files to see if Debug is set to True.  In high-traffic production web servers, debug statements should be set to false.  Debug should only be set to true in a pre-production, developmental, and/or QA environment.  If Debug is set to True in any web.config file on the afflicted server, please set Debug to False in every web.config file followed by an iisreset at your next available server maintenance window.

To read more on this, please consider my ASPNETDebugTrue page.

Install DebugDiag v. 1.1

Debugdiag may be downloaded from:

http://www.microsoft.com/downloads/results.aspx?pocId=&freetext=debugdiag&DisplayLang=en

(This is the same as going to http://microsoft.com/downloads  and searching all downloads for “Debugdiag”)

Notes:

· FREESPACE - Since dump files will generally be created in the same path, please be sure to install this on a drive that has at least one gigabyte of free space.  The default install path is C:\Program Files\IIS Resources\DebugDiag but may be changed at the beginning of the installation.  If free space is limited, you may get an idea of the minimum size of the .dmp files by opening taskmanager and adding up the amounts of virtual memory used by inetinfo.exe, dllhost.exe’s, aspnet_wp.exe, and/or w3wp.exe’s. We will have a .dmp file for each of these processes that you see in taskmanager.

  Create and Start a Performance Monitor capture

Start Button > Run > Open: Perfmon [Enter]

Expand "Performance Logs and Alerts"

Right Click on "Counter Logs"

Choose "New Log Settings..."

Enter a descriptive name (such as High-CPU)

Note the log file location for later (or go to the "Log Files" tab and change the location)

Click the "Add" button

Click the "All Counters" and "All Instances" radio buttons

Select the following from the "Performance Object" dropdown, being sure to "Add" each one as you select it:

Add every Object that begins with “.NET”  (such as, .NET CLR Data, .NET CLR Exceptions,  .NET CLR Interop, etc.)

Add every Object that begins with ASP.NET (such as ASP.NET, ASP.NET Applications, etc.)

Add Active Server Pages

Add Memory

Add Process

Add Processor

Add Thread

Add Web Service

Click "Close"

Click "OK"

Wait for the high CPU to begin to rise

Wait for CPU to begin to spike in inetinfo.exe, dllhost.exe, or aspnet_wp.exe. And/or wait until the process starts to become nonperformant/slow or hung due to high cpu.

Manually Triggering Five Hang Dumps

When the web application(s) begin to exhibit hung performance, here are the steps to create five sets of hang dumps of the IIS processes

a) Launch DebugDiag

b) Click Cancel if given the choice of making a Crash Rule or Hang Rule

c) Expand the Tools menu

d) Select “Create IIS/Com+ Hang Dump” and wait for dump creation to occur (may take 30 seconds or more depending on size of the w3wp.exe)

e) Wait 20-30 seconds after the first set of hang dumps is finished

f) Trigger a second set of hang dumps with the “Create IIS/Com+ Hang Dump” again

g)   Wait 20-30 seconds after the first set of hang dumps is finished

h) Trigger a third set of hang dumps with the “Create IIS/Com+ Hang Dump” again

i) Wait 20-30 seconds after the first set of hang dumps is finished

j) Trigger a fourth set of hang dumps with the “Create IIS/Com+ Hang Dump” again

k) Wait 20-30 seconds after the first set of hang dumps is finished

l) Trigger a fifth set of hang dumps with the “Create IIS/Com+ Hang Dump” again

m) Wait for the fifth set of dumps to finish being written to.

IISRESET

After the dmp files are fully written to, feel free to restart IIS to recover temporarily from the hang.

(Or recycle the Com+ component if applicable.)

STOP PERFMON

Stop the Performance Monitor log

In Performance Monitor:

1. Right click on your log that is now listed under "Counter Logs"

2. Choose "Stop Log"

3. Save it as a .blg file to the location of your choice

Zip the Data

In Debugdiag’s Tools menu, select “Advanced Data Collection” and select “Create Full Cabinet File”

Creating the full cabinet file will compress the event logs, the most recent IIS log, the metabase, the httperr log, all .net config files, and all dmp files together into one convenient .cab file.

Upload the Data

Upload the files to your Developer Support engineer for post-mortem dump analysis. Your support engineer should have created a secure workspace (https) or may have an alternative FTP site you can upload too.

Please stop, save, zip and upload the Perfmon .blg file

After the debugdiag “advanced data collection” compression process is finished, the easiest way to find the cabinet file is to click the icon of the manila folder in debugdiag.

The dumpfiles and event logs should already be compressed into one cabinet file which by default is located at:

C:\Program Files\IIS Resources\DebugDiag\Logs\<Name of Rule>

Please upload whatever .cab file exists at this location.

If you chose a non-default location for the dumps to be sent to, you will probably need to zip those dmp files and upload them as well.  (The Advanced Data Collection > Create Full Cabinet File only compresses the folders and files found in C:\Program Files\IIS Resources\DebugDiag\Logs\