Troubleshooting high CPU hangs in a
W3WP.exe with Perfmon and DebugDiag 1.1
in IIS 6.0 and ASP.net web
applications
Preliminaries:
DEBUG TRUE CHECK
If you’re running any asp.net web
applications in the IIS processes that are showing high cpu, please double-check
all web.config files to see if Debug is set to True. In production web
servers, I highly recommend that debug be set to false. If Debug is set
to True in any web.config file on the afflicted server, I’d recommend setting
Debug to False in every web.config file followed by an iisreset at your next
available server maintenance window. To
read more on this, please consider this page: ASPNETDebugTrue.
INSTALL DEBUGDIAG v 1.1
Debugdiag may be downloaded from http://microsoft.com/downloads
Search “All Products” there
for: Debugdiag
DISABLE APP POOL RECYCLING - ***This step is not
necessary but often tends to be a good idea***
Disable Health Monitoring with the
following steps:
Open the IIS manager and drill down
into the Application Pools
Right-click the Application Pool that
has been crashing and select ‘Properties’
On the recycling tab, please uncheck
all three of the “recycle worker processes” options.
Also uncheck both of the options for
“memory recycling” while trying to get the dumps.
They can be added again after we are
finished troubleshooting.
On the Performance tab please uncheck
"Shutdown worker processes after being idle for"
On the Health tab please uncheck
"Enable rapid-fail protection" and “Enable pinging.”
Please ensure that Web Garden is set
to only 1.
Click Apply and OK
Create and Start a Performance
Monitor capture
Start Button > Run > Open:
Perfmon [Enter]
Expand "Performance Logs and
Alerts"
Right Click on "Counter
Logs"
Choose "New Log
Settings..."
Enter a descriptive name (such as
“High-CPU-IIS”)
Note the log file location for later
(or go to the "Log Files" tab and change the location)
Click the "Add" button
Click the "All Counters"
and "All Instances" radio buttons
Select the following from the
"Performance Object" dropdown, being sure to "Add" each one
as you select it:
·
Add
every Object that begins with “.NET” (such as, .NET CLR Data, .NET CLR
Exceptions, .NET CLR Interop, etc.)
·
Add
every Object that begins with ASP.NET (such as ASP.NET, ASP.NET Applications,
etc.)
·
Memory
·
Process
·
Processor
·
Thread
·
Web
Service
Feel free to add whatever else you
might wish
Click "Close"
Click "OK"
The icon of the perfmon capture you’ve
created should be green when it is actively capturing performance data. It will be red when it is stopped. Please make sure that it is green/running
before the high cpu returns.
Wait for the high-cpu hang to occur.
Manually Trigger 3 Hang Dumps
When the IIS process begins to exhibit
unhealthy levels of high cpu, and when the hang (or extreme performance
hit) is seen by the clients browsing the web application, here are the steps to
create three sets of hang dumps of the IIS processes:
1. Launch DebugDiag
2. Click Cancel if given the choice of
making a Crash Rule or Hang Rule
3. Expand the Tools menu
4. Select “Create IIS/Com+ Hang Dump” and wait for dump
creation to occur (may take 30 seconds or more depending on size of the
w3wp.exe)
5. Wait for thirty seconds minutes after
the first set of hang dumps is finished
6. Trigger a second set of hang dumps
with the “Create IIS/Com+
Hang Dump” again
7. Wait for thirty seconds minutes after
the second set of hang dumps is finished
8. Trigger a third set of hang dumps
with the “Create IIS/Com+
Hang Dump” again
IISRESET
After the dmp files are fully written,
feel free to restart IIS (or restart the application pool or whatever you
normally do) to recover temporarily from the hang
But please remember that it is
important to NOT restart any IIS processes before the logs and dmp files are
all completely finished being written to as an IISRESET will potentially ruin
the dumps. Please wait for DebugDiag to finish its work before recycling
IIS. This could take anywhere from thirty seconds to perhaps more than
three minutes (depending on the size of the iis processes).
STOP PERFMON
Stop the Performance Monitor log
In Performance Monitor:
1. Right click on your log that is now
listed under "Counter Logs"
2. Choose "Stop Log"
3. Save it as a .blg file to the
location of your choice
4. please zip the .blg file
ZIP DUMP FILES
In debugdiag, expand the tools menu, select “Advanced Data Collection,” and
select “Create Full Cabinet File”
Creating the full cabinet file will
compress the event logs, the most recent IIS log, the metabase, the httperr
log, all .net config files, and all dmp files together into one convenient .cab
file.
After the compression process is
finished, the easiest way to find the cabinet file is to click the icon of the
manila folder in debugdiag.
This usually turns out to be: C:\Program Files\IIS
Resources\DebugDiag\Logs\<Name of Rule>
Please upload whatever .cab file exists at this location.
Also upload the zipped Perfmon .blg
file.
Please upload the files to the secure
file transfer workspace your Microsoft Engineer has prepared for you.