Troubleshooting High CPU in a 64-bit IIS/ASP.net process with ADPlus and Perfmon

Overview:

The aim here is to get a perfmon capture started before the high-cpu begins and to get at least three “hang dumps” with adplus while cpu is high. After the three hang dumps are made, stop the perfmon capture. Zip and upload the dumps and perfmon .cap file for analysis.

1. Install the 64 bit debuggers

Since we don’t yet (as of October 2008) have a working 64bit version of Debugdiag, please install the 64 bit version of the Debugging Tools for Windows. http://www.microsoft.com/whdc/devtools/debugging/install64bit.mspx#

2. Create a Performance Monitor capture

Start Button > Run > Open: Perfmon [Enter]

Expand "Performance Logs and Alerts"

Right Click on "Counter Logs"

Choose "New Log Settings..."

Enter a descriptive name (such as High CPU)

Note the log file location for later (or go to the "Log Files" tab and change the location)

Click the "Add" button (or “Add Objects” depending on your OS)

Click the "All Counters" and "All Instances" radio buttons (You may just need to choose Add Objects depending on your OS and all the counters available for each object will automatically be included.)

Select the following from the "Performance Object" dropdown, being sure to "Add" each one as you select it:

Add every Object that begins with “.NET”  (such as, .NET CLR Data, .NET CLR Exceptions,  .NET CLR Interop, etc.)

Add every Object that begins with ASP.NET (such as ASP.NET, ASP.NET Applications, etc.)

Memory

Process

Processor

Thread

Web Service

After you’ve added all of these objects,

Click "Close"

If you expect that the high cpu may take many hours or many days to return, consider changing the sampling interval from 15 seconds to something more like 45 seconds or 90 seconds. This will keep the size of the perfmon capture down.

Click "OK"

Notes:

Normally Perfmon is opened from C:\windows\system32\perfmon.exe.  If however your server is a 64-bit operating system and your “aspnet” and “aspnet applications” objects are missing from Perfmon, try opening it from this alternate location: C:\windows\SysWOW64\perfmon.exe

3. Start the Perfmon capture

You don’t necessarily need to start the capture immediately. If it is going to be several hours before the high cpu starts, you might want to stop the capture during the initial phase of waiting and start the capture an hour before you expect the high cpu to possibly return.

If the counter log is showing up with a green icon, the capture has been started.

To stop the perfmon counter, right click the Green icon and choose stop.

If you have to wait a few days for the high cpu to show up, you might consider stopping the perfmon capture every day (or every few hours) and restarting the capture. That way the size of the .cap file will not be gargantuan by the time it needs to be uploaded. The aim here is to get a perfmon capture started before the high-cpu begins and to get at least three “hang dumps” while cpu is high.

4. Wait for the CPU to begin to soar

We want the perfmon capture running before the high cpu really begins to skyrocket. But we don’t want to make any hang dumps until problems really begin due to high cpu.

C:\Program Files\Debugging Tools for Windows 64-bit\> cscript.exe adplus.vbs –hang –iis –quiet

You’ll see minimized command windows appear for each IIS process that the debugger is attached to. As the dump files are written, the command windows will disappear automatically. When all the command windows have disappeared, the first set of dumps is finished and you’re ready to make your next set of dumps.

Wait ten more seconds and then run the same adplus command again. Wait for the second set of dumps to finish being written to.

Wait ten more seconds and run the same adplus command again to create the third set of dumps.

If you’d like to be thorough, go for five sets of dumps total. Usually three is enough. But five is even better.

Optional Note: While waiting for the high cpu, you might consider setting up a Perfmon Alert. The Alert can be crafted to send a network message, send an email through smtp, or can even trigger a dump with adplus through a simple batch file. This is not a necessary step however.

5. Stop the Perfmon Counter Log capture

Right-click the green icon and choose stop.

6. Recover IIS

After the third set of dumps has finished being produced, feel free to run IISRESET, recycle the application pool, or do whatever it is you do to get some temporary relief from the high cpu.

7. Collect, Zip and Upload Data

Please upload the zipped dump files and the zipped perfmon blg file to the workspace or ftp site your IIS/ASP.net support professional has provided