DebugDiag v1.2 Crash Dump Rules for .Net Exceptions
Overview/Index
The stuff you have to do before doing any logging or dumping.
· Steps to Log all .net Exceptions to a Logfile
If the problem is very mysterious, it may be best to start by getting a log file that shows all the .net exceptions are being thrown
· Steps to Create Dump files of a Process when ANY/ALL .net Exceptions are thrown
How to create dumps while reproducing the problem. If you have plenty of free space, if the problem is easy to reproduce, and if you’re the only one hitting the website, this might be worth considering.
· Steps to trigger a dump of a process when a specific exception is thrown
How to dump a process when you know exactly which exception you’re hunting
· What to do with the dump files after you have produced them?
Preliminary steps
On the web server that is exhibiting problems, install DebugDiag v1.2 from here:
http://www.microsoft.com/download/en/details.aspx?id=26798
Launch Debug Diagnostics from the list of programs. (Be sure to “Run as Administrator” to avoid this possible error: “error while attaching to process via the DbgSvc service. ReloadControlScriptFailed. Could not open handle to control script shared memory mutex.”)
If possible, reproduce the problem in a Test environment. If the problem only occurs on web servers in the production environment, one way to be cautious with these steps and only run this when client traffic to the production servers is at its lowest. Doing so will help minimize any chance of bogging down the server (unlikely but possible) and will reduce the amount of data you may have to sift through later.
During installation of DebugDiag, I generally recommend you accept all the default settings. One exception I’d make here is if you have limited free space on the system partition or if you have policies that prevent you from installing software to the system partition. In either of those cases, you should install it to another drive on the server.
Steps to Log all .net Exceptions to a Log file
If you’re not already sure which exception to focus upon (perhaps it’s not shown in the Application Event log, custom logging, or shown clearly in the browser), it may be best to start by logging all .net exceptions while reproducing the problem. ASP.net is pretty good about making its complaints known in the form of exceptions. Many or perhaps most exceptions can be ignored. But if you’re trying to troubleshoot a mysterious problem that isn’t leaving clues in the event logs, this may be a good way to begin to unravel the problem.
Launch Debugdiag from the Programs Menu
Select “Crash” as the rule type
Assuming you’re troubleshooting a problem that is occurring in an IIS/ASP.net process, try to select
“A Specific IIS Web Application Pool”
If you know which application pool to focus upon.
If you need to figure out which w3wp.exe matches which IIS Application Pool,
you can visit the processes tab of DebugDiag to get this information.
If you’re not sure which IIS process to focus on, you can select
“All IIS/Com+ related processes.”
If it’s not an IIS process but another process throwing .net exceptions, you can select
“A specific process”
In this demo I’m selecting a specific AppPool and clicking Next.
Select the AppPool to focus on and click Next again
Select the Exceptions button
Select the Add Exception button
Highlight the Exception Code E0434F4D in the left pane.
This assumes that the web application we’re troubleshooting is written in asp.net 2.0, 3.5, 1.1 or 1.0.
(But if it’s asp.net 4.0, select the E0434352 code of course.)
For now, let’s keep the exception type blank,
Keep the action type set to “Log Stack Trace,”
And bump the Action Limit up from 1 to some high number – 9999, for example.
Click OK
Click the “Save and Close” button.
On the Advanced Configuration (Optional) window, simply click NEXT.
There is no need to make changes here.
On the Select Dump Location and Rule Name window, feel free to change the rule name and the userdump location.
Neither is relevant at this point. No userdumps will be created as things are so far configured.
And the logfiles that are produced will probably go to C:\Program Files\DebugDiag\Logs regardless.
Click Next.
On the Rule Completed window, you have a choice on whether you’re ready to activate the rule or not.
If you’re ready to reproduce the problem (one or two mouse clicks away) then activate it now.
Otherwise select “Do not activate the rule at this time” for now and activate it later.
Click Finish.
If you see this message about Symbol Paths, you can select YES.
Assuming the rule was not activated upon rule finish, you can activate the rule when you are ready to reproduce the problem.
If possible, activate the rule when you are one or two mouse clicks away from reproducing the problem.
Also, if possible, reproduce the problem in a test environment rather than in a production environment.
If you do reproduce the problem while logging exceptions with debugdiag, try to reproduce the problem when traffic to the application pool or process that you’re monitoring is at its lowest levels.
Once the rule is activated, most of the asp.net exceptions thrown inside of the selected process should be logged to a logfile.
After reproducing your problem, you can locate the log file by clicking the icon of the manila folder.
This will open the folder C:\Program Files\DebugDiag\Logs
The file you’re looking for may look a little like this one:
w3wp__DefaultAppPool__PID__4840__Date__07_19_2011__Time_04_02_18PM__81__Log.txt
The exceptions logged in the log file will not have verbose information. They will not show full stack traces, for example. Mainly this is useful for knowing which exceptions are occurring when the problem is reproduced. You can then use that information to help you produce a dump file of the iis process at the right time the next time that exception is thrown.
Ideally, you can isolate that asp.net application into its own application pool if possible, make sure no one else is browsing to that application, activate the rule, reproduce the problem, deactivate the rule, and check your log files to see which exception is thrown. Several exceptions may be thrown. Not all exceptions are bad. You can do some logging of exceptions while running the application in a way that does NOT reproduce the problem to get a baseline and know better which exceptions are “normal” and expected.
If there is a particular exception that stands out as something you’d like to find root cause for, copy it to your clipboard and go on to Steps to trigger a dump of a process when a specific exception is thrown.
Steps to Create Dump files of a Process when ANY/ALL .net Exceptions are thrown
Warning: If you do attempt to create process dumps when any and every .net exception is thrown, there is a slight possibility that you could make your server so unresponsive that you may have to cold boot it from the data center to recover. This might depend on how many exceptions are being thrown. I recommend you start with the instructions above to merely log the exceptions to a log file first before you consider proceeding to make dumps for all exceptions. If you see less than ten exceptions in the log file when reproducing the problem you are troubleshooting, you could consider dumping all of those exceptions to a dump file if you’re not sure which specific exception to focus on.
Launch Debugdiag from the Programs Menu
Select “Crash” as the rule type
Assuming you’re troubleshooting a problem that is occurring in an IIS/ASP.net process, try to select
“A Specific IIS Web Application Pool”
If you know which application pool to focus upon.
If you need to figure out which w3wp.exe matches which IIS Application Pool,
you can visit the processes tab of DebugDiag to get this information.
If you’re not sure which IIS process to focus on, you can select
“All IIS/Com+ related processes.”
If it’s not an IIS process but another process throwing .net exceptions, you can select
“A specific process”
In this demo I’m selecting a specific AppPool and clicking Next.
Select the AppPool to focus on and click Next again
Select the Exceptions button
Select the Add Exception button
Optional: Remove any exceptions that may have been set up earlier.
Highlight the Exception Code E0434F4D in the left pane.
This assumes that the web application we’re troubleshooting is written in asp.net 2.0, 3.5, 1.1 or 1.0.
(But if it’s asp.net 4.0, select the E0434352 code of course.)
Set the Action Type to: Full Userdump
Set the Action Limit to: 15 (or fewer)(do you have enough freespace?)
Click Save and Close button
Click the “Save and Close” button.
On the Advanced Configuration (Optional) window, simply click NEXT.
There is no need to make changes here.
Maximum number of userdumps created by this rule could limit you to 10 dumps total.
On the Select Dump Location and Rule Name window, feel free to change the rule name and the userdump location.
On the Rule Completed window, you have a choice on whether you’re ready to activate the rule or not.
If you’re ready to reproduce the problem (one or two mouse clicks away) then activate it now.
Otherwise select “Do not activate the rule at this time” for now and activate it later.
Click Finish.
If you see this message about Symbol Paths, you can select YES.
Assuming the rule was not activated upon rule finish, you can activate the rule when you are ready to reproduce the problem.
If possible, activate the rule when you are one or two mouse clicks away from reproducing the problem.
Also, if possible, reproduce the problem in a test environment rather than in a production environment.
If you do reproduce the problem while logging exceptions with DebugDiag, try to reproduce the problem when traffic to the application pool or process that you’re monitoring is at its lowest levels.
Whenever an CLR/.Net exception is thrown inside of the selected process, debugdiag should create a memory dump of the process being monitored when any .net exception is thrown. If you reproduce the problem and five exceptions are thrown, you should expect to see five dump files created. There is very possibly going to be some unnecessary redundancy with this method if several dumps are created.
You might try to be more “granular” or “surgical” with the next method instead. . .
Steps to trigger a dump of a process when a specific exception is thrown
Launch Debugdiag from the Programs Menu
Select “Crash” as the rule type
Assuming you’re troubleshooting a problem that is occurring in an IIS/ASP.net process, try to select
“A Specific IIS Web Application Pool”
If you know which application pool to focus upon.
If you need to figure out which w3wp.exe matches which IIS Application Pool,
you can visit the processes tab of DebugDiag to get this information.
If you’re not sure which IIS process to focus on, you can select
“All IIS/Com+ related processes.”
If it’s not an IIS process but another process throwing .net exceptions, you can select
“A specific process”
In this demo I’m selecting a specific AppPool and clicking Next.
Select the AppPool to focus on and click Next again
Select the Exceptions button
Select the Add Exception button
Highlight the Exception Code E0434F4D in the left pane.
This assumes that the web application we’re troubleshooting is written in asp.net 2.0, 3.5, 1.1 or 1.0.
(But if it’s asp.net 4.0, select the E0434352 code of course.)
Set Action Type to: Full Userdump
Set Action Limit to: 1 or 2.
If you can reproduce the problem quickly with one or two mouse clicks, 1 should be fine.
If you have to let this rule monitor the process for hours before your problem is reproduced, you might want to set the action limit higher in case there are “false positives” that occur.
In the .NET Exception Type field type in the specific exception you’re troubleshooting.
If you leave it blank, you’ll get dumps for any and every .net exception
If you mistype it or don’t pay attention to case sensitivity you may never get the dump. (Because the exception its monitoring for is never thrown.)
The exception should be typed System.Whatever (two levels deep) or even System.Whatever.Whatever (three levels deep).
Examples of exceptions:
System.InvalidCastException
System.OutOfMemoryException
System.IO.FileNotFoundException
System.NullReferenceException
System.CannotUnloadAppDomain
System.Threading.SynchronizationLock
System.InvalidOperationException
System.OverflowException
System.Data.SqlClient.SqlExeption
System.Data.VersionNotFoundException
System.ArgumentException
System.ExecutionEngineException
System.ArgumentOutOfRangeException
System.IndexOutOfRangeException
System.Net.WebException
System.Security.SecurityException
Click OK
Click the “Save and Close” button.
On the Advanced Configuration (Optional) window, simply click NEXT.
There is no need to make changes here.
On the Select Dump Location and Rule Name window, feel free to change the rule name and the userdump location.
On the Rule Completed window, you have a choice on whether you’re ready to activate the rule or not.
If you’re ready to reproduce the problem (one or two mouse clicks away) then activate it now.
Otherwise select “Do not activate the rule at this time” for now and activate it later.
Click Finish.
If you see this message about Symbol Paths, you can select YES.
Assuming the rule was not activated upon rule finish, you can activate the rule when you are ready to reproduce the problem.
If possible, activate the rule when you are one or two mouse clicks away from reproducing the problem.
Also, if possible, reproduce the problem in a test environment rather than in a production environment.
If you do reproduce the problem while logging exceptions with debugdiag, try to reproduce the problem when traffic to the application pool or process that you’re monitoring is at its lowest levels.
Whenever an CLR/.Net exception is thrown inside of the selected process, debugdiag should create a memory dump of the process being monitored when that exception is thrown.
What to do with the dump files after you have produced them?
DebugDiag 1.2 has its own analysis scripts!
Those scripts use psscor.dll, psscor2.dll, and psscor4.dll extensions to help you to begin to make sense of the dumps.
This is a good place to start. Just click ADD DATA FILES and guide it to the .dmp file, click START ANALYSIS. (This assumes the machine running debugdiag has outbound internet access. Many web servers do not. If not, install DebugDiag on a workstation that does have internet access and share out the dump file.)
You can also open the dmp files up with Windbg.exe (from the debugging tools for windows), load one of the psscor extensions (which you can borrow from C:\Program Files\DebugDiag\Exts or download from http://microsoft.com/downloads) and start debugging on your own.
But often it’s just best to open a support case with Microsoft (http://microsoft.com/support) and have them debug the dumps for you.