Should you Set up a Web Garden in IIS?
IIS 6.0
IIS 7.5
Summary:
Should you set up a web garden in IIS?
There is some debate over how useful
this is. My general rule of thumb is:
In 90% of environments/scenarios, it would NOT be helpful to increase
this value. Occasionally, however, when there are multiple CPUs on the
server and/or there are long running requests from IIS to database, this may be
helpful.
There are very few instances where web
gardening is recommended. One of the few times we’d recommend considering
the testing of web gardening is when it is fully expected that some calls from
the web application to a backend database may be “long running” (several
minutes?).
More often than not, web gardening
actually causes problems, especially with web applications that might suffer
from session being broken. When testing, pay special attention to (1) session
maintenance and (2) CPU usage. Those are probably the two key worries to focus
on. The only way to know if this is
right for your server is to try it out and test it thoroughly.
Generally it is better to create
additional application pools and attempt to isolate websites, web apps, and/or
virtual directories into their own application pools than to use web gardening.
The following excerpts address web
gardening in IIS 6/7/7.5 as set up through the IIS Manager console. Some links, however, address web gardening in
Asp.net 1.0 and 1.1 as set up in web.config files (as was done in Windows 2000/IIS
5.0 and Windows 2003 if IIS is running in IIS 5 isolation mode). The same principles apply.
http://msdn.microsoft.com/en-us/library/5dws599a(VS.71).aspx
Choose your
session-state provider carefully. ASP.NET
provides three distinct ways to store session data for your application:
in-process session state, out-of-process session state as a Windows service,
and out-of-process session state in a SQL Server database. Each has it
advantages, but in-process session state is by far the fastest solution. If you
are storing only small amounts of volatile data in session state, it is
recommended that you use the in-process provider. The out-of-process solutions are useful primarily if you
scale your application across multiple processors or multiple computers, or
where data cannot be lost if a server or process is restarted. For more
information, see ASP.NET
State Management.
For applications that rely extensively on external
resources, consider enabling Web gardening on multiprocessor computers. The ASP.NET process model
helps enable scalability on multiprocessor computers by distributing work to
several processes, one per CPU, each with processor affinity set to its CPU.
This technique is called Web gardening. If your application uses a slow database server or calls
COM objects that have external dependencies, to name only a couple of
possibilities, it can be beneficial to enable Web gardening for your
application. However, you should test how well your application performs in a
Web garden before your decide to enable this.
http://blogs.technet.com/mscom/archive/2007/07/10/gardening-on-the-web-server.aspx
Gardening
(on the Web Server)
While some people are born with a green thumb, others of us have the brown
thumb that comes from working to get a garden to grow. While we dream of sweet fruits and crisp,
fresh vegetables, sometimes all we get for our troubles is dirt under our
fingernails. There is a feature in IIS6
that can have similar results and oddly enough shares the name.
IIS6 Web Gardens is a nifty idea.
With the simple click of the mouse (or a few key strokes using
adsutil.vbs), you can magically make your application pool use many worker
processes. While this seems to imply a
"green thumb" approach to solving issues like scalability, with just
a little clearer understanding, we can see the dirt (and compost) that appears
under the glossy exterior.
What does enabling a Web Garden really do?
As the configuration suggests, it causes the application pool (and
therefore each application in that apppool) to be running in multiple
processes. The key that people tend to not notice (or not care to
think about) is the fact that each worker process (and the applications loaded)
work independently and concurrently.
What does this mean? It means, if
you use a caching system, the cache will be duplicated in each application
pool. It means, if you use a resource,
that resource needs to be shared by each instance in the application. It means, that you now have doubled (or worse
depending on how high you set the number) the number of threads vying for CPU
time. It means, if you are using ASP.NET
there will be that many more Garbage Collection threads - all running at high
priority if using the default server GC - consuming the CPU time at an alarming
rate. In short (yeah, too late for
that), it means more resources used with worse performance, due to the decrease
in resource reuse.
Note that many applications have problems dealing with Web Gardens. The systemic problem is usually the same that
doesn't allow a given application to run in multiple application pools or
enabling over-lapped recycling - namely that many applications assume dedicated
access to some system level resource. In
other words when this kind of application starts, it takes exclusive access to
a resource; so when a second version of the application starts, it fails to get
this exclusive access.
In much of the positive "literature" about this feature, one of
the reasons people give for enabling Web Gardens is "to enhance performance" (IIS6
Resource Kit page 666 - just the page number alone should give us some hint
about this feature and performance).
While using multiple process can remove a bottleneck related to a highly
latent back-end dependency, from a performance point of view, the cost is
significant. Especially considering that
as an administrator you can generally control the number of threads (and
therefore connections) that are used by IIS.
Thus, rather than scaling the number of processes to scale the number of
connections or threads, why not just scale the number you want directly? In the (same) literature, there are claims of
scaling improvements due to relieving contention. While it is possible for this to happen, in
most applications in the wild, the resource in question is of the kind that
requires exclusive access - as stated before this makes Web Gardens a
non-option. Also, it is not pointed out
that one of the ways contention on non-exclusively accessed resources is
decreased is by reproducing that resource in each process - again consuming
more system resources, memory in particular.
At this point you may be wondering "Why does the feature
exist?" or at least "Why are
you bothering to talk about this feature beyond saying: 'Don't Use
it'?". Oddly enough, even though
the feature shows up on the Performance tab of InetMgr, the one redeeming
feature or positive use is as a reliability tool. For example, if we follow the performance
statements from above and bump up the number of threads to increase the
concurrency to a backend dependency, our one worker process would have an
extremely large number of outstanding HTTP requests and customers. If something bad happened to the worker
process (not that we ever have anything go wrong with any worker process at
Microsoft.com), all of those requests are lost, the HTTP connections
dropped. If it happens often enough,
your availability numbers start to drop.
In this scenario (highly latent back-end that is not capacity bound,
just latent), if you suspect (i.e. know) you have unreliable code running in
your worker processes (obviously the work of some third-party, since no one in
your company would write such garbage ;^), you *might* be able to improve your
availability by trading performance and resources for it using Web
Gardens. It works because each worker
process only has 1/nth of the customers (since HTTP.SYS uses a round-robin
algorithm to assign requests to existing worker processes), so if a given
worker process dies only 1/nth the customers get their connections reset.
A better solution to consider is not Web Gardens, but multiple application
pools. It is often possible to split
applications up into parts that are independent. If this is possible, then scaling up the
number of application pools allows for the reliability wins of Web Gardens
without the negativity of resource usage duplication - though some resource usage
increase is expected, the Garbage Collection example from before still
applies. While not as simple clicking
the checkbox and inputting a number, breaking applications into multiple
application pools has better potential from increasing both Performance and
Reliability of your application. Your
caches will run cleaner, and your applications will be bounded.
In the end, while not a performance feature, Web Gardens could be used as a
stop-gap measure to help get you through a hard time while you try to improve
the reliability of the applications running in IIS. This is similar to what recycling is supposed
to do - in both cases, IIS6 helps you improve the availability of poor
applications but neither should be considered "the right solution". If you have an unreliable application, you
should be working through the issues, getting fixes from the people who own the
code. If you have highly latent
back-ends, you can work to reduce the dependencies or the amount of traffic
that is communicated to the back-end.
Obviously this kind of work takes time, so it is nice that IIS6 can help
you out a bit - however it is important to understand if you sew Brussels
sprouts don't expect to reap strawberries for your ice cream.
Published Tuesday, July 10, 2007 1:29 PM by MSCOM
Filed
under: IIS,
WEB
Gardens, Performance
Comments
#
re: Gardening (on the Web Server)
Thursday, July 19, 2007 2:02 PM by jamesbl
Another reason for web gardens (honestly the only
one I can truly stand behind, in addition to the above) is to break the 2GB x86
process barrier. I worked with a customer that was reaching the 2GB mark
every week or so, thus making the worker process to crash. The app was not using session state, so we
enabled web gardens and used 3 "flowers" (get it? flowers,
gardens...my own coined term, btw. quite nerdy I admit), which spread the load over 3
worker processes and prevented them from hitting the upper 2GB limit and
crashing.
#
re: Gardening (on the Web Server)
Thursday, June 26, 2008 6:13 AM by Tom McDonald
I like
splitting the application up idea better than web gardens, but there are
development environment drawbacks to that solution.
We split
our main web site a while back and it help overall improve site performance.
We split the app along data and presentation lines with the data part
being xml over http and asp.net xml webservices (soap).
This
improved the performance of the production site, but made development a
nightmare. On the developers' laptops (XP running VS 2005) it was extremely
difficult to mirror production.
IIS on
XP can only run one site at a time. Developers can get around this by
using Cassini and different ports, but Cassini wasn't viable because some of
the pages (on both the data and presentation sides of the split) were classic
asp pages and Cassini doesn't serve classic asp.
Bottom
line: we were were able to split the app up but only along the lines of classic
asp vs. asp.net.
Configuring Web Gardens (IIS 6.0)
To improve availability for a Web site
or application, you can increase the number of worker processes servicing the
application pool by implementing a Web garden, an application pool that
uses more than one worker process on a single computer. This might be helpful
if you have already configured a single-application application pool and need
to improve availability.
Creating a Web garden for an application
pool enhances performance by providing the following benefits:
|
• |
Robust processing of requests: When a worker process in an
application pool is tied up (for example, when a script engine stops
responding), other worker processes can accept and process requests for the
application pool. |
|
• |
Reduced contention for resources: When a Web garden reaches a steady
state, each new TCP/IP connection is assigned, according to a round-robin
scheme, to a worker process in the Web garden. This helps smooth out
workloads and reduce contention for resources that are bound to a worker
process. |
As a best practice, create Web gardens
only for Web applications that meet the following criteria:
|
• |
The application runs multi-instantiated, so that a different
instance of the application can be assigned to each worker process. |
|
• |
The Web application is not CPU-intensive. If the CPU is the
bottleneck, then adding worker processes cannot help improve performance. |
|
• |
The application is subject to synchronous high latency. For example,
if an application calls a back-end database and the response is slow, then a
Web garden supports other concurrent connections without waiting for the slow
connection to complete. |
Because Web gardens enable the use of
multiple processes, all processes have their own copy of application state,
in-process session state, caches, and static data. Web gardens are not
advantageous for applications that depend on application state. Identify the
standard relative performance in a Web garden before you decide whether Web
gardens are the best solution for your server.
A Web garden differs from a Web farm,
which distributes the load across multiple Web servers. You can design Web
farms to provide your applications with the highest availability. For more
information about Web farms, see Scalability.
By default, the Web garden feature is
disabled in IIS 6.0, and the default number of worker processes assigned
to an application pool is set to 1. You can enable Web garden functionality by
setting the number of worker processes assigned to an application pool to a
number greater than 1.
For information about how to configure a
Web garden by using IIS Manager, see Configure Application Pool Performance. For
information about how to configure a Web garden by using a command-line
procedure or scripts, see the MaxProcesses Metabase Property.
From IIS6 Programming Handbook
by Srinivasa Sivakumar, S. Vaidyaraman, Michael De Marco, Alex Homer, Christopher Ambler
p.13
Web Gardens are an especially good solution in the case of a
multi-processor machine, where the multiple worker processes may have their
processor affinity configured to run on individual processors
p.36
If the machine’s CPU is not at its maximum load, Web Gardens are a
possible solution, especially
when an application is causing a queue to back up due to request processing
taking an abnormally long amount of time (perhaps doing a very complex database
operation). In cases where the cpu is at its maximum load,
and the application cannot be further optimized, it might be time to invest in
more server hardware.
p.48
Considering the overhead involved in operations that are serialized for
locking, this method provides performance advantages.
p.48 (warning) Remember that http is a
stateless protocol. What that means is that if a connection is closed
after processing a request, there
is no guarantee that the same worker process will service future requests,
even from the same client. In cases where session state must be
maintained across different connections, the session state could be lost if a
subsequent request were sent to a different worker process. While
keeping the connection open is one possible solution, it is recommended that
state be persisted to ensure that it is available. What this means is
storing the state in an external location such as a database – for example,
asp.net provides the ability to store session state in a sql server database.
p.49
a web garden provides its best performance gains on a multi-processor
server, where each worker process in a web garden may be assigned to run on a
separate processor. For example, on a server that has eight processors, an
application pool might be configured to run on four of those processors.
SMPProcessorAffinityMask is the mask for
all processes running in a web garden. To set processor affinity, you
must set the smpaffinitized property to true, and then set the
smpprocessoraffinitymask to the range of cpu’s you wish your worker process to
be bound. In the previous example, the affinity mask would be set for the
processors numbered zero through three, and all worker processes in the
application pool would run on the first four processors.
http://msmvps.com/blogs/bernard/default.aspx
Grow a Web Garden
This can be tricky, although this will boots up your application response time,
do take note that if you are using any session variables make sure you have
out-of-process session management in place, otherwise - since all worker
processes inside the web garden do not share the memory space, having an
in-process session management will cause session detail lost if the request was
previously handle by other worker process. The following KB explains about the
session management for ASP.NET.
307598 INFO:
ASP.NET State Management Overview
http://blogs.iis.net/chrisad/archive/2006/07/14/1342059.aspx
Web
gardens was designed for one single
reason – Offering applications that are not CPU-bound but execute long
running requests the ability to scale and not use up all threads available in
the worker process. The examples might be things like -
·
Applications
that make long running database requests (e.g. high computational database
transaction)
·
Applications
that have threads occupied by long-running synchronous and network intensive
transactions
The question that you must ask yourself
-
What is the current CPU usage of the
server?
What is the application’s threads
executing and what type of requests are they?
Based on the criteria above, you should
understand better when to use Web Gardens. Web Gardens, in the metabase,
equals the MaxProcesses metabase property if you are not using the User
Interface to configure this feature.
cscript
adsutil.vbs set w3svc/apppools/defaultapppool/maxprocesses 4
I hope that I get some mileage out of
having this blog and more importantly I hope it helps you understand this
better…
Web Garden
Documentation:
New Features Improve Your Web
Server’s Performance, Reliability, and Scalability
webGarden and cpuMask Equivalent
Settings in IIS 6.0 Worker Process Isolation Mode
The webGarden attribute controls Web
garden mode, or using multiple affinitized worker processes. The cpuMask
controls the number of processes and the way the Web garden works.
Since webGarden enables the use of
multiple processes, it means that all processes will have their own copy of
things like application state, in-process session state, caches, and static
data.
When
using Web gardens, keep in mind the following:
|
• |
If your application uses
session state, it must choose an out-of-process provider (NT Service or SQL). |
|
• |
Application state and
application statics are per process, not per computer. |
|
• |
Caching is per process, not
per computer. |
Web gardens can be also set up in IIS
6.0 worker process isolation mode by increasing the maximum number of worker
processes per application pool from 1 to n (as required).
Developing High-Performance ASP.NET
Applications (.NET Framework Developer's Guide)
For applications that rely extensively
on external resources, consider enabling Web gardening on multiprocessor
computers. The ASP.NET process model helps enable scalability on multiprocessor
computers by distributing work to several processes, one per CPU, each with
processor affinity set to its CPU. This technique is called Web gardening. If
your application uses a slow database server or calls COM objects that have
external dependencies, to name only a couple of possibilities, it can be
beneficial to enable Web gardening for your application. However, you should
test how well your application performs in a Web garden before your decide to
enable this.
Enabling Web Gardens on Multiprocessor Computers
The process model helps enable scalability on multiprocessor computers by distributing the work to several processes, one per CPU, each with the processor affinity set to its own CPU. This eliminates cross-processor lock contention and is ideal for large symmetric multiprocessing (SMP) systems. This technique is called Web gardening. To enable Web gardening, you need to set the cpuMask and webGarden configuration settings in the <processModel> element… [unless specified in IIS 6] …
Web gardening has some side effects that you should be aware of:
|
• |
If your application uses session state, it must choose
an out-of-process provider (NT Service or SQL). |
|
• |
Application state and application statics are per process,
not per computer. |
|
• |
Caching is per process, not per computer. |
Q16) Web gardens cannot be configured for In Process session handling but can be
configured when using SQL server for session management. How is session information
stored in SQL server accessed when using a web garden?
A16) When using In Proc, the session data is stored in the same process as the
page. Web gardening creates multiple processes on the same server. If a users
browses and the request is handled by w3wp.exe #1, there session is stored in this
process. If they post back and the request is handled by w3wp.exe #2, the session
doesn’t exist in this process and a new session is generated. SQL server session
changes this by all the processes using the same SQL DB to store the session. When
the request comes in, the DB is accessed to retrieve the session data. This
removes the dependency on which process handles the request.
http://msdn2.microsoft.com/en-us/library/ms998549.aspx
The in-process store limits application scalability because you cannot use it in conjunction with multiple worker processes; for example, it prevents Web farm or Web garden deployment.
Web Gardens
By default, ASP.NET uses all CPUs available. In Web garden mode, ASP.NET creates one process per CPU. Each process creates an affinity to a single CPU. Web gardens offer an addition layer of reliability and robustness. If a process crashes, there are other processes that still service incoming requests.
Web gardens may perform better under the following scenarios:
Your application uses STA objects heavily.
Your application accesses a pool of resources that are bound by the number of processes. For example, a single process is restricted to using a particular number of resources.
To determine the effectiveness of Web gardens for your application, run performance tests, and then compare your results with and without Web gardens. Typically, in the two scenarios that are described in this section, you are likely to notice a greater benefit with servers that contain four or eight CPUs.
Note Do not use the in-process session state store or any technique that causes process affinity if Web gardens are enabled.
http://blogs.iis.net/webtopics/archive/2009/07/21/in-proc-session-state-management.aspx
If you are
running Web farm and using InProc Session, then this could be a possible cause
of session loss as all the Session information is stored within the Worker
process. Ensure that you have NLB with sticky sessions enabled. In order to
verify this we need to take a network trace and ensure that all the requests
are reaching the same server they originated from
http://www.hanselman.com/blog/TheASPNETCacheObjectInAWebGardeningScenarioWithIIS5OrIIS6.aspx
The ASP.NET Cache object in a Web Gardening scenario
with IIS5 or IIS6
Posted 2004-08-17 08:34 PM in ASP.NET.
Remember that "Web Gardening" is like having a tiny Web Farm on a multi-proc
box. Each processor gets an ASP.NET Worker Process with affinity (that
means the process "sticks") to that processor.
But what happens if you use the Cache object? Well,
each AppDomain gets a Cache object, and each ASP.NET Worker Process has one
AppDomain per Web Site, and each processor gets one WP, so...if you have four
processors, the Cache object will exist four times.
That means, ASP.NET Cache state is NOT shared amongst
processors in a single-box multi-proc system.
Make sure you grok this if you ever find yourself working
on a single box web site with multiple procs and web gardening on. If
you've:
coded your site to assume that the cache will fill once and only once per
machine
used physical files as CacheDependancies to key off the cache
assumed that any one user will magically return to the same process on the
same machine on subsequent requests
then things may well behave differently than you thinkin
a web gardening scenario.
Enabling Web Gardens on Multiprocessor Computers
The process model helps enable scalability on multiprocessor computers by distributing the work to several processes, one per CPU, each with the processor affinity set to its own CPU. This eliminates cross-processor lock contention and is ideal for large symmetric multiprocessing (SMP) systems. This technique is called Web gardening. To enable Web gardening, you need to set the cpuMask and webGarden configuration settings in the <processModel> element. The following example enables Web gardening on a server running ASP.NET.
<configuration>
<system.web>
<processModel
enable="true"
timeout="15"
webGarden="true"
maxWorkerThreads="25"
maxIoThreads="25"/>
</system.web>
</configuration>
Web gardening has some side effects that you should be aware of:
|
• |
If your application uses session state, it must choose an out-of-process provider (NT Service or SQL). |
|
• |
Application state and application statics are per process, not per computer. |
|
• |
Caching is per process, not per computer. |
Other comments by IIS gurus:
·
After much performance investigation, my advice
is to NOT apply web gardens. In 95%+ of the cases they do not help
performance, they hurt it. The main case where performance improves by
adding web gardens is if you have insanely latent backends which cause all
threads in a given thread pool to be blocked (causing queuing, making for
REALLY bad response times). It helps this case, strictly because more
threads can be created in a second threadpool (in the new process).
·
My advice to customers about web gardens is “do
not use unless you have a specific technical reason to do so that you have
verified benefits from using them”.
·
I would have left it at “Do Not Use”.
There are very few instances in which they help and it is rare to
find one that couldn’t be solved in different (and better) ways. . . In the
end, splitting different content up into different apppools is much more likely
to solve customer issues than using web gardens (same content in multiple
processes).
http://forums.iis.net/t/1160590.aspx
Thread: Web gardening while using
InProc
Last post
10-18-2009
By definition in
proc means in process, using a web garden means multiple processes so in proc
would be in that specific process.
An improvement
could be some formula to keep the same people in the same wp, but I would agree
generally with the why do this.
Looking at the code
for why you need so much in your session variables is probably a good answer,
but of course some apps need different things than others - just saying put
less in session vars isn't always an answer.
Depending on your
budget you may want to investigate
why you need a web garden - the advantage of a garden is if you have a single /
apartment threaded task which blocks other requests - by placing the requests
in seperate gardens you generally get around threading issues - Well designed
code shouldn't have these kind of issues though. The performance SHOULD be
better in one proc instead f the garden.
There's also an
exception on 32 bit of ram but with 64 bit w3wp that is less likely a
justification for web gardens.
http://msdn.microsoft.com/en-us/library/aa479328.aspx
The
Microsoft® ASP.NET runtime environment was architected according to design
principles that place a premium on reliability and performance. The resultant
ASP.NET process model comprises two system elements—an in-process connector
living inside the Web server process and an external worker process. In
addition, the ASP.NET runtime infrastructure is scalable enough to
automatically exploit any selected processor on multi-processor hardware. This
model, known as a Web garden, enables multiple worker processes to run at the
same time, each on a distinct processor. . .
The worker
process is a small Win32 shell of code that hosts the common language runtime
(CLR) and runs managed code. It takes care of servicing requests for ASPX,
ASMX, and ASHX resources. There is generally only one instance of this process
on a given machine. All currently active ASP.NET applications run inside of it,
each in a separate AppDomain. As mentioned earlier, though, the worker process
supports the Web garden mode, meaning that identical copies of the process run
on all of the CPUs with affinity to the process. (More on this later in
"The Web Garden Model" section.) . . .
The Web Garden
Model
The
Web garden model is configurable through the <processModel> section of
the machine.config file. Notice that the <processModel> section is the
only configuration section that cannot be placed in an application-specific
web.config file. This means that the Web garden mode applies to all
applications running on the machine. However, by using the <location>
node in the machine.config source, you can adapt machine-wide settings on a
per-application basis.
Two
attributes in the <processModel> section affect the Web garden model.
They are webGarden and cpuMask. The webGarden
attribute takes a Boolean value that indicates whether or not multiple
worker processes (one per each affinitized CPU) have to be used. The attribute
is set to false by default. The cpuMask attribute stores a
DWORD value whose binary representation provides a bit mask for the CPUs that
are eligible to run the ASP.NET worker process. The default value is -1
(0xFFFFFF), which means that all available CPUs can be used. The contents of
the cpuMask attribute is ignored when the webGarden
attribute is false. The cpuMask attribute also sets an upper
bound to the number of copies of aspnet_wp.exe that are running.
The
old motto "not everything that shines is gold" is an apt quotation
here. Web gardening enables multiple worker processes to run at the same time.
However, you should note that all processes will have their own copy of
application state, in-process session state, ASP.NET cache, static data, and
all that is needed to run applications. When the Web garden mode is enabled,
the ASP.NET ISAPI launches as many worker processes as there are CPUs, each a
full clone of the next (and each affinitized with the corresponding CPU). To
balance the workload, incoming requests are partitioned among running processes
in a round-robin manner. Worker processes get recycled as in the single
processor case. Note that ASP.NET inherits any CPU usage restriction from the
operating system and doesn’t include any custom semantics for doing this.
All
in all, the Web garden model is not necessarily a big win for all applications.
The more stateful applications are, the more they risk to pay in terms of real
performance. Working data is stored in blocks of shared memory so that any
changes entered by a process are immediately visible to others. However, for
the time it takes to service a request, working data is copied in the context
of the process. Each worker process, therefore, will handle its own copy of
working data, and the more stateful the application, the higher the cost in
performance. In this context, careful and savvy application benchmarking is an
absolute must.
Changes
made to the <processModel> section of the configuration file are
effective only after IIS is restarted. In IIS 6, Web gardening parameters are
stored in the IIS metabase; the webGarden and cpuMask
attributes are ignored.
Adding additional application pools and isolating different applciations may be a superior alternative to creating web gardens.
Here is a link for the technet article which discusses various Application Pool configurations: