I created a small program using XMLHTTP to test the performance of pdf
reports rendering. The program basically sends 20-30 XMLHTTP async
requests with callback function in the format of:
XMLHttpRequest.open "POST", <RS url access>, True
This statement was executed 20-30 times in less than a second to test
if the server can handle multiple request in a short time.
Monitoring the CPU, memory usage and generation of the actual pdf
files on the server, we noticed that the pdf reports rendering were
processed sequentially because files were generated one after another
and the CPU utilization registered a spike that hits around 50% for
each report. If we count the spikes, it matched the no. of reports
generated. No other CPU/memory intensive services are running on the
server when we perform the said test. Each generated pdf file is about
30-40k in size and took about 2-3 sec.
We understand the report rendering time varies because of its
complexity. But our concern is if this is how RS behaves, when there
are 1000 users requesting for their report, it will take years for the
1000th user to get their copy generated and displayed on their
browser.
Is my investigation correct? Anything we can do to improve the
rendering and reduce the time needed?
Thanks in advance for any assistance/comments!!!
CCCThe Report Server is no different than any other ASP.NET web application. It
runs under IIS and processes report request in multi-threaded fashion. As
usual with server-side web applications, CPU is the first resource that will
be maxed out. Therefore, beefing up your report server CPU power should come
as a no-brainer. A dual CPU box (quad recommended) would have probably made
a difference.
To estimate how well your server configuration meet the user load I would
suggest you conduct a capacity planning study. For example, assuming that
the 1,000 users will generate 20 reports per hour during normal working
hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 = 45
reports (requests)/sec. Once you establish your performance goal, you could
stress-test the server to find out its maximum throughput, e.g. by using ACT
scripts. As you could see, this is no much different than stress-testing web
applications.
If the actual server throughput is beyond your expectations, you should
first try optimizing your report performance. I found the RS execution log
indispensable for this purpose because it gives you time breakdown of how
much the Report Server has spent fetching the report data, generating and
rendering the report. In addition, don't forget caching. You can increase
substantially the server throughput by using different forms of caching. For
example, most likely in your initial tests you didn't pass the session id
with the report requests. However, if the reports are rendered by URL, the
browser will automatically pass the report session id which in many cases
could help the Report Server retrieve the report intermediate definition
without querying the data source and generating the report anew. Execution
caching will speed up things even more because now multiple users can share
the same cached report instance. Same with snapshot caching.
Finally, to achieve fault tollerance and linear scalability (to a point) you
should consider clustering multiple report servers in a web farm
environment.
--
Hope this helps.
---
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
http://www.prologika.com
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> I created a small program using XMLHTTP to test the performance of pdf
> reports rendering. The program basically sends 20-30 XMLHTTP async
> requests with callback function in the format of:
> XMLHttpRequest.open "POST", <RS url access>, True
> This statement was executed 20-30 times in less than a second to test
> if the server can handle multiple request in a short time.
> Monitoring the CPU, memory usage and generation of the actual pdf
> files on the server, we noticed that the pdf reports rendering were
> processed sequentially because files were generated one after another
> and the CPU utilization registered a spike that hits around 50% for
> each report. If we count the spikes, it matched the no. of reports
> generated. No other CPU/memory intensive services are running on the
> server when we perform the said test. Each generated pdf file is about
> 30-40k in size and took about 2-3 sec.
> We understand the report rendering time varies because of its
> complexity. But our concern is if this is how RS behaves, when there
> are 1000 users requesting for their report, it will take years for the
> 1000th user to get their copy generated and displayed on their
> browser.
> Is my investigation correct? Anything we can do to improve the
> rendering and reduce the time needed?
> Thanks in advance for any assistance/comments!!!
>
> CCC|||Hi Teo,
Thank you very much for your inputs.
Tried using ACT and created a script with 500 users to test RS but RPS
never goes more than 1.5 during the testing. The test took about 10
mins to complete and all requests were logged into ExecutionLog as
"rsSuccess".
Although it was a success run, it seems to me that RS is still
rendering the requests 1 by 1 and not in a "multi-thread manner" (but
I do believe it should run multi-thread). 10 mins to manage 500 user's
requests is unacceptable for the application I'm working on. Btw, CPU
usage never goes beyond 50% during the execution of the script with
3GB of RAM in the box.
My plan is, starts with 500 users using ACT on a server, then get 4
more server to run another 2000 users. After that, increase each
server by 100 users from 500 to 1000 and see how it goes.
Unfortunately, RPS issue and high Processing Failure counter in
Performance monitoring of RS Web Service make me feel so frustrated
and don't know what to do.
We need to address these 2 issues (RPS & high Processing Failure
counter) ASAP to continue with our performance testing. If not, the
delay in shipping of this project will affect thousands of user in the
upcoming month. I hope MS can do something to help out. Thanks!
CCC
"Teo Lachev" <teo@.nospam.prologika.com> wrote in message news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> The Report Server is no different than any other ASP.NET web application. It
> runs under IIS and processes report request in multi-threaded fashion. As
> usual with server-side web applications, CPU is the first resource that will
> be maxed out. Therefore, beefing up your report server CPU power should come
> as a no-brainer. A dual CPU box (quad recommended) would have probably made
> a difference.
> To estimate how well your server configuration meet the user load I would
> suggest you conduct a capacity planning study. For example, assuming that
> the 1,000 users will generate 20 reports per hour during normal working
> hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 = 45
> reports (requests)/sec. Once you establish your performance goal, you could
> stress-test the server to find out its maximum throughput, e.g. by using ACT
> scripts. As you could see, this is no much different than stress-testing web
> applications.
> If the actual server throughput is beyond your expectations, you should
> first try optimizing your report performance. I found the RS execution log
> indispensable for this purpose because it gives you time breakdown of how
> much the Report Server has spent fetching the report data, generating and
> rendering the report. In addition, don't forget caching. You can increase
> substantially the server throughput by using different forms of caching. For
> example, most likely in your initial tests you didn't pass the session id
> with the report requests. However, if the reports are rendered by URL, the
> browser will automatically pass the report session id which in many cases
> could help the Report Server retrieve the report intermediate definition
> without querying the data source and generating the report anew. Execution
> caching will speed up things even more because now multiple users can share
> the same cached report instance. Same with snapshot caching.
> Finally, to achieve fault tollerance and linear scalability (to a point) you
> should consider clustering multiple report servers in a web farm
> environment.
> --
> Hope this helps.
> ---
> Teo Lachev, MCSD, MCT
> Author: "Microsoft Reporting Services in Action"
> http://www.prologika.com
>
> "CCC" <tommytan25@.yahoo.com> wrote in message
> news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > I created a small program using XMLHTTP to test the performance of pdf
> > reports rendering. The program basically sends 20-30 XMLHTTP async
> > requests with callback function in the format of:
> >
> > XMLHttpRequest.open "POST", <RS url access>, True
> >
> > This statement was executed 20-30 times in less than a second to test
> > if the server can handle multiple request in a short time.
> >
> > Monitoring the CPU, memory usage and generation of the actual pdf
> > files on the server, we noticed that the pdf reports rendering were
> > processed sequentially because files were generated one after another
> > and the CPU utilization registered a spike that hits around 50% for
> > each report. If we count the spikes, it matched the no. of reports
> > generated. No other CPU/memory intensive services are running on the
> > server when we perform the said test. Each generated pdf file is about
> > 30-40k in size and took about 2-3 sec.
> >
> > We understand the report rendering time varies because of its
> > complexity. But our concern is if this is how RS behaves, when there
> > are 1000 users requesting for their report, it will take years for the
> > 1000th user to get their copy generated and displayed on their
> > browser.
> >
> > Is my investigation correct? Anything we can do to improve the
> > rendering and reduce the time needed?
> >
> > Thanks in advance for any assistance/comments!!!
> >
> >
> > CCC|||Hi,
You can observe the number of threads by monitoring the Process:Thread Count
counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn additional
threads to handle the load.
You results puzzle me. CPU doesn't go above 50% but you can't get more
requests. Do you use think time in ACT? I would start with 2 connections
then add additional connections by number of 2, e.g. 4, 8, 16 to find out
where the server throughput maxes out. Then, you need to find out what the
resource constraint is by looking at various performance counters. RS also
comes with useful performance counters, such as report requests, reports
executed/sec, etc. Total Rejected Threads is an interesting one in your
case. "Total number of data processing threads rejected for asynchronous
processing, and subsequently handled as a synchronous process in the same
thread. Each data source is processed on one thread. If the volume of
threads exceeds capacity, threads will be rejected for asynchronous
processing, and will be processed in a serial manner."
Also, did you look at the RS execution log to find out where the time goes:
querying the database, processing the report or rendering it in PDF?
--
Hope this helps.
---
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
http://www.manning.com/lachev
Home page and blog http://www.prologika.com
---
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> Hi Teo,
> Thank you very much for your inputs.
> Tried using ACT and created a script with 500 users to test RS but RPS
> never goes more than 1.5 during the testing. The test took about 10
> mins to complete and all requests were logged into ExecutionLog as
> "rsSuccess".
> Although it was a success run, it seems to me that RS is still
> rendering the requests 1 by 1 and not in a "multi-thread manner" (but
> I do believe it should run multi-thread). 10 mins to manage 500 user's
> requests is unacceptable for the application I'm working on. Btw, CPU
> usage never goes beyond 50% during the execution of the script with
> 3GB of RAM in the box.
> My plan is, starts with 500 users using ACT on a server, then get 4
> more server to run another 2000 users. After that, increase each
> server by 100 users from 500 to 1000 and see how it goes.
> Unfortunately, RPS issue and high Processing Failure counter in
> Performance monitoring of RS Web Service make me feel so frustrated
> and don't know what to do.
> We need to address these 2 issues (RPS & high Processing Failure
> counter) ASAP to continue with our performance testing. If not, the
> delay in shipping of this project will affect thousands of user in the
> upcoming month. I hope MS can do something to help out. Thanks!
> CCC
> "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > The Report Server is no different than any other ASP.NET web
application. It
> > runs under IIS and processes report request in multi-threaded fashion.
As
> > usual with server-side web applications, CPU is the first resource that
will
> > be maxed out. Therefore, beefing up your report server CPU power should
come
> > as a no-brainer. A dual CPU box (quad recommended) would have probably
made
> > a difference.
> >
> > To estimate how well your server configuration meet the user load I
would
> > suggest you conduct a capacity planning study. For example, assuming
that
> > the 1,000 users will generate 20 reports per hour during normal working
> > hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 = 45
> > reports (requests)/sec. Once you establish your performance goal, you
could
> > stress-test the server to find out its maximum throughput, e.g. by using
ACT
> > scripts. As you could see, this is no much different than stress-testing
web
> > applications.
> >
> > If the actual server throughput is beyond your expectations, you should
> > first try optimizing your report performance. I found the RS execution
log
> > indispensable for this purpose because it gives you time breakdown of
how
> > much the Report Server has spent fetching the report data, generating
and
> > rendering the report. In addition, don't forget caching. You can
increase
> > substantially the server throughput by using different forms of caching.
For
> > example, most likely in your initial tests you didn't pass the session
id
> > with the report requests. However, if the reports are rendered by URL,
the
> > browser will automatically pass the report session id which in many
cases
> > could help the Report Server retrieve the report intermediate definition
> > without querying the data source and generating the report anew.
Execution
> > caching will speed up things even more because now multiple users can
share
> > the same cached report instance. Same with snapshot caching.
> >
> > Finally, to achieve fault tollerance and linear scalability (to a point)
you
> > should consider clustering multiple report servers in a web farm
> > environment.
> >
> > --
> > Hope this helps.
> >
> > ---
> > Teo Lachev, MCSD, MCT
> > Author: "Microsoft Reporting Services in Action"
> > http://www.prologika.com
> >
> >
> > "CCC" <tommytan25@.yahoo.com> wrote in message
> > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > I created a small program using XMLHTTP to test the performance of pdf
> > > reports rendering. The program basically sends 20-30 XMLHTTP async
> > > requests with callback function in the format of:
> > >
> > > XMLHttpRequest.open "POST", <RS url access>, True
> > >
> > > This statement was executed 20-30 times in less than a second to test
> > > if the server can handle multiple request in a short time.
> > >
> > > Monitoring the CPU, memory usage and generation of the actual pdf
> > > files on the server, we noticed that the pdf reports rendering were
> > > processed sequentially because files were generated one after another
> > > and the CPU utilization registered a spike that hits around 50% for
> > > each report. If we count the spikes, it matched the no. of reports
> > > generated. No other CPU/memory intensive services are running on the
> > > server when we perform the said test. Each generated pdf file is about
> > > 30-40k in size and took about 2-3 sec.
> > >
> > > We understand the report rendering time varies because of its
> > > complexity. But our concern is if this is how RS behaves, when there
> > > are 1000 users requesting for their report, it will take years for the
> > > 1000th user to get their copy generated and displayed on their
> > > browser.
> > >
> > > Is my investigation correct? Anything we can do to improve the
> > > rendering and reduce the time needed?
> > >
> > > Thanks in advance for any assistance/comments!!!
> > >
> > >
> > > CCC|||Hi Teo,
Thanks for you comments.
Regardless of number of concurrent users I defined in ACT (1 server
with 500 users or 4 servers with 2000 users to request data from RS),
the RS's processor utilization never goes above 50-60%. They system
doesn't seems to have resource constraint but rather processing
problem. That's why I'm saying RS is managing the throughput and
processing the requests sequentially.
- 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
RPS
to complete the reports
- 4 ACT server with 500x4 concurrent connections took 16mins at 0.5-1
RPS to complete the reports
Yes there's a slight drop in the throughput but I was expecting RS to
handle more than 1 RPS for both scenarios.
I understand Total Processing Failure and Total Rejected Thread
counters were 2 different things, the first one is giving more
headache at thousands mark while the later is manageable between 5-10
for 500 concurrent users.
Checking the ExecutionLog, out of 500 reports, there are only 40-50
report with very minimal (0.05 msec) overlapped execution start and
end time. The rest were processed in sequential manner.
I posted the details of the ACT report and my script on the other
thread with the subject "Performance Testing Using ACT", please refer
to the thread for more info.
What else do I need to look at?
Really appreciate your comments, thanks!
CCC
"Teo Lachev" <teo@.nospam.prologika.com> wrote in message news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> Hi,
> You can observe the number of threads by monitoring the Process:Thread Count
> counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn additional
> threads to handle the load.
> You results puzzle me. CPU doesn't go above 50% but you can't get more
> requests. Do you use think time in ACT? I would start with 2 connections
> then add additional connections by number of 2, e.g. 4, 8, 16 to find out
> where the server throughput maxes out. Then, you need to find out what the
> resource constraint is by looking at various performance counters. RS also
> comes with useful performance counters, such as report requests, reports
> executed/sec, etc. Total Rejected Threads is an interesting one in your
> case. "Total number of data processing threads rejected for asynchronous
> processing, and subsequently handled as a synchronous process in the same
> thread. Each data source is processed on one thread. If the volume of
> threads exceeds capacity, threads will be rejected for asynchronous
> processing, and will be processed in a serial manner."
> Also, did you look at the RS execution log to find out where the time goes:
> querying the database, processing the report or rendering it in PDF?
> --
> Hope this helps.
> ---
> Teo Lachev, MCSD, MCT
> Author: "Microsoft Reporting Services in Action"
> http://www.manning.com/lachev
> Home page and blog http://www.prologika.com
> ---
> "CCC" <tommytan25@.yahoo.com> wrote in message
> news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > Hi Teo,
> >
> > Thank you very much for your inputs.
> >
> > Tried using ACT and created a script with 500 users to test RS but RPS
> > never goes more than 1.5 during the testing. The test took about 10
> > mins to complete and all requests were logged into ExecutionLog as
> > "rsSuccess".
> >
> > Although it was a success run, it seems to me that RS is still
> > rendering the requests 1 by 1 and not in a "multi-thread manner" (but
> > I do believe it should run multi-thread). 10 mins to manage 500 user's
> > requests is unacceptable for the application I'm working on. Btw, CPU
> > usage never goes beyond 50% during the execution of the script with
> > 3GB of RAM in the box.
> >
> > My plan is, starts with 500 users using ACT on a server, then get 4
> > more server to run another 2000 users. After that, increase each
> > server by 100 users from 500 to 1000 and see how it goes.
> > Unfortunately, RPS issue and high Processing Failure counter in
> > Performance monitoring of RS Web Service make me feel so frustrated
> > and don't know what to do.
> >
> > We need to address these 2 issues (RPS & high Processing Failure
> > counter) ASAP to continue with our performance testing. If not, the
> > delay in shipping of this project will affect thousands of user in the
> > upcoming month. I hope MS can do something to help out. Thanks!
> >
> > CCC
> >
> > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > The Report Server is no different than any other ASP.NET web
> application. It
> > > runs under IIS and processes report request in multi-threaded fashion.
> As
> > > usual with server-side web applications, CPU is the first resource that
> will
> > > be maxed out. Therefore, beefing up your report server CPU power should
> come
> > > as a no-brainer. A dual CPU box (quad recommended) would have probably
> made
> > > a difference.
> > >
> > > To estimate how well your server configuration meet the user load I
> would
> > > suggest you conduct a capacity planning study. For example, assuming
> that
> > > the 1,000 users will generate 20 reports per hour during normal working
> > > hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 = 45
> > > reports (requests)/sec. Once you establish your performance goal, you
> could
> > > stress-test the server to find out its maximum throughput, e.g. by using
> ACT
> > > scripts. As you could see, this is no much different than stress-testing
> web
> > > applications.
> > >
> > > If the actual server throughput is beyond your expectations, you should
> > > first try optimizing your report performance. I found the RS execution
> log
> > > indispensable for this purpose because it gives you time breakdown of
> how
> > > much the Report Server has spent fetching the report data, generating
> and
> > > rendering the report. In addition, don't forget caching. You can
> increase
> > > substantially the server throughput by using different forms of caching.
> For
> > > example, most likely in your initial tests you didn't pass the session
> id
> > > with the report requests. However, if the reports are rendered by URL,
> the
> > > browser will automatically pass the report session id which in many
> cases
> > > could help the Report Server retrieve the report intermediate definition
> > > without querying the data source and generating the report anew.
> Execution
> > > caching will speed up things even more because now multiple users can
> share
> > > the same cached report instance. Same with snapshot caching.
> > >
> > > Finally, to achieve fault tollerance and linear scalability (to a point)
> you
> > > should consider clustering multiple report servers in a web farm
> > > environment.
> > >
> > > --
> > > Hope this helps.
> > >
> > > ---
> > > Teo Lachev, MCSD, MCT
> > > Author: "Microsoft Reporting Services in Action"
> > > http://www.prologika.com
> > >
> > >
> > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > I created a small program using XMLHTTP to test the performance of pdf
> > > > reports rendering. The program basically sends 20-30 XMLHTTP async
> > > > requests with callback function in the format of:
> > > >
> > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > >
> > > > This statement was executed 20-30 times in less than a second to test
> > > > if the server can handle multiple request in a short time.
> > > >
> > > > Monitoring the CPU, memory usage and generation of the actual pdf
> > > > files on the server, we noticed that the pdf reports rendering were
> > > > processed sequentially because files were generated one after another
> > > > and the CPU utilization registered a spike that hits around 50% for
> > > > each report. If we count the spikes, it matched the no. of reports
> > > > generated. No other CPU/memory intensive services are running on the
> > > > server when we perform the said test. Each generated pdf file is about
> > > > 30-40k in size and took about 2-3 sec.
> > > >
> > > > We understand the report rendering time varies because of its
> > > > complexity. But our concern is if this is how RS behaves, when there
> > > > are 1000 users requesting for their report, it will take years for the
> > > > 1000th user to get their copy generated and displayed on their
> > > > browser.
> > > >
> > > > Is my investigation correct? Anything we can do to improve the
> > > > rendering and reduce the time needed?
> > > >
> > > > Thanks in advance for any assistance/comments!!!
> > > >
> > > >
> > > > CCC|||OK, tell me a bit more about your ACT testing methodology. First, which
version of ACT are you using? How do you set up 4 servers with 2000 users?
Second, why do you start with 500 concurrent connections? Please note that
500 users means that ACT will try to create a huge number of background
threads on the ACT client which may be what is causing the delay. What is
your CPU utilization on the client machine?
As I said, I would establish a performance metrics in requests/sec. Then, i
will pump up the server load by factor of 2, but I won't start with 500
users. This is in no way equivalent to 500 users requesting reports because
you have to factor in the user think time. Intead, I will start with 1, 2,
4, 8, etc. until I reach the troughput peak. Could you do this and let me
know at what point the Report Server throughput peaks.
Does it make sense?
--
Hope this helps.
---
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
http://www.manning.com/lachev
Home page and blog http://www.prologika.com
---
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408241732.730edd8b@.posting.google.com...
> Hi Teo,
> Thanks for you comments.
> Regardless of number of concurrent users I defined in ACT (1 server
> with 500 users or 4 servers with 2000 users to request data from RS),
> the RS's processor utilization never goes above 50-60%. They system
> doesn't seems to have resource constraint but rather processing
> problem. That's why I'm saying RS is managing the throughput and
> processing the requests sequentially.
> - 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
> RPS
> to complete the reports
> - 4 ACT server with 500x4 concurrent connections took 16mins at 0.5-1
> RPS to complete the reports
> Yes there's a slight drop in the throughput but I was expecting RS to
> handle more than 1 RPS for both scenarios.
> I understand Total Processing Failure and Total Rejected Thread
> counters were 2 different things, the first one is giving more
> headache at thousands mark while the later is manageable between 5-10
> for 500 concurrent users.
> Checking the ExecutionLog, out of 500 reports, there are only 40-50
> report with very minimal (0.05 msec) overlapped execution start and
> end time. The rest were processed in sequential manner.
> I posted the details of the ACT report and my script on the other
> thread with the subject "Performance Testing Using ACT", please refer
> to the thread for more info.
> What else do I need to look at?
> Really appreciate your comments, thanks!
> CCC
> "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> > Hi,
> >
> > You can observe the number of threads by monitoring the Process:Thread
Count
> > counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn additional
> > threads to handle the load.
> >
> > You results puzzle me. CPU doesn't go above 50% but you can't get more
> > requests. Do you use think time in ACT? I would start with 2 connections
> > then add additional connections by number of 2, e.g. 4, 8, 16 to find
out
> > where the server throughput maxes out. Then, you need to find out what
the
> > resource constraint is by looking at various performance counters. RS
also
> > comes with useful performance counters, such as report requests, reports
> > executed/sec, etc. Total Rejected Threads is an interesting one in your
> > case. "Total number of data processing threads rejected for asynchronous
> > processing, and subsequently handled as a synchronous process in the
same
> > thread. Each data source is processed on one thread. If the volume of
> > threads exceeds capacity, threads will be rejected for asynchronous
> > processing, and will be processed in a serial manner."
> >
> > Also, did you look at the RS execution log to find out where the time
goes:
> > querying the database, processing the report or rendering it in PDF?
> >
> > --
> > Hope this helps.
> >
> > ---
> > Teo Lachev, MCSD, MCT
> > Author: "Microsoft Reporting Services in Action"
> > http://www.manning.com/lachev
> > Home page and blog http://www.prologika.com
> > ---
> >
> > "CCC" <tommytan25@.yahoo.com> wrote in message
> > news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > > Hi Teo,
> > >
> > > Thank you very much for your inputs.
> > >
> > > Tried using ACT and created a script with 500 users to test RS but RPS
> > > never goes more than 1.5 during the testing. The test took about 10
> > > mins to complete and all requests were logged into ExecutionLog as
> > > "rsSuccess".
> > >
> > > Although it was a success run, it seems to me that RS is still
> > > rendering the requests 1 by 1 and not in a "multi-thread manner" (but
> > > I do believe it should run multi-thread). 10 mins to manage 500 user's
> > > requests is unacceptable for the application I'm working on. Btw, CPU
> > > usage never goes beyond 50% during the execution of the script with
> > > 3GB of RAM in the box.
> > >
> > > My plan is, starts with 500 users using ACT on a server, then get 4
> > > more server to run another 2000 users. After that, increase each
> > > server by 100 users from 500 to 1000 and see how it goes.
> > > Unfortunately, RPS issue and high Processing Failure counter in
> > > Performance monitoring of RS Web Service make me feel so frustrated
> > > and don't know what to do.
> > >
> > > We need to address these 2 issues (RPS & high Processing Failure
> > > counter) ASAP to continue with our performance testing. If not, the
> > > delay in shipping of this project will affect thousands of user in the
> > > upcoming month. I hope MS can do something to help out. Thanks!
> > >
> > > CCC
> > >
> > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> > news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > > The Report Server is no different than any other ASP.NET web
> > application. It
> > > > runs under IIS and processes report request in multi-threaded
fashion.
> > As
> > > > usual with server-side web applications, CPU is the first resource
that
> > will
> > > > be maxed out. Therefore, beefing up your report server CPU power
should
> > come
> > > > as a no-brainer. A dual CPU box (quad recommended) would have
probably
> > made
> > > > a difference.
> > > >
> > > > To estimate how well your server configuration meet the user load I
> > would
> > > > suggest you conduct a capacity planning study. For example, assuming
> > that
> > > > the 1,000 users will generate 20 reports per hour during normal
working
> > > > hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 =45
> > > > reports (requests)/sec. Once you establish your performance goal,
you
> > could
> > > > stress-test the server to find out its maximum throughput, e.g. by
using
> > ACT
> > > > scripts. As you could see, this is no much different than
stress-testing
> > web
> > > > applications.
> > > >
> > > > If the actual server throughput is beyond your expectations, you
should
> > > > first try optimizing your report performance. I found the RS
execution
> > log
> > > > indispensable for this purpose because it gives you time breakdown
of
> > how
> > > > much the Report Server has spent fetching the report data,
generating
> > and
> > > > rendering the report. In addition, don't forget caching. You can
> > increase
> > > > substantially the server throughput by using different forms of
caching.
> > For
> > > > example, most likely in your initial tests you didn't pass the
session
> > id
> > > > with the report requests. However, if the reports are rendered by
URL,
> > the
> > > > browser will automatically pass the report session id which in many
> > cases
> > > > could help the Report Server retrieve the report intermediate
definition
> > > > without querying the data source and generating the report anew.
> > Execution
> > > > caching will speed up things even more because now multiple users
can
> > share
> > > > the same cached report instance. Same with snapshot caching.
> > > >
> > > > Finally, to achieve fault tollerance and linear scalability (to a
point)
> > you
> > > > should consider clustering multiple report servers in a web farm
> > > > environment.
> > > >
> > > > --
> > > > Hope this helps.
> > > >
> > > > ---
> > > > Teo Lachev, MCSD, MCT
> > > > Author: "Microsoft Reporting Services in Action"
> > > > http://www.prologika.com
> > > >
> > > >
> > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > > I created a small program using XMLHTTP to test the performance of
pdf
> > > > > reports rendering. The program basically sends 20-30 XMLHTTP async
> > > > > requests with callback function in the format of:
> > > > >
> > > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > > >
> > > > > This statement was executed 20-30 times in less than a second to
test
> > > > > if the server can handle multiple request in a short time.
> > > > >
> > > > > Monitoring the CPU, memory usage and generation of the actual pdf
> > > > > files on the server, we noticed that the pdf reports rendering
were
> > > > > processed sequentially because files were generated one after
another
> > > > > and the CPU utilization registered a spike that hits around 50%
for
> > > > > each report. If we count the spikes, it matched the no. of reports
> > > > > generated. No other CPU/memory intensive services are running on
the
> > > > > server when we perform the said test. Each generated pdf file is
about
> > > > > 30-40k in size and took about 2-3 sec.
> > > > >
> > > > > We understand the report rendering time varies because of its
> > > > > complexity. But our concern is if this is how RS behaves, when
there
> > > > > are 1000 users requesting for their report, it will take years for
the
> > > > > 1000th user to get their copy generated and displayed on their
> > > > > browser.
> > > > >
> > > > > Is my investigation correct? Anything we can do to improve the
> > > > > rendering and reduce the time needed?
> > > > >
> > > > > Thanks in advance for any assistance/comments!!!
> > > > >
> > > > >
> > > > > CCC|||Hi Teo,
The version of ACT is 1.0.536.0 that comes with my VS.NET Enterprise.
I have 4 servers with ACT installed:
* Server A - P4 1.8 GHz & 512RAM
* Servers B, C & D - 4 x Xeon 2.8 GHz & 4GB RAM
* RS Server - 2 x Xeon 3.06GHz & 3GB RAM (abc.xyz.com)
* Each serve has T1 connection to the internet
For each server, I created a test script containing only 1 line:
Test.SendRequest("http://abc.xyz.com/ReportServer?/StudentReportCard&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false&UserID=3232")
I tried running 5 separate tests using only Server B by setting the
simultaneous browser connections in test properties to 20, 100, 150,
200 & 500 (also specify the test duration run time to 3 mins w/ 0
warmup time). During the entire 5 tests, RPS always stay at 2-3 RPS
(CPU = 90 to 100%) until the tests were ended. I also tried lowering
the number of concurrent users as what you said (1, 2, 4, 8, 10, 20),
the RPS never changed, it remains between 2-3 RPS regardless of the
number of users specified. This is what I got from one of the test
with 2 concurrent users defined:
===========================================================Properties
Test type: Dynamic
Simultaneous browser connections: 2
Warm up time (secs): 0
Test duration: 00:00:03:00
Test iterations: 270
Detailed test results generated: Yes
Summary
Total number of requests: 270
Total number of connections: 270
Average requests per second: 1.50
Average time to first byte (msecs): 822.73
Average time to last byte (msecs): 1,281.97
Average time to last byte per iteration (msecs): 1,281.97
Number of unique requests made in test: 1
Number of unique response codes: 1
Errors Counts
HTTP: 0
DNS: 0
Socket: 0
Additional Network Statistics
Average bandwidth (bytes/sec): 107,324.83
Number of bytes sent (bytes): 120,490
Number of bytes received (bytes): 19,197,980
Average rate of sent bytes (bytes/sec): 669.39
Average rate of received bytes (bytes/sec): 106,655.44
Number of connection errors: 0
Number of send errors: 0
Number of receive errors: 0
Number of timeout errors: 0
Response Codes
Response Code: 200 - The request completed successfully.
Count: 270
Percent (%): 100.00
===========================================================
With 2 concurrent users defined in ACT, RS Server's CPU utilization is
around 50%. The throughput reached 100% when users were increased to
4. I tried another simple login.asp link of another project on a
server over internet (not a RS server/site) with 2000 users using
server B and I was able to get 250+ RPS. The only thing I can think of
is RS or my report requires more resources or is trying to manage the
requests that comes in. That's why regardless of number of users, it
only allow 2-3 RPS to avoid bug down. Or, simply some DoS prevention
feature is in place that limit the requests to be processed. Comments?
The problem here is we were not able to even establish performance
metric we need (reducing no. of users to achieve at least 100 RPS).
That's why I was thinking if there's anything I need to look at to
increase the RPS before I can proceed with further testing.
I will try to test the RS sample reports and see if the behaviors are
the same...
CCC
"Teo Lachev" <teo@.nospam.prologika.com> wrote in message news:<eJaAHxpiEHA.536@.TK2MSFTNGP11.phx.gbl>...
> OK, tell me a bit more about your ACT testing methodology. First, which
> version of ACT are you using? How do you set up 4 servers with 2000 users?
> Second, why do you start with 500 concurrent connections? Please note that
> 500 users means that ACT will try to create a huge number of background
> threads on the ACT client which may be what is causing the delay. What is
> your CPU utilization on the client machine?
> As I said, I would establish a performance metrics in requests/sec. Then, i
> will pump up the server load by factor of 2, but I won't start with 500
> users. This is in no way equivalent to 500 users requesting reports because
> you have to factor in the user think time. Intead, I will start with 1, 2,
> 4, 8, etc. until I reach the troughput peak. Could you do this and let me
> know at what point the Report Server throughput peaks.
> Does it make sense?
> --
> Hope this helps.
> ---
> Teo Lachev, MCSD, MCT
> Author: "Microsoft Reporting Services in Action"
> http://www.manning.com/lachev
> Home page and blog http://www.prologika.com
> ---
> "CCC" <tommytan25@.yahoo.com> wrote in message
> news:ab9130f9.0408241732.730edd8b@.posting.google.com...
> > Hi Teo,
> >
> > Thanks for you comments.
> >
> > Regardless of number of concurrent users I defined in ACT (1 server
> > with 500 users or 4 servers with 2000 users to request data from RS),
> > the RS's processor utilization never goes above 50-60%. They system
> > doesn't seems to have resource constraint but rather processing
> > problem. That's why I'm saying RS is managing the throughput and
> > processing the requests sequentially.
> >
> > - 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
> > RPS
> > to complete the reports
> >
> > - 4 ACT server with 500x4 concurrent connections took 16mins at 0.5-1
> > RPS to complete the reports
> >
> > Yes there's a slight drop in the throughput but I was expecting RS to
> > handle more than 1 RPS for both scenarios.
> >
> > I understand Total Processing Failure and Total Rejected Thread
> > counters were 2 different things, the first one is giving more
> > headache at thousands mark while the later is manageable between 5-10
> > for 500 concurrent users.
> >
> > Checking the ExecutionLog, out of 500 reports, there are only 40-50
> > report with very minimal (0.05 msec) overlapped execution start and
> > end time. The rest were processed in sequential manner.
> >
> > I posted the details of the ACT report and my script on the other
> > thread with the subject "Performance Testing Using ACT", please refer
> > to the thread for more info.
> >
> > What else do I need to look at?
> >
> > Really appreciate your comments, thanks!
> >
> > CCC
> >
> > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> > > Hi,
> > >
> > > You can observe the number of threads by monitoring the Process:Thread
> Count
> > > counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn additional
> > > threads to handle the load.
> > >
> > > You results puzzle me. CPU doesn't go above 50% but you can't get more
> > > requests. Do you use think time in ACT? I would start with 2 connections
> > > then add additional connections by number of 2, e.g. 4, 8, 16 to find
> out
> > > where the server throughput maxes out. Then, you need to find out what
> the
> > > resource constraint is by looking at various performance counters. RS
> also
> > > comes with useful performance counters, such as report requests, reports
> > > executed/sec, etc. Total Rejected Threads is an interesting one in your
> > > case. "Total number of data processing threads rejected for asynchronous
> > > processing, and subsequently handled as a synchronous process in the
> same
> > > thread. Each data source is processed on one thread. If the volume of
> > > threads exceeds capacity, threads will be rejected for asynchronous
> > > processing, and will be processed in a serial manner."
> > >
> > > Also, did you look at the RS execution log to find out where the time
> goes:
> > > querying the database, processing the report or rendering it in PDF?
> > >
> > > --
> > > Hope this helps.
> > >
> > > ---
> > > Teo Lachev, MCSD, MCT
> > > Author: "Microsoft Reporting Services in Action"
> > > http://www.manning.com/lachev
> > > Home page and blog http://www.prologika.com
> > > ---
> > >
> > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > > > Hi Teo,
> > > >
> > > > Thank you very much for your inputs.
> > > >
> > > > Tried using ACT and created a script with 500 users to test RS but RPS
> > > > never goes more than 1.5 during the testing. The test took about 10
> > > > mins to complete and all requests were logged into ExecutionLog as
> > > > "rsSuccess".
> > > >
> > > > Although it was a success run, it seems to me that RS is still
> > > > rendering the requests 1 by 1 and not in a "multi-thread manner" (but
> > > > I do believe it should run multi-thread). 10 mins to manage 500 user's
> > > > requests is unacceptable for the application I'm working on. Btw, CPU
> > > > usage never goes beyond 50% during the execution of the script with
> > > > 3GB of RAM in the box.
> > > >
> > > > My plan is, starts with 500 users using ACT on a server, then get 4
> > > > more server to run another 2000 users. After that, increase each
> > > > server by 100 users from 500 to 1000 and see how it goes.
> > > > Unfortunately, RPS issue and high Processing Failure counter in
> > > > Performance monitoring of RS Web Service make me feel so frustrated
> > > > and don't know what to do.
> > > >
> > > > We need to address these 2 issues (RPS & high Processing Failure
> > > > counter) ASAP to continue with our performance testing. If not, the
> > > > delay in shipping of this project will affect thousands of user in the
> > > > upcoming month. I hope MS can do something to help out. Thanks!
> > > >
> > > > CCC
> > > >
> > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > > > The Report Server is no different than any other ASP.NET web
> application. It
> > > > > runs under IIS and processes report request in multi-threaded
> fashion.
> As
> > > > > usual with server-side web applications, CPU is the first resource
> that
> will
> > > > > be maxed out. Therefore, beefing up your report server CPU power
> should
> come
> > > > > as a no-brainer. A dual CPU box (quad recommended) would have
> probably
> made
> > > > > a difference.
> > > > >
> > > > > To estimate how well your server configuration meet the user load I
> would
> > > > > suggest you conduct a capacity planning study. For example, assuming
> that
> > > > > the 1,000 users will generate 20 reports per hour during normal
> working
> > > > > hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 => 45
> > > > > reports (requests)/sec. Once you establish your performance goal,
> you
> could
> > > > > stress-test the server to find out its maximum throughput, e.g. by
> using
> ACT
> > > > > scripts. As you could see, this is no much different than
> stress-testing
> web
> > > > > applications.
> > > > >
> > > > > If the actual server throughput is beyond your expectations, you
> should
> > > > > first try optimizing your report performance. I found the RS
> execution
> log
> > > > > indispensable for this purpose because it gives you time breakdown
> of
> how
> > > > > much the Report Server has spent fetching the report data,
> generating
> and
> > > > > rendering the report. In addition, don't forget caching. You can
> increase
> > > > > substantially the server throughput by using different forms of
> caching.
> For
> > > > > example, most likely in your initial tests you didn't pass the
> session
> id
> > > > > with the report requests. However, if the reports are rendered by
> URL,
> the
> > > > > browser will automatically pass the report session id which in many
> cases
> > > > > could help the Report Server retrieve the report intermediate
> definition
> > > > > without querying the data source and generating the report anew.
> Execution
> > > > > caching will speed up things even more because now multiple users
> can
> share
> > > > > the same cached report instance. Same with snapshot caching.
> > > > >
> > > > > Finally, to achieve fault tollerance and linear scalability (to a
> point)
> you
> > > > > should consider clustering multiple report servers in a web farm
> > > > > environment.
> > > > >
> > > > > --
> > > > > Hope this helps.
> > > > >
> > > > > ---
> > > > > Teo Lachev, MCSD, MCT
> > > > > Author: "Microsoft Reporting Services in Action"
> > > > > http://www.prologika.com
> > > > >
> > > > >
> > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > > > I created a small program using XMLHTTP to test the performance of
> pdf
> > > > > > reports rendering. The program basically sends 20-30 XMLHTTP async
> > > > > > requests with callback function in the format of:
> > > > > >
> > > > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > > > >
> > > > > > This statement was executed 20-30 times in less than a second to
> test
> > > > > > if the server can handle multiple request in a short time.
> > > > > >
> > > > > > Monitoring the CPU, memory usage and generation of the actual pdf
> > > > > > files on the server, we noticed that the pdf reports rendering
> were
> > > > > > processed sequentially because files were generated one after
> another
> > > > > > and the CPU utilization registered a spike that hits around 50%
> for
> > > > > > each report. If we count the spikes, it matched the no. of reports
> > > > > > generated. No other CPU/memory intensive services are running on
> the
> > > > > > server when we perform the said test. Each generated pdf file is
> about
> > > > > > 30-40k in size and took about 2-3 sec.
> > > > > >
> > > > > > We understand the report rendering time varies because of its
> > > > > > complexity. But our concern is if this is how RS behaves, when
> there
> > > > > > are 1000 users requesting for their report, it will take years for
> the
> > > > > > 1000th user to get their copy generated and displayed on their
> > > > > > browser.
> > > > > >
> > > > > > Is my investigation correct? Anything we can do to improve the
> > > > > > rendering and reduce the time needed?
> > > > > >
> > > > > > Thanks in advance for any assistance/comments!!!
> > > > > >
> > > > > >
> > > > > > CCC|||Just tried using sample reports included in the RS installer, still
the same behavior.
============================================================URL - Test.SendRequest("http://abc.xyz.com/ReportServer?/company%20sales&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false")
No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
Test duration run time: 3 mins.
1 User -
RS Server CPU Utilization: 50-60%
Ave. RPS: 1.07
2 Users -
RS Server CPU Utilization: 70-80% (Drop to 30-40% for the last 0.5
min)
Ave. RPS: 2.43 (1-1.5 RPS for the first 2.5 mins, then jump to 8-9 RPS
for the remaining 0.5 mins)
4 Users -
RS Server CPU Utilization: 95-100% (Drop to 30-40% for the last 20
sec.)
Ave. RPS: 3.03 (1-1.5 RPS for the first 2.7 mins, then jump to 16-17
RPS for the remaining 20 sec)
============================================================URL - Test.SendRequest("http://staging.atitesting.com/ReportServer?/Product%20Catalog&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false")
No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
Test duration run time: 3 mins.
1 User -
RS Server CPU Utilization: 7-40% (Up and down for the entire 3 mins.)
Ave. RPS: 0.26 (0.5-1 RPS)
2 Users -
RS Server CPU Utilization: 20-70% (Up and down for the entire 3 mins.)
Ave. RPS: 0.41 (0.5-1 RPS)
4 Users -
RS Server CPU Utilization: 20-100% (Up and down for the entire 3
mins.)
Ave. RPS: 0.53 (0.5-1.5 RPS)
Can anybody tell me why is it like that' Hello Microsoft'
CCC
tommytan25@.yahoo.com (CCC) wrote in message news:<ab9130f9.0408251851.75145c67@.posting.google.com>...
> Hi Teo,
> The version of ACT is 1.0.536.0 that comes with my VS.NET Enterprise.
> I have 4 servers with ACT installed:
> * Server A - P4 1.8 GHz & 512RAM
> * Servers B, C & D - 4 x Xeon 2.8 GHz & 4GB RAM
> * RS Server - 2 x Xeon 3.06GHz & 3GB RAM (abc.xyz.com)
> * Each serve has T1 connection to the internet
>
> For each server, I created a test script containing only 1 line:
> Test.SendRequest("http://abc.xyz.com/ReportServer?/StudentReportCard&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false&UserID=3232")
> I tried running 5 separate tests using only Server B by setting the
> simultaneous browser connections in test properties to 20, 100, 150,
> 200 & 500 (also specify the test duration run time to 3 mins w/ 0
> warmup time). During the entire 5 tests, RPS always stay at 2-3 RPS
> (CPU = 90 to 100%) until the tests were ended. I also tried lowering
> the number of concurrent users as what you said (1, 2, 4, 8, 10, 20),
> the RPS never changed, it remains between 2-3 RPS regardless of the
> number of users specified. This is what I got from one of the test
> with 2 concurrent users defined:
> ===========================================================> Properties
> Test type: Dynamic
> Simultaneous browser connections: 2
> Warm up time (secs): 0
> Test duration: 00:00:03:00
> Test iterations: 270
> Detailed test results generated: Yes
> Summary
> Total number of requests: 270
> Total number of connections: 270
> Average requests per second: 1.50
> Average time to first byte (msecs): 822.73
> Average time to last byte (msecs): 1,281.97
> Average time to last byte per iteration (msecs): 1,281.97
> Number of unique requests made in test: 1
> Number of unique response codes: 1
> Errors Counts
> HTTP: 0
> DNS: 0
> Socket: 0
> Additional Network Statistics
> Average bandwidth (bytes/sec): 107,324.83
> Number of bytes sent (bytes): 120,490
> Number of bytes received (bytes): 19,197,980
> Average rate of sent bytes (bytes/sec): 669.39
> Average rate of received bytes (bytes/sec): 106,655.44
> Number of connection errors: 0
> Number of send errors: 0
> Number of receive errors: 0
> Number of timeout errors: 0
> Response Codes
> Response Code: 200 - The request completed successfully.
> Count: 270
> Percent (%): 100.00
> ===========================================================> With 2 concurrent users defined in ACT, RS Server's CPU utilization is
> around 50%. The throughput reached 100% when users were increased to
> 4. I tried another simple login.asp link of another project on a
> server over internet (not a RS server/site) with 2000 users using
> server B and I was able to get 250+ RPS. The only thing I can think of
> is RS or my report requires more resources or is trying to manage the
> requests that comes in. That's why regardless of number of users, it
> only allow 2-3 RPS to avoid bug down. Or, simply some DoS prevention
> feature is in place that limit the requests to be processed. Comments?
> The problem here is we were not able to even establish performance
> metric we need (reducing no. of users to achieve at least 100 RPS).
> That's why I was thinking if there's anything I need to look at to
> increase the RPS before I can proceed with further testing.
> I will try to test the RS sample reports and see if the behaviors are
> the same...
> CCC
>
> "Teo Lachev" <teo@.nospam.prologika.com> wrote in message news:<eJaAHxpiEHA.536@.TK2MSFTNGP11.phx.gbl>...
> > OK, tell me a bit more about your ACT testing methodology. First, which
> > version of ACT are you using? How do you set up 4 servers with 2000 users?
> > Second, why do you start with 500 concurrent connections? Please note that
> > 500 users means that ACT will try to create a huge number of background
> > threads on the ACT client which may be what is causing the delay. What is
> > your CPU utilization on the client machine?
> >
> > As I said, I would establish a performance metrics in requests/sec. Then, i
> > will pump up the server load by factor of 2, but I won't start with 500
> > users. This is in no way equivalent to 500 users requesting reports because
> > you have to factor in the user think time. Intead, I will start with 1, 2,
> > 4, 8, etc. until I reach the troughput peak. Could you do this and let me
> > know at what point the Report Server throughput peaks.
> >
> > Does it make sense?
> >
> > --
> > Hope this helps.
> >
> > ---
> > Teo Lachev, MCSD, MCT
> > Author: "Microsoft Reporting Services in Action"
> > http://www.manning.com/lachev
> > Home page and blog http://www.prologika.com
> > ---
> >
> > "CCC" <tommytan25@.yahoo.com> wrote in message
> > news:ab9130f9.0408241732.730edd8b@.posting.google.com...
> > > Hi Teo,
> > >
> > > Thanks for you comments.
> > >
> > > Regardless of number of concurrent users I defined in ACT (1 server
> > > with 500 users or 4 servers with 2000 users to request data from RS),
> > > the RS's processor utilization never goes above 50-60%. They system
> > > doesn't seems to have resource constraint but rather processing
> > > problem. That's why I'm saying RS is managing the throughput and
> > > processing the requests sequentially.
> > >
> > > - 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
> > > RPS
> > > to complete the reports
> > >
> > > - 4 ACT server with 500x4 concurrent connections took 16mins at 0.5-1
> > > RPS to complete the reports
> > >
> > > Yes there's a slight drop in the throughput but I was expecting RS to
> > > handle more than 1 RPS for both scenarios.
> > >
> > > I understand Total Processing Failure and Total Rejected Thread
> > > counters were 2 different things, the first one is giving more
> > > headache at thousands mark while the later is manageable between 5-10
> > > for 500 concurrent users.
> > >
> > > Checking the ExecutionLog, out of 500 reports, there are only 40-50
> > > report with very minimal (0.05 msec) overlapped execution start and
> > > end time. The rest were processed in sequential manner.
> > >
> > > I posted the details of the ACT report and my script on the other
> > > thread with the subject "Performance Testing Using ACT", please refer
> > > to the thread for more info.
> > >
> > > What else do I need to look at?
> > >
> > > Really appreciate your comments, thanks!
> > >
> > > CCC
> > >
> > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> > > > Hi,
> > > >
> > > > You can observe the number of threads by monitoring the Process:Thread
> Count
> > > > counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn additional
> > > > threads to handle the load.
> > > >
> > > > You results puzzle me. CPU doesn't go above 50% but you can't get more
> > > > requests. Do you use think time in ACT? I would start with 2 connections
> > > > then add additional connections by number of 2, e.g. 4, 8, 16 to find
> out
> > > > where the server throughput maxes out. Then, you need to find out what
> the
> > > > resource constraint is by looking at various performance counters. RS
> also
> > > > comes with useful performance counters, such as report requests, reports
> > > > executed/sec, etc. Total Rejected Threads is an interesting one in your
> > > > case. "Total number of data processing threads rejected for asynchronous
> > > > processing, and subsequently handled as a synchronous process in the
> same
> > > > thread. Each data source is processed on one thread. If the volume of
> > > > threads exceeds capacity, threads will be rejected for asynchronous
> > > > processing, and will be processed in a serial manner."
> > > >
> > > > Also, did you look at the RS execution log to find out where the time
> goes:
> > > > querying the database, processing the report or rendering it in PDF?
> > > >
> > > > --
> > > > Hope this helps.
> > > >
> > > > ---
> > > > Teo Lachev, MCSD, MCT
> > > > Author: "Microsoft Reporting Services in Action"
> > > > http://www.manning.com/lachev
> > > > Home page and blog http://www.prologika.com
> > > > ---
> > > >
> > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > > > > Hi Teo,
> > > > >
> > > > > Thank you very much for your inputs.
> > > > >
> > > > > Tried using ACT and created a script with 500 users to test RS but RPS
> > > > > never goes more than 1.5 during the testing. The test took about 10
> > > > > mins to complete and all requests were logged into ExecutionLog as
> > > > > "rsSuccess".
> > > > >
> > > > > Although it was a success run, it seems to me that RS is still
> > > > > rendering the requests 1 by 1 and not in a "multi-thread manner" (but
> > > > > I do believe it should run multi-thread). 10 mins to manage 500 user's
> > > > > requests is unacceptable for the application I'm working on. Btw, CPU
> > > > > usage never goes beyond 50% during the execution of the script with
> > > > > 3GB of RAM in the box.
> > > > >
> > > > > My plan is, starts with 500 users using ACT on a server, then get 4
> > > > > more server to run another 2000 users. After that, increase each
> > > > > server by 100 users from 500 to 1000 and see how it goes.
> > > > > Unfortunately, RPS issue and high Processing Failure counter in
> > > > > Performance monitoring of RS Web Service make me feel so frustrated
> > > > > and don't know what to do.
> > > > >
> > > > > We need to address these 2 issues (RPS & high Processing Failure
> > > > > counter) ASAP to continue with our performance testing. If not, the
> > > > > delay in shipping of this project will affect thousands of user in the
> > > > > upcoming month. I hope MS can do something to help out. Thanks!
> > > > >
> > > > > CCC
> > > > >
> > > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > > > > The Report Server is no different than any other ASP.NET web
> application. It
> > > > > > runs under IIS and processes report request in multi-threaded
> > fashion.
> > As
> > > > > > usual with server-side web applications, CPU is the first resource
> > that
> > will
> > > > > > be maxed out. Therefore, beefing up your report server CPU power
> > should
> > come
> > > > > > as a no-brainer. A dual CPU box (quad recommended) would have
> > probably
> > made
> > > > > > a difference.
> > > > > >
> > > > > > To estimate how well your server configuration meet the user load I
> would
> > > > > > suggest you conduct a capacity planning study. For example, assuming
> that
> > > > > > the 1,000 users will generate 20 reports per hour during normal
> working
> > > > > > hours, the Report Server needs to handle (1000 x 8 x 20) / 3,600 => 45
> > > > > > reports (requests)/sec. Once you establish your performance goal,
> > you
> > could
> > > > > > stress-test the server to find out its maximum throughput, e.g. by
> > using
> > ACT
> > > > > > scripts. As you could see, this is no much different than
> > stress-testing
> > web
> > > > > > applications.
> > > > > >
> > > > > > If the actual server throughput is beyond your expectations, you
> should
> > > > > > first try optimizing your report performance. I found the RS
> > execution
> > log
> > > > > > indispensable for this purpose because it gives you time breakdown
> > of
> > how
> > > > > > much the Report Server has spent fetching the report data,
> > generating
> > and
> > > > > > rendering the report. In addition, don't forget caching. You can
> increase
> > > > > > substantially the server throughput by using different forms of
> > caching.
> > For
> > > > > > example, most likely in your initial tests you didn't pass the
> > session
> > id
> > > > > > with the report requests. However, if the reports are rendered by
> > URL,
> > the
> > > > > > browser will automatically pass the report session id which in many
> cases
> > > > > > could help the Report Server retrieve the report intermediate
> definition
> > > > > > without querying the data source and generating the report anew.
> Execution
> > > > > > caching will speed up things even more because now multiple users
> > can
> > share
> > > > > > the same cached report instance. Same with snapshot caching.
> > > > > >
> > > > > > Finally, to achieve fault tollerance and linear scalability (to a
> > point)
> > you
> > > > > > should consider clustering multiple report servers in a web farm
> > > > > > environment.
> > > > > >
> > > > > > --
> > > > > > Hope this helps.
> > > > > >
> > > > > > ---
> > > > > > Teo Lachev, MCSD, MCT
> > > > > > Author: "Microsoft Reporting Services in Action"
> > > > > > http://www.prologika.com
> > > > > >
> > > > > >
> > > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > > > > I created a small program using XMLHTTP to test the performance of
> pdf
> > > > > > > reports rendering. The program basically sends 20-30 XMLHTTP async
> > > > > > > requests with callback function in the format of:
> > > > > > >
> > > > > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > > > > >
> > > > > > > This statement was executed 20-30 times in less than a second to
> test
> > > > > > > if the server can handle multiple request in a short time.
> > > > > > >
> > > > > > > Monitoring the CPU, memory usage and generation of the actual pdf
> > > > > > > files on the server, we noticed that the pdf reports rendering
> were
> > > > > > > processed sequentially because files were generated one after
> another
> > > > > > > and the CPU utilization registered a spike that hits around 50%
> for
> > > > > > > each report. If we count the spikes, it matched the no. of reports
> > > > > > > generated. No other CPU/memory intensive services are running on
> the
> > > > > > > server when we perform the said test. Each generated pdf file is
> about
> > > > > > > 30-40k in size and took about 2-3 sec.
> > > > > > >
> > > > > > > We understand the report rendering time varies because of its
> > > > > > > complexity. But our concern is if this is how RS behaves, when
> there
> > > > > > > are 1000 users requesting for their report, it will take years for
> the
> > > > > > > 1000th user to get their copy generated and displayed on their
> > > > > > > browser.
> > > > > > >
> > > > > > > Is my investigation correct? Anything we can do to improve the
> > > > > > > rendering and reduce the time needed?
> > > > > > >
> > > > > > > Thanks in advance for any assistance/comments!!!
> > > > > > >
> > > > > > >
> > > > > > > CCC|||Hi,
Well, this is more inline with what I would expect. So what conclusions
could you draw from your tests? Here are mine:
1. Obviously, report generation is CPU-intensive task. As you can imaging
generating and rendering the report take quite a bit of CPU cycles.
Therefore, the more CPU power you throw in the better. To get linear
scalabity to a point consider clustering the Report Server. This will also
give fault tollerance.
2. The big unknown is how good 2 RPS are? Assuming 8 hrs working day even 2
RPS will give you about 60,000 reports total per day. That's very healthy
report load if you ask me. So, you may be fine with just 2 RPS. Again, to be
more determinstic you need to do a feasibility study.
3. Consider using report execution caching or report session caching. When
caching is used, the report intermediate format is cached. The data source
is not queried and the report is not processed again. The Report Server only
renders the report because it can be requested in different formats.
Hope this helps.
----
Teo Lachev, MCSD, MCT
Author: "Microsoft Reporting Services in Action"
Publisher website: http://www.manning.com/lachev
Buy it from Amazon.com : http://tinyurl.com/3l49j
Home page and blog: http://www.prologika.com/
----
"CCC" <tommytan25@.yahoo.com> wrote in message
news:ab9130f9.0408260042.2333dd6e@.posting.google.com...
> Just tried using sample reports included in the RS installer, still
> the same behavior.
> ============================================================> URL -
Test.SendRequest("http://abc.xyz.com/ReportServer?/company%20sales&rs:Comman
d=Render&rs:format=html4.0&rc:Toolbar=false")
> No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
> Test duration run time: 3 mins.
> 1 User -
> RS Server CPU Utilization: 50-60%
> Ave. RPS: 1.07
> 2 Users -
> RS Server CPU Utilization: 70-80% (Drop to 30-40% for the last 0.5
> min)
> Ave. RPS: 2.43 (1-1.5 RPS for the first 2.5 mins, then jump to 8-9 RPS
> for the remaining 0.5 mins)
> 4 Users -
> RS Server CPU Utilization: 95-100% (Drop to 30-40% for the last 20
> sec.)
> Ave. RPS: 3.03 (1-1.5 RPS for the first 2.7 mins, then jump to 16-17
> RPS for the remaining 20 sec)
> ============================================================> URL -
Test.SendRequest("http://staging.atitesting.com/ReportServer?/Product%20Cata
log&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false")
> No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
> Test duration run time: 3 mins.
> 1 User -
> RS Server CPU Utilization: 7-40% (Up and down for the entire 3 mins.)
> Ave. RPS: 0.26 (0.5-1 RPS)
> 2 Users -
> RS Server CPU Utilization: 20-70% (Up and down for the entire 3 mins.)
> Ave. RPS: 0.41 (0.5-1 RPS)
> 4 Users -
> RS Server CPU Utilization: 20-100% (Up and down for the entire 3
> mins.)
> Ave. RPS: 0.53 (0.5-1.5 RPS)
> Can anybody tell me why is it like that' Hello Microsoft'
> CCC
>
> tommytan25@.yahoo.com (CCC) wrote in message
news:<ab9130f9.0408251851.75145c67@.posting.google.com>...
> > Hi Teo,
> >
> > The version of ACT is 1.0.536.0 that comes with my VS.NET Enterprise.
> >
> > I have 4 servers with ACT installed:
> > * Server A - P4 1.8 GHz & 512RAM
> > * Servers B, C & D - 4 x Xeon 2.8 GHz & 4GB RAM
> > * RS Server - 2 x Xeon 3.06GHz & 3GB RAM (abc.xyz.com)
> > * Each serve has T1 connection to the internet
> >
> >
> > For each server, I created a test script containing only 1 line:
> >
Test.SendRequest("http://abc.xyz.com/ReportServer?/StudentReportCard&rs:Comm
and=Render&rs:format=html4.0&rc:Toolbar=false&UserID=3232")
> >
> > I tried running 5 separate tests using only Server B by setting the
> > simultaneous browser connections in test properties to 20, 100, 150,
> > 200 & 500 (also specify the test duration run time to 3 mins w/ 0
> > warmup time). During the entire 5 tests, RPS always stay at 2-3 RPS
> > (CPU = 90 to 100%) until the tests were ended. I also tried lowering
> > the number of concurrent users as what you said (1, 2, 4, 8, 10, 20),
> > the RPS never changed, it remains between 2-3 RPS regardless of the
> > number of users specified. This is what I got from one of the test
> > with 2 concurrent users defined:
> >
> > ===========================================================> > Properties
> >
> > Test type: Dynamic
> > Simultaneous browser connections: 2
> > Warm up time (secs): 0
> > Test duration: 00:00:03:00
> > Test iterations: 270
> > Detailed test results generated: Yes
> >
> > Summary
> >
> > Total number of requests: 270
> > Total number of connections: 270
> >
> > Average requests per second: 1.50
> > Average time to first byte (msecs): 822.73
> > Average time to last byte (msecs): 1,281.97
> > Average time to last byte per iteration (msecs): 1,281.97
> >
> > Number of unique requests made in test: 1
> > Number of unique response codes: 1
> >
> > Errors Counts
> >
> > HTTP: 0
> > DNS: 0
> > Socket: 0
> >
> > Additional Network Statistics
> >
> > Average bandwidth (bytes/sec): 107,324.83
> >
> > Number of bytes sent (bytes): 120,490
> > Number of bytes received (bytes): 19,197,980
> >
> > Average rate of sent bytes (bytes/sec): 669.39
> > Average rate of received bytes (bytes/sec): 106,655.44
> >
> > Number of connection errors: 0
> > Number of send errors: 0
> > Number of receive errors: 0
> > Number of timeout errors: 0
> >
> > Response Codes
> >
> > Response Code: 200 - The request completed successfully.
> > Count: 270
> > Percent (%): 100.00
> >
> > ===========================================================> >
> > With 2 concurrent users defined in ACT, RS Server's CPU utilization is
> > around 50%. The throughput reached 100% when users were increased to
> > 4. I tried another simple login.asp link of another project on a
> > server over internet (not a RS server/site) with 2000 users using
> > server B and I was able to get 250+ RPS. The only thing I can think of
> > is RS or my report requires more resources or is trying to manage the
> > requests that comes in. That's why regardless of number of users, it
> > only allow 2-3 RPS to avoid bug down. Or, simply some DoS prevention
> > feature is in place that limit the requests to be processed. Comments?
> >
> > The problem here is we were not able to even establish performance
> > metric we need (reducing no. of users to achieve at least 100 RPS).
> > That's why I was thinking if there's anything I need to look at to
> > increase the RPS before I can proceed with further testing.
> >
> > I will try to test the RS sample reports and see if the behaviors are
> > the same...
> >
> > CCC
> >
> >
> > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
news:<eJaAHxpiEHA.536@.TK2MSFTNGP11.phx.gbl>...
> > > OK, tell me a bit more about your ACT testing methodology. First,
which
> > > version of ACT are you using? How do you set up 4 servers with 2000
users?
> > > Second, why do you start with 500 concurrent connections? Please note
that
> > > 500 users means that ACT will try to create a huge number of
background
> > > threads on the ACT client which may be what is causing the delay. What
is
> > > your CPU utilization on the client machine?
> > >
> > > As I said, I would establish a performance metrics in requests/sec.
Then, i
> > > will pump up the server load by factor of 2, but I won't start with
500
> > > users. This is in no way equivalent to 500 users requesting reports
because
> > > you have to factor in the user think time. Intead, I will start with
1, 2,
> > > 4, 8, etc. until I reach the troughput peak. Could you do this and let
me
> > > know at what point the Report Server throughput peaks.
> > >
> > > Does it make sense?
> > >
> > > --
> > > Hope this helps.
> > >
> > > ---
> > > Teo Lachev, MCSD, MCT
> > > Author: "Microsoft Reporting Services in Action"
> > > http://www.manning.com/lachev
> > > Home page and blog http://www.prologika.com
> > > ---
> > >
> > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > news:ab9130f9.0408241732.730edd8b@.posting.google.com...
> > > > Hi Teo,
> > > >
> > > > Thanks for you comments.
> > > >
> > > > Regardless of number of concurrent users I defined in ACT (1 server
> > > > with 500 users or 4 servers with 2000 users to request data from
RS),
> > > > the RS's processor utilization never goes above 50-60%. They system
> > > > doesn't seems to have resource constraint but rather processing
> > > > problem. That's why I'm saying RS is managing the throughput and
> > > > processing the requests sequentially.
> > > >
> > > > - 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
> > > > RPS
> > > > to complete the reports
> > > >
> > > > - 4 ACT server with 500x4 concurrent connections took 16mins at
0.5-1
> > > > RPS to complete the reports
> > > >
> > > > Yes there's a slight drop in the throughput but I was expecting RS
to
> > > > handle more than 1 RPS for both scenarios.
> > > >
> > > > I understand Total Processing Failure and Total Rejected Thread
> > > > counters were 2 different things, the first one is giving more
> > > > headache at thousands mark while the later is manageable between
5-10
> > > > for 500 concurrent users.
> > > >
> > > > Checking the ExecutionLog, out of 500 reports, there are only 40-50
> > > > report with very minimal (0.05 msec) overlapped execution start and
> > > > end time. The rest were processed in sequential manner.
> > > >
> > > > I posted the details of the ACT report and my script on the other
> > > > thread with the subject "Performance Testing Using ACT", please
refer
> > > > to the thread for more info.
> > > >
> > > > What else do I need to look at?
> > > >
> > > > Really appreciate your comments, thanks!
> > > >
> > > > CCC
> > > >
> > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> > news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> > > > > Hi,
> > > > >
> > > > > You can observe the number of threads by monitoring the
Process:Thread
> > Count
> > > > > counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn
additional
> > > > > threads to handle the load.
> > > > >
> > > > > You results puzzle me. CPU doesn't go above 50% but you can't get
more
> > > > > requests. Do you use think time in ACT? I would start with 2
connections
> > > > > then add additional connections by number of 2, e.g. 4, 8, 16 to
find
> > out
> > > > > where the server throughput maxes out. Then, you need to find out
what
> > the
> > > > > resource constraint is by looking at various performance counters.
RS
> > also
> > > > > comes with useful performance counters, such as report requests,
reports
> > > > > executed/sec, etc. Total Rejected Threads is an interesting one in
your
> > > > > case. "Total number of data processing threads rejected for
asynchronous
> > > > > processing, and subsequently handled as a synchronous process in
the
> > same
> > > > > thread. Each data source is processed on one thread. If the volume
of
> > > > > threads exceeds capacity, threads will be rejected for
asynchronous
> > > > > processing, and will be processed in a serial manner."
> > > > >
> > > > > Also, did you look at the RS execution log to find out where the
time
> > goes:
> > > > > querying the database, processing the report or rendering it in
PDF?
> > > > >
> > > > > --
> > > > > Hope this helps.
> > > > >
> > > > > ---
> > > > > Teo Lachev, MCSD, MCT
> > > > > Author: "Microsoft Reporting Services in Action"
> > > > > http://www.manning.com/lachev
> > > > > Home page and blog http://www.prologika.com
> > > > > ---
> > > > >
> > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > > > > > Hi Teo,
> > > > > >
> > > > > > Thank you very much for your inputs.
> > > > > >
> > > > > > Tried using ACT and created a script with 500 users to test RS
but RPS
> > > > > > never goes more than 1.5 during the testing. The test took about
10
> > > > > > mins to complete and all requests were logged into ExecutionLog
as
> > > > > > "rsSuccess".
> > > > > >
> > > > > > Although it was a success run, it seems to me that RS is still
> > > > > > rendering the requests 1 by 1 and not in a "multi-thread manner"
(but
> > > > > > I do believe it should run multi-thread). 10 mins to manage 500
user's
> > > > > > requests is unacceptable for the application I'm working on.
Btw, CPU
> > > > > > usage never goes beyond 50% during the execution of the script
with
> > > > > > 3GB of RAM in the box.
> > > > > >
> > > > > > My plan is, starts with 500 users using ACT on a server, then
get 4
> > > > > > more server to run another 2000 users. After that, increase each
> > > > > > server by 100 users from 500 to 1000 and see how it goes.
> > > > > > Unfortunately, RPS issue and high Processing Failure counter in
> > > > > > Performance monitoring of RS Web Service make me feel so
frustrated
> > > > > > and don't know what to do.
> > > > > >
> > > > > > We need to address these 2 issues (RPS & high Processing Failure
> > > > > > counter) ASAP to continue with our performance testing. If not,
the
> > > > > > delay in shipping of this project will affect thousands of user
in the
> > > > > > upcoming month. I hope MS can do something to help out. Thanks!
> > > > > >
> > > > > > CCC
> > > > > >
> > > > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> > news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > > > > > The Report Server is no different than any other ASP.NET web
> > application. It
> > > > > > > runs under IIS and processes report request in multi-threaded
> > > fashion.
> > > As
> > > > > > > usual with server-side web applications, CPU is the first
resource
> > > that
> > > will
> > > > > > > be maxed out. Therefore, beefing up your report server CPU
power
> > > should
> > > come
> > > > > > > as a no-brainer. A dual CPU box (quad recommended) would have
> > > probably
> > > made
> > > > > > > a difference.
> > > > > > >
> > > > > > > To estimate how well your server configuration meet the user
load I
> > would
> > > > > > > suggest you conduct a capacity planning study. For example,
assuming
> > that
> > > > > > > the 1,000 users will generate 20 reports per hour during
normal
> > working
> > > > > > > hours, the Report Server needs to handle (1000 x 8 x 20) /
3,600 => > 45
> > > > > > > reports (requests)/sec. Once you establish your performance
goal,
> > > you
> > > could
> > > > > > > stress-test the server to find out its maximum throughput,
e.g. by
> > > using
> > > ACT
> > > > > > > scripts. As you could see, this is no much different than
> > > stress-testing
> > > web
> > > > > > > applications.
> > > > > > >
> > > > > > > If the actual server throughput is beyond your expectations,
you
> > should
> > > > > > > first try optimizing your report performance. I found the RS
> > > execution
> > > log
> > > > > > > indispensable for this purpose because it gives you time
breakdown
> > > of
> > > how
> > > > > > > much the Report Server has spent fetching the report data,
> > > generating
> > > and
> > > > > > > rendering the report. In addition, don't forget caching. You
can
> > increase
> > > > > > > substantially the server throughput by using different forms
of
> > > caching.
> > > For
> > > > > > > example, most likely in your initial tests you didn't pass the
> > > session
> > > id
> > > > > > > with the report requests. However, if the reports are rendered
by
> > > URL,
> > > the
> > > > > > > browser will automatically pass the report session id which in
many
> > cases
> > > > > > > could help the Report Server retrieve the report intermediate
> > definition
> > > > > > > without querying the data source and generating the report
anew.
> > Execution
> > > > > > > caching will speed up things even more because now multiple
users
> > > can
> > > share
> > > > > > > the same cached report instance. Same with snapshot caching.
> > > > > > >
> > > > > > > Finally, to achieve fault tollerance and linear scalability
(to a
> > > point)
> > > you
> > > > > > > should consider clustering multiple report servers in a web
farm
> > > > > > > environment.
> > > > > > >
> > > > > > > --
> > > > > > > Hope this helps.
> > > > > > >
> > > > > > > ---
> > > > > > > Teo Lachev, MCSD, MCT
> > > > > > > Author: "Microsoft Reporting Services in Action"
> > > > > > > http://www.prologika.com
> > > > > > >
> > > > > > >
> > > > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > > > > > I created a small program using XMLHTTP to test the
performance of
> > pdf
> > > > > > > > reports rendering. The program basically sends 20-30 XMLHTTP
async
> > > > > > > > requests with callback function in the format of:
> > > > > > > >
> > > > > > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > > > > > >
> > > > > > > > This statement was executed 20-30 times in less than a
second to
> > test
> > > > > > > > if the server can handle multiple request in a short time.
> > > > > > > >
> > > > > > > > Monitoring the CPU, memory usage and generation of the
actual pdf
> > > > > > > > files on the server, we noticed that the pdf reports
rendering
> > were
> > > > > > > > processed sequentially because files were generated one
after
> > another
> > > > > > > > and the CPU utilization registered a spike that hits around
50%
> > for
> > > > > > > > each report. If we count the spikes, it matched the no. of
reports
> > > > > > > > generated. No other CPU/memory intensive services are
running on
> > the
> > > > > > > > server when we perform the said test. Each generated pdf
file is
> > about
> > > > > > > > 30-40k in size and took about 2-3 sec.
> > > > > > > >
> > > > > > > > We understand the report rendering time varies because of
its
> > > > > > > > complexity. But our concern is if this is how RS behaves,
when
> > there
> > > > > > > > are 1000 users requesting for their report, it will take
years for
> > the
> > > > > > > > 1000th user to get their copy generated and displayed on
their
> > > > > > > > browser.
> > > > > > > >
> > > > > > > > Is my investigation correct? Anything we can do to improve
the
> > > > > > > > rendering and reduce the time needed?
> > > > > > > >
> > > > > > > > Thanks in advance for any assistance/comments!!!
> > > > > > > >
> > > > > > > >
> > > > > > > > CCC|||Hi Teo,
Well, I think this is not an acceptable performance for our
application. We're not looking at how much RS can generate in 1 day
but rather how many concurrent user it can handle at the same time. We
have at least 20-30 user accessing the reports simultaneously using
URL Access and I'm expecting RS can provide at least 10-20 RPS when
requests flow in.
Caching is good but I highly doubt it can be applied to real time
reports we need (real time statistical reports containing data from
10000+ users). I wonder if anyone had performed a similar test as I
did and hope you can give me you opinions.
I understand report rendering requires lots of CPU resources but I'm
just surprised that 2-3 RPS is what RS can handle given I'm using a
dual Xeon processor and 3GB RAM already (comparing to 250+ RPS for a
simple login page on another server with a P4 CPU and 512MB RAM).
Thank you very much Teo for your comments, I believe we need a more
formal explanation from MS to explain what is happening. I tried
running 4 ACT servers at the same time (2000 users in total) and the
RPS has no far difference from running 1 ACT server with 500 users.
Why' We are really suspecting the DoS protection of RS (some guys
from MS told me that) is causing the bottleneck and a small change in
setting should be able to correct this problem. Can any one help?
CCC
"Teo Lachev" <teo@.nospam.prologika.com> wrote in message news:<e6R8Gu5iEHA.384@.TK2MSFTNGP10.phx.gbl>...
> Hi,
> Well, this is more inline with what I would expect. So what conclusions
> could you draw from your tests? Here are mine:
> 1. Obviously, report generation is CPU-intensive task. As you can imaging
> generating and rendering the report take quite a bit of CPU cycles.
> Therefore, the more CPU power you throw in the better. To get linear
> scalabity to a point consider clustering the Report Server. This will also
> give fault tollerance.
> 2. The big unknown is how good 2 RPS are? Assuming 8 hrs working day even 2
> RPS will give you about 60,000 reports total per day. That's very healthy
> report load if you ask me. So, you may be fine with just 2 RPS. Again, to be
> more determinstic you need to do a feasibility study.
> 3. Consider using report execution caching or report session caching. When
> caching is used, the report intermediate format is cached. The data source
> is not queried and the report is not processed again. The Report Server only
> renders the report because it can be requested in different formats.
>
> --
> Hope this helps.
> ----
> Teo Lachev, MCSD, MCT
> Author: "Microsoft Reporting Services in Action"
> Publisher website: http://www.manning.com/lachev
> Buy it from Amazon.com : http://tinyurl.com/3l49j
> Home page and blog: http://www.prologika.com/
> ----
> "CCC" <tommytan25@.yahoo.com> wrote in message
> news:ab9130f9.0408260042.2333dd6e@.posting.google.com...
> > Just tried using sample reports included in the RS installer, still
> > the same behavior.
> > ============================================================> > URL -
> Test.SendRequest("http://abc.xyz.com/ReportServer?/company%20sales&rs:Comman
> d=Render&rs:format=html4.0&rc:Toolbar=false")
> > No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
> > Test duration run time: 3 mins.
> >
> > 1 User -
> > RS Server CPU Utilization: 50-60%
> > Ave. RPS: 1.07
> >
> > 2 Users -
> > RS Server CPU Utilization: 70-80% (Drop to 30-40% for the last 0.5
> > min)
> > Ave. RPS: 2.43 (1-1.5 RPS for the first 2.5 mins, then jump to 8-9 RPS
> > for the remaining 0.5 mins)
> >
> > 4 Users -
> > RS Server CPU Utilization: 95-100% (Drop to 30-40% for the last 20
> > sec.)
> > Ave. RPS: 3.03 (1-1.5 RPS for the first 2.7 mins, then jump to 16-17
> > RPS for the remaining 20 sec)
> >
> > ============================================================> > URL -
> Test.SendRequest("http://staging.atitesting.com/ReportServer?/Product%20Cata
> log&rs:Command=Render&rs:format=html4.0&rc:Toolbar=false")
> > No. of concurrent users defined in ACT: 1, 2, 4, 8 ,10, 20
> > Test duration run time: 3 mins.
> >
> > 1 User -
> > RS Server CPU Utilization: 7-40% (Up and down for the entire 3 mins.)
> > Ave. RPS: 0.26 (0.5-1 RPS)
> >
> > 2 Users -
> > RS Server CPU Utilization: 20-70% (Up and down for the entire 3 mins.)
> > Ave. RPS: 0.41 (0.5-1 RPS)
> >
> > 4 Users -
> > RS Server CPU Utilization: 20-100% (Up and down for the entire 3
> > mins.)
> > Ave. RPS: 0.53 (0.5-1.5 RPS)
> >
> > Can anybody tell me why is it like that' Hello Microsoft'
> >
> > CCC
> >
> >
> > tommytan25@.yahoo.com (CCC) wrote in message
> news:<ab9130f9.0408251851.75145c67@.posting.google.com>...
> > > Hi Teo,
> > >
> > > The version of ACT is 1.0.536.0 that comes with my VS.NET Enterprise.
> > >
> > > I have 4 servers with ACT installed:
> > > * Server A - P4 1.8 GHz & 512RAM
> > > * Servers B, C & D - 4 x Xeon 2.8 GHz & 4GB RAM
> > > * RS Server - 2 x Xeon 3.06GHz & 3GB RAM (abc.xyz.com)
> > > * Each serve has T1 connection to the internet
> > >
> > >
> > > For each server, I created a test script containing only 1 line:
> > >
> Test.SendRequest("http://abc.xyz.com/ReportServer?/StudentReportCard&rs:Comm
> and=Render&rs:format=html4.0&rc:Toolbar=false&UserID=3232")
> > >
> > > I tried running 5 separate tests using only Server B by setting the
> > > simultaneous browser connections in test properties to 20, 100, 150,
> > > 200 & 500 (also specify the test duration run time to 3 mins w/ 0
> > > warmup time). During the entire 5 tests, RPS always stay at 2-3 RPS
> > > (CPU = 90 to 100%) until the tests were ended. I also tried lowering
> > > the number of concurrent users as what you said (1, 2, 4, 8, 10, 20),
> > > the RPS never changed, it remains between 2-3 RPS regardless of the
> > > number of users specified. This is what I got from one of the test
> > > with 2 concurrent users defined:
> > >
> > > ===========================================================> > > Properties
> > >
> > > Test type: Dynamic
> > > Simultaneous browser connections: 2
> > > Warm up time (secs): 0
> > > Test duration: 00:00:03:00
> > > Test iterations: 270
> > > Detailed test results generated: Yes
> > >
> > > Summary
> > >
> > > Total number of requests: 270
> > > Total number of connections: 270
> > >
> > > Average requests per second: 1.50
> > > Average time to first byte (msecs): 822.73
> > > Average time to last byte (msecs): 1,281.97
> > > Average time to last byte per iteration (msecs): 1,281.97
> > >
> > > Number of unique requests made in test: 1
> > > Number of unique response codes: 1
> > >
> > > Errors Counts
> > >
> > > HTTP: 0
> > > DNS: 0
> > > Socket: 0
> > >
> > > Additional Network Statistics
> > >
> > > Average bandwidth (bytes/sec): 107,324.83
> > >
> > > Number of bytes sent (bytes): 120,490
> > > Number of bytes received (bytes): 19,197,980
> > >
> > > Average rate of sent bytes (bytes/sec): 669.39
> > > Average rate of received bytes (bytes/sec): 106,655.44
> > >
> > > Number of connection errors: 0
> > > Number of send errors: 0
> > > Number of receive errors: 0
> > > Number of timeout errors: 0
> > >
> > > Response Codes
> > >
> > > Response Code: 200 - The request completed successfully.
> > > Count: 270
> > > Percent (%): 100.00
> > >
> > > ===========================================================> > >
> > > With 2 concurrent users defined in ACT, RS Server's CPU utilization is
> > > around 50%. The throughput reached 100% when users were increased to
> > > 4. I tried another simple login.asp link of another project on a
> > > server over internet (not a RS server/site) with 2000 users using
> > > server B and I was able to get 250+ RPS. The only thing I can think of
> > > is RS or my report requires more resources or is trying to manage the
> > > requests that comes in. That's why regardless of number of users, it
> > > only allow 2-3 RPS to avoid bug down. Or, simply some DoS prevention
> > > feature is in place that limit the requests to be processed. Comments?
> > >
> > > The problem here is we were not able to even establish performance
> > > metric we need (reducing no. of users to achieve at least 100 RPS).
> > > That's why I was thinking if there's anything I need to look at to
> > > increase the RPS before I can proceed with further testing.
> > >
> > > I will try to test the RS sample reports and see if the behaviors are
> > > the same...
> > >
> > > CCC
> > >
> > >
> > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<eJaAHxpiEHA.536@.TK2MSFTNGP11.phx.gbl>...
> > > > OK, tell me a bit more about your ACT testing methodology. First,
> which
> > > > version of ACT are you using? How do you set up 4 servers with 2000
> users?
> > > > Second, why do you start with 500 concurrent connections? Please note
> that
> > > > 500 users means that ACT will try to create a huge number of
> background
> > > > threads on the ACT client which may be what is causing the delay. What
> is
> > > > your CPU utilization on the client machine?
> > > >
> > > > As I said, I would establish a performance metrics in requests/sec.
> Then, i
> > > > will pump up the server load by factor of 2, but I won't start with
> 500
> > > > users. This is in no way equivalent to 500 users requesting reports
> because
> > > > you have to factor in the user think time. Intead, I will start with
> 1, 2,
> > > > 4, 8, etc. until I reach the troughput peak. Could you do this and let
> me
> > > > know at what point the Report Server throughput peaks.
> > > >
> > > > Does it make sense?
> > > >
> > > > --
> > > > Hope this helps.
> > > >
> > > > ---
> > > > Teo Lachev, MCSD, MCT
> > > > Author: "Microsoft Reporting Services in Action"
> > > > http://www.manning.com/lachev
> > > > Home page and blog http://www.prologika.com
> > > > ---
> > > >
> > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > news:ab9130f9.0408241732.730edd8b@.posting.google.com...
> > > > > Hi Teo,
> > > > >
> > > > > Thanks for you comments.
> > > > >
> > > > > Regardless of number of concurrent users I defined in ACT (1 server
> > > > > with 500 users or 4 servers with 2000 users to request data from
> RS),
> > > > > the RS's processor utilization never goes above 50-60%. They system
> > > > > doesn't seems to have resource constraint but rather processing
> > > > > problem. That's why I'm saying RS is managing the throughput and
> > > > > processing the requests sequentially.
> > > > >
> > > > > - 1 ACT server with 500 concurrent connections took 10mins at 0.8-1
> > > > > RPS
> > > > > to complete the reports
> > > > >
> > > > > - 4 ACT server with 500x4 concurrent connections took 16mins at
> 0.5-1
> > > > > RPS to complete the reports
> > > > >
> > > > > Yes there's a slight drop in the throughput but I was expecting RS
> to
> > > > > handle more than 1 RPS for both scenarios.
> > > > >
> > > > > I understand Total Processing Failure and Total Rejected Thread
> > > > > counters were 2 different things, the first one is giving more
> > > > > headache at thousands mark while the later is manageable between
> 5-10
> > > > > for 500 concurrent users.
> > > > >
> > > > > Checking the ExecutionLog, out of 500 reports, there are only 40-50
> > > > > report with very minimal (0.05 msec) overlapped execution start and
> > > > > end time. The rest were processed in sequential manner.
> > > > >
> > > > > I posted the details of the ACT report and my script on the other
> > > > > thread with the subject "Performance Testing Using ACT", please
> refer
> > > > > to the thread for more info.
> > > > >
> > > > > What else do I need to look at?
> > > > >
> > > > > Really appreciate your comments, thanks!
> > > > >
> > > > > CCC
> > > > >
> > > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<#CpCmydiEHA.1384@.TK2MSFTNGP10.phx.gbl>...
> > > > > > Hi,
> > > > > >
> > > > > > You can observe the number of threads by monitoring the
> Process:Thread
> Count
> > > > > > counter for aspnet_wp (IIS 5) or w3wp (IIS6). IIS does spawn
> additional
> > > > > > threads to handle the load.
> > > > > >
> > > > > > You results puzzle me. CPU doesn't go above 50% but you can't get
> more
> > > > > > requests. Do you use think time in ACT? I would start with 2
> connections
> > > > > > then add additional connections by number of 2, e.g. 4, 8, 16 to
> find
> out
> > > > > > where the server throughput maxes out. Then, you need to find out
> what
> the
> > > > > > resource constraint is by looking at various performance counters.
> RS
> also
> > > > > > comes with useful performance counters, such as report requests,
> reports
> > > > > > executed/sec, etc. Total Rejected Threads is an interesting one in
> your
> > > > > > case. "Total number of data processing threads rejected for
> asynchronous
> > > > > > processing, and subsequently handled as a synchronous process in
> the
> same
> > > > > > thread. Each data source is processed on one thread. If the volume
> of
> > > > > > threads exceeds capacity, threads will be rejected for
> asynchronous
> > > > > > processing, and will be processed in a serial manner."
> > > > > >
> > > > > > Also, did you look at the RS execution log to find out where the
> time
> goes:
> > > > > > querying the database, processing the report or rendering it in
> PDF?
> > > > > >
> > > > > > --
> > > > > > Hope this helps.
> > > > > >
> > > > > > ---
> > > > > > Teo Lachev, MCSD, MCT
> > > > > > Author: "Microsoft Reporting Services in Action"
> > > > > > http://www.manning.com/lachev
> > > > > > Home page and blog http://www.prologika.com
> > > > > > ---
> > > > > >
> > > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > > news:ab9130f9.0408240116.1f8fb8b2@.posting.google.com...
> > > > > > > Hi Teo,
> > > > > > >
> > > > > > > Thank you very much for your inputs.
> > > > > > >
> > > > > > > Tried using ACT and created a script with 500 users to test RS
> but RPS
> > > > > > > never goes more than 1.5 during the testing. The test took about
> 10
> > > > > > > mins to complete and all requests were logged into ExecutionLog
> as
> > > > > > > "rsSuccess".
> > > > > > >
> > > > > > > Although it was a success run, it seems to me that RS is still
> > > > > > > rendering the requests 1 by 1 and not in a "multi-thread manner"
> (but
> > > > > > > I do believe it should run multi-thread). 10 mins to manage 500
> user's
> > > > > > > requests is unacceptable for the application I'm working on.
> Btw, CPU
> > > > > > > usage never goes beyond 50% during the execution of the script
> with
> > > > > > > 3GB of RAM in the box.
> > > > > > >
> > > > > > > My plan is, starts with 500 users using ACT on a server, then
> get 4
> > > > > > > more server to run another 2000 users. After that, increase each
> > > > > > > server by 100 users from 500 to 1000 and see how it goes.
> > > > > > > Unfortunately, RPS issue and high Processing Failure counter in
> > > > > > > Performance monitoring of RS Web Service make me feel so
> frustrated
> > > > > > > and don't know what to do.
> > > > > > >
> > > > > > > We need to address these 2 issues (RPS & high Processing Failure
> > > > > > > counter) ASAP to continue with our performance testing. If not,
> the
> > > > > > > delay in shipping of this project will affect thousands of user
> in the
> > > > > > > upcoming month. I hope MS can do something to help out. Thanks!
> > > > > > >
> > > > > > > CCC
> > > > > > >
> > > > > > > "Teo Lachev" <teo@.nospam.prologika.com> wrote in message
> news:<eKBIr35fEHA.3024@.TK2MSFTNGP10.phx.gbl>...
> > > > > > > > The Report Server is no different than any other ASP.NET web
> application. It
> > > > > > > > runs under IIS and processes report request in multi-threaded
> > > > fashion.
> > > > As
> > > > > > > > usual with server-side web applications, CPU is the first
> resource
> > > > that
> > > > will
> > > > > > > > be maxed out. Therefore, beefing up your report server CPU
> power
> > > > should
> > > > come
> > > > > > > > as a no-brainer. A dual CPU box (quad recommended) would have
> > > > probably
> > > > made
> > > > > > > > a difference.
> > > > > > > >
> > > > > > > > To estimate how well your server configuration meet the user
> load I
> would
> > > > > > > > suggest you conduct a capacity planning study. For example,
> assuming
> that
> > > > > > > > the 1,000 users will generate 20 reports per hour during
> normal
> working
> > > > > > > > hours, the Report Server needs to handle (1000 x 8 x 20) /
> 3,600 => 45
> > > > > > > > reports (requests)/sec. Once you establish your performance
> goal,
> > > > you
> > > > could
> > > > > > > > stress-test the server to find out its maximum throughput,
> e.g. by
> > > > using
> > > > ACT
> > > > > > > > scripts. As you could see, this is no much different than
> > > > stress-testing
> > > > web
> > > > > > > > applications.
> > > > > > > >
> > > > > > > > If the actual server throughput is beyond your expectations,
> you
> should
> > > > > > > > first try optimizing your report performance. I found the RS
> > > > execution
> > > > log
> > > > > > > > indispensable for this purpose because it gives you time
> breakdown
> > > > of
> > > > how
> > > > > > > > much the Report Server has spent fetching the report data,
> > > > generating
> > > > and
> > > > > > > > rendering the report. In addition, don't forget caching. You
> can
> increase
> > > > > > > > substantially the server throughput by using different forms
> of
> > > > caching.
> > > > For
> > > > > > > > example, most likely in your initial tests you didn't pass the
> > > > session
> > > > id
> > > > > > > > with the report requests. However, if the reports are rendered
> by
> > > > URL,
> > > > the
> > > > > > > > browser will automatically pass the report session id which in
> many
> cases
> > > > > > > > could help the Report Server retrieve the report intermediate
> definition
> > > > > > > > without querying the data source and generating the report
> anew.
> Execution
> > > > > > > > caching will speed up things even more because now multiple
> users
> > > > can
> > > > share
> > > > > > > > the same cached report instance. Same with snapshot caching.
> > > > > > > >
> > > > > > > > Finally, to achieve fault tollerance and linear scalability
> (to a
> > > > point)
> > > > you
> > > > > > > > should consider clustering multiple report servers in a web
> farm
> > > > > > > > environment.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Hope this helps.
> > > > > > > >
> > > > > > > > ---
> > > > > > > > Teo Lachev, MCSD, MCT
> > > > > > > > Author: "Microsoft Reporting Services in Action"
> > > > > > > > http://www.prologika.com
> > > > > > > >
> > > > > > > >
> > > > > > > > "CCC" <tommytan25@.yahoo.com> wrote in message
> > > > > > > > news:ab9130f9.0408101717.3691b8fd@.posting.google.com...
> > > > > > > > > I created a small program using XMLHTTP to test the
> performance of
> pdf
> > > > > > > > > reports rendering. The program basically sends 20-30 XMLHTTP
> async
> > > > > > > > > requests with callback function in the format of:
> > > > > > > > >
> > > > > > > > > XMLHttpRequest.open "POST", <RS url access>, True
> > > > > > > > >
> > > > > > > > > This statement was executed 20-30 times in less than a
> second to
> test
> > > > > > > > > if the server can handle multiple request in a short time.
> > > > > > > > >
> > > > > > > > > Monitoring the CPU, memory usage and generation of the
> actual pdf
> > > > > > > > > files on the server, we noticed that the pdf reports
> rendering
> were
> > > > > > > > > processed sequentially because files were generated one
> after
> another
> > > > > > > > > and the CPU utilization registered a spike that hits around
> 50%
> for
> > > > > > > > > each report. If we count the spikes, it matched the no. of
> reports
> > > > > > > > > generated. No other CPU/memory intensive services are
> running on
> the
> > > > > > > > > server when we perform the said test. Each generated pdf
> file is
> about
> > > > > > > > > 30-40k in size and took about 2-3 sec.
> > > > > > > > >
> > > > > > > > > We understand the report rendering time varies because of
> its
> > > > > > > > > complexity. But our concern is if this is how RS behaves,
> when
> there
> > > > > > > > > are 1000 users requesting for their report, it will take
> years for
> the
> > > > > > > > > 1000th user to get their copy generated and displayed on
> their
> > > > > > > > > browser.
> > > > > > > > >
> > > > > > > > > Is my investigation correct? Anything we can do to improve
> the
> > > > > > > > > rendering and reduce the time needed?
> > > > > > > > >
> > > > > > > > > Thanks in advance for any assistance/comments!!!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > CCC
