Apache JMeter Part 4: Testing the Throughput and Performance of InfoSec Institute

Testing the Throughput of InfoSec Institute

Running the JMeter

When everything is set-up all that is left for us to do is run the JMeter and observe the results. We can do that by simply pressing the Menu – Run – Start button.

Setting-up JMeter

Let's present the options that we used when running the test cases in this tutorial. We used three elements, that we won't describe in detail, since they have already been described in other parts of this tutorial series.

HTTP Request Defaults

We specified only the "Server Name or IP" to be www.infosecinstitute.com.

HTTP Request

In the HTTP Request element we specified that we want to access a resource /index.html as can be seen in the picture below:

Graph Result Listener

We also added the graph result listener that can show us the results of the test plan. There is no need to configure the Graph Result element, so we didn't provide a picture.

Presenting the Results

In this section we'll present various results we received with the JMeter when testing the performance of the www.infosecinstitute.com web site.

First, let's change the Ramp-up period to 0, 2, 10, 100, 1000 seconds and observe the results when simulating 1000 users and repeating the loop only once.

The picture below contains the results of performance testing with a Ramp-Up period of 0 seconds (therefore all requests are sent at once):

The picture below contains the results of performance testing with a Ramp-Up period of 2 seconds:

The picture below contains the results of performance testing with a Ramp-Up period of 10 seconds:

The picture below contains the results of performance testing with a Ramp-Up period of 100 seconds:

The picture below contains the results of performance testing with a Ramp-Up period of 1000 seconds (therefore 1 request will be sent per second):

If we take a look at the throughput we can see that in the first three cases it's rising considerably, but in the last two it is constant. A constant throughput with a low requests/minute value means that we've set the ramp-up period too high, because the target application (server) can process the previous requests before the next request is even sent. This further implies that we should lower the ramp-up period to increase the load on the server.

The first case has a throughput of 429.098 requests/minute, the second case has a throughput of 416.437 requests/minute, the third case has a throughput of 1015.916 requests/minute, the fourth case has a throughput of 597.699 requests/minute and the last case has a throughput of 60.022 requests/minute. We can notice that the number of requests is increasing with the bigger ramp-up period up until some point (til fourth case). This means that when we're sending all requests close together, the server can still process them fairly quickly, but not as fast as when we're introducing a slight delay between the requests, which is logical.

In the above cases the best ramp-up period was somewhere in between 10 and 100 seconds. We must look for the maximum throughput, which we observed in the third use case, but we must also have a constant throughput, which we observed in the fourth use case, which implies that the best ramp-up period is somewhere in between 10 and 100 seconds.

Testing the Performance of InfoSec Institute

Testing with Google Statistics

In the previous parts of the JMeter tutorials we talked about measuring the best throughput with the specified average number of users we expect to be visiting the website at any given time. Normally we can do a better job than guessing a number of users that we expect to be visiting a web page at a time. We can turn to the site statistics, like Google Analytics and write down the average number of users, or better yet, the maximum number of users visiting our web page at any given time.

Google Analytics stores the following pieces of information for us:

Total number of visits: Nt
Total number of unique visitors: Nu
Total number of pageviews: Np
Average number of visited pages per visit: Na
Total time: Tt (in seconds)
Average time a visitor spends on the site: Tu (in seconds)

Then we must calculate the number of users per second, which we'll directly input into our JMeter Thread Group configuration. We can calculate the number of users per second with the formula:

#Users = Nt / Tt * Tu

The obtained number will give us the average amount of users visiting our web page in any given time. We should also calculate the number of users in a time around the time when there was maximum number of users visiting our web site. This should give us the real indication of the total number of users that we must test for.

Afterward we must enter the obtained number directly into the Thread Group element in JMeter and begin our testing.

Testing without Google Statistics

Since I don't have Google statistics that I need to perform my tests against the InfoSec Institute web site, I'll have to resort to the method that can test the maximum number of users that can visit the mentioned web page. This will give us the best indication what the server can handle.

There is a limitation of simulating 1700 users per a desktop PC when running JMeter [1]. If we configure JMeter to use that many threads, we won't get the most accurate results, which is why I prefer to run JMeter with at most 1000 threads per one PC. If we need to test a web page with more users than 1000, we should connect a bunch of JMeter machines together remotely [2].

The following table summarizes the number of JMeter instances we need to load the testing server with the corresponding number of users:

# Users	# Jmeter Instances
100	1
1000	1
5000	5
10000	10
50000	50

The relation between the number of users and the JMeter instances is linear, since one PC can only handle so many concurrent connections.

Manage JMeter Instances Remotely

Here we'll take a look at how to manage Jmeter instances remotely. We'll install two JMeter instances, one on the remote computer and the other locally. Then we'll manage both simultaneously to run a performance test with a higher number of threads per minute than we could with only one instance.

By using JMeter we can run one JMeter GUI client that controls all the other remote JMeter instances and collects the data from them. With that we can distribute the load between multiple clients, each presenting a considerate load on the testing server.

To do that, we only need to configure one test plan, which is distributed among all remote JMeter instances, thus giving us control over all instanced from a single GUI client. The results are also presented in the GUI client so there is no need to copy the JMeter results from multiple clients over to the local machine.

Starting the Remote Instance

To run the JMeter remote instance, first we need to download the JMeter and extract it:

# wget http://www.apache.si//jmeter/binaries/apache-jmeter-2.7.tgz
# tar xvzf apache-jmeter-2.7.tgz
# cd apache-jmeter-2.7/

If we don't have Java installed, we need to install it. On the Ubuntu Linux distribution we can do that by executing the following command:

# apt-get install openjdk-6-jre

Afterward we can start the remote JMeter instance by issuing the command below:

# ./bin/jmeter-server

Created remote object: UnicastServerRef [liveRef: [endpoint:[1.2.3.4:54883](local),objID:[-2e56539d:139abf4f832:-7fff, -8592326131918894022]]]
Server failed to start: java.rmi.RemoteException: Cannot start. See server log file.
An error occurred: Cannot start. See server log file

We can see that the output is telling us that we should check the server log for a more detailed error of what happened. Ok, let's present the jmeter-server.log:

INFO - jmeter.engine.RemoteJMeterEngineImpl: Starting backing engine on 1099
INFO - jmeter.engine.RemoteJMeterEngineImpl: IP address=1.2.3.4
ERROR - jmeter.engine.RemoteJMeterEngineImpl: rmiregistry needs to be running to start JMeter in server mode java.rmi.ConnectException: Connection refused to host: 1.2.3.4; nested exception is: java.net.ConnectException: Connection refused

An error occurred, why? It's because the Java RMI (Remote Method Invocation) hasn't been started yet. To automatically start it, we should edit bin/jmeter.properties and set the server.rmi.create variable to be true:

# From JMeter 2.3.1, the jmeter server creates the RMI registry+ as part of the server process.
# To stop the server creating the RMI registry: server.rmi.create=true

When starting the jmeter-server, another error occurs:

# /.bin/jmeter-server

The jmeter-server.log contains the following debugging information:

INFO - jmeter.engine.RemoteJMeterEngineImpl: Starting backing engine on 1099
INFO - jmeter.engine.RemoteJMeterEngineImpl: IP address=1.2.3.4
INFO - jmeter.engine.RemoteJMeterEngineImpl: Creating RMI registry (server.rmi.create=true)
ERROR - jmeter.engine.RemoteJMeterEngineImpl: rmiregistry needs to be running to start JMeter in server mode java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.AccessException: Registry.Registry.rebind disallowed; origin /192.168.1.1 is non-local host

The above error happened because our hostname points to our WAN IP not LAN IP, thus giving us the IP of a router, which is 192.168.1.1. If we want to start jmeter-server we need to change the /etc/hosts and edit our hostname to point to our local address like the output below:

192.168.1.2 myhostname.com

Instead of:

1.2.3.4 myhostname.com

After that the server starts normally:

# ./bin/jmeter-server
Created remote object: UnicastServerRef [liveRef: [endpoint:[192.168.1.2:58807](local),objID:[-5549f37:139ac0cbbed:-7fff, 6781158136036771797]]]

And it's listening on port 1099:

# netstat -landtp
tcp6 0 0 192.168.1.111:1099 192.168.1.111:48900 ESTABLISHED 17995/java

Starting the GUI Client Instance

On the controlling machine, edit the jmeter.properties and change the remote_hosts variable to hold the IP addresses of all remote JMeter instances, like below:

remote_hosts=127.0.0.1:1099, 192.168.1.2:1099

Then we can start the local GUI JMeter normally, but we'll be able to use multiple remote JMeter instances by clicking on Run - Remote Start and choosing the appropriate remote JMeter instance. We can start individual JMeter remote instances or we can start them all at the same time. In any case, each remote JMeter instance will proceed and execute the whole test plan defined in the local GUI JMeter.

For this to work, we need to ensure that a couple of ports are open and not firewalled, regardless of whether we're doing the performance testing on a local network or over the Internet. Each JMeter remote instance will have to have the port 1099 open for the client GUI JMeter to be able to connect to the remote JMeter instance. But in addition to this, a reverse connection is also required to return the results from the remote JMeter instance to the client GUI instance. By default the JMeter uses a random high-numbered port, but this can be adjusted with the client.rmi.localport variable in jmeter.properties.

The following picture summarizes the appropriate variables and ports that need to be open when connecting from localhost GUI JMeter to remote JMeter instance.

Conclusion

We've seen how to determine the best ramp-up period to use when testing the performance of a web page. We also measured the performance of a web site by analyzing the results and looking for the best throughput; the best throughput means the maximum number of requests/minute the web site can handle.

We've also looked at the remote JMeter testing, which we can use to present a bigger load on the testing server.

References

[1] Jmeter threads test failed, http://www.skill-guru.com/blog/2010/08/10/error-jmeter-threads-jmeterthread-test-failed-java-lang-outofmemoryerror-java-heap-space/.

[2] Jmeter threads, http://wiki.apache.org/jmeter/HowManyThreads.