How to test Vaadin web application performance with JMeter

This article describes how to make load testing of your Vaadin web application with JMeter.

Get the latest JMeter

Download JMeter from http://jmeter.apache.org/download_jmeter.cgi

Configure JMeter

Unzip the apache-jmeter-x.x.x.zip file.

Edit JMETERHOME/bin/jmeter.bat (or jmeter.sh) and check that the JVM memory parameters are ok (e.g. set HEAP=-Xms512m -Xmx1500m -Xss128k). The maximum heap size (-Xmx) should be at least 1024m. I would also recommend that the thread stack size is set to 512k or below if a large number of threads is used in testing.

You should read this to ensure you follow best-practices:

Start JMeter

E.g. double clicking jmeter.bat

Configure your Test Plan and WorkBench

Right click the WorkBench icon and select ‘Add’ → ‘Non-Test Elements’ → ‘HTTP(S) Test Script Recorder’. Edit the Recorder parameters and set Port: to e.g. 9999 (you could leave it to 8080 if your web application servers do not use the port 8080). Typically requests related to loading static web content like css files and images are excluded from the load test. You can use ‘Url Patterns to Exclude’ section of the recorder to define what content is excluded. For instance to exclude css files add following pattern: .*\.css

JMeter patterns to exclude

Right click the Recorder icon and select ‘Add’ → ‘Timer’ → ‘Uniform random timer’. Configure timer by setting the Constant Delays Offset into ${T}. This setting means that JMeter records also the delays between the http requests. You could also test without the timer but with the timer your test is more realistic.

JMeter uniform random timer

Optionally you could also add ‘View Result Tree’ listener under the Recorder. With ‘View Result Tree’ listener it is possible to inspect every recorded request and response.

Note since JMeter you can do this in one step using menu item “Templates…​” and selecting “Recording” template.

JMeter View Results Tree

Next, configure the Test Plan. Add a ‘Thread Group’ to it and then add a ‘Config Element’ → ‘HTTP Cookie Manager’ to the thread group. Set Cookie policy of the cookie manager to be ‘compatibility’. Remember also to set the “Clear cookies each iteration” setting to ‘checked’. Add also a ‘Config Element’ → ‘HTTP Request Defaults’ into Thread group.

You could also add a ‘Config Element’ → ‘User Defined Variables’ and a ‘Logic Controller’ → ‘Recording Controller’ into Thread Group.

Your JMeter should now looks something like the screenshot below:

JMeter User Defined Variables

Configure your Vaadin application

Disable the xsrf-protection

In Vaadin you have to disable the xsrf-protection of your application or otherwise the JMeter test may fail. The way how xsrf protection is disabled differs in Vaadin 6 and Vaadin 7.

In Vaadin 7

If you use web.xml in your Vaadin 7 project, add the following context-parameter in the web.xml or optionally add it as an init parameter just like in the Vaadin 6 project below.

XML

  1. <context-param>
  2.   <param-name>disable-xsrf-protection</param-name>
  3.   <param-value>true</param-value>
  4. </context-param>

If you use annotation based (Servlet 3.0) Vaadin servlet configuration, you can currently (in Vaadin 7.1.11) either fall back to Servlet 2.4 web.xml configuration or set the parameter value for ‘disable-xsrf-protection’ as a java.lang.System property before the Vaadin’s DefaultDeploymentConfiguration is loaded. This can be done for example by extending VaadinServlet class. At the end of this Wiki article there is an example servlet (JMeterServlet) that implements this functionality. See the example code below for how to replace the default VaadinServlet with your custom VaadinServlet in the UI class.

Java

  1. public class YourUI extends UI {
  3.   @WebServlet(value = "/*", asyncSupported = true)
  4.   @VaadinServletConfiguration(productionMode = false, ui = YourUI.class)
  5.   public static class Servlet extends JMeterServlet {
  6.   }
  8.   @Override
  9.   protected void init(VaadinRequest request) {
  10.     //...
  11.   }

In Vaadin 6

See the example below for how to disable the protection from the web.xml file:

XML

  1. <servlet>
  2.   <servlet-name>FeatureBrowser</servlet-name>
  3.   <servlet-class>com.vaadin.terminal.gwt.server.ApplicationServlet</servlet-class>
  4.   <init-param>
  5.     <param-name>application</param-name>
  6.     <param-value>com.vaadin.demo.featurebrowser.FeatureBrowser</param-value>
  7.   </init-param>
  9.   <init-param>
  10.     <param-name>disable-xsrf-protection</param-name>
  11.     <param-value>true</param-value>
  12.   </init-param>
  13. </servlet>

Important! Remember to enable the protection after the testing is done!

Disabling syncId happens with similar parameter

XML

  1. <context-param>
  2.   <param-name>syncId</param-name>
  3.   <param-value>false</param-value>
  4. </context-param>

If you want to do the above with Java Servlet 3.0 annotations, use the following:

Java

  1. initParams = {
  2.   @WebInitParam(name = "disable-xsrf-protection", value = "true"),
  3.   @WebInitParam(name = "syncIdCheck", value = "false")}

Use debug id:s within your Vaadin application

Normally a Vaadin application sets a sequential id for each user interface component of the application. These ids are used in the ajax-requests when the component state is synchronized between the server and the client side. The aforementioned id sequence is likely the same between different runs of the application, but this is not guaranteed. In Vaadin 6 these ids can be manually set by calling setDebugId() method.

In Vaadin 7 there no more exists a setDebugId() method; instead there is setId()) method. Unfortunately this method won’t set component ids used in the ajax-request. Therefore, by default, JMeter tests of a Vaadin 7 application are not stable to UI changes. To overcome this problem you can use our JMeterServlet (see the end of this article) instead of the default VaadinServlet. When using the JMeterServlet component ids are again used in the ajax requests. See example above for how to replace default VaadinServlet with JMeterServlet. For additional information, see the Vaadin ticket #13396.

Use named windows in your application

Setting the name for the Windows in the Vaadin (< 6.4.X) application is important since otherwise these names are randomly generated. Window name could be set using the setName()-method.

Configure your browser

Since JMeter is used as a proxy server, you have to configure the proxy settings of your browser. You can find the proxy settings of Firefox from Tools → Options → Connections → Settings: ‘Manual proxy configuration’. Set the correct IP of your computer (or ‘localhost’ string) and the same port that you set into proxy server settings above.

Start recording

Start your web application server. Start the proxy server from the JMeter. Open the URL of your web application into the browser configured above. You should append ?restartApplication to the URL used when recording the tests to make sure that the UI gets initialized properly. Thus the URL becomes something like (http://localhost:8080/test/TestApplication/?restartApplication). If everything is ok your web application opens normally and you can see how the different HTTP requests appear into JMeter’s thread group (see screenshot below). When you have done the recording, stop the proxy server.

JMeter Thread Groups

Performance testing

Clean up the recorded request

Before you start the test, you may have to delete the first timer object which is located below the first HTTP request in the thread group since its time delay may be unrealistically big (see the screenshot above). It is also very much recommended to check the recorded data and delete all unessential requests.

Detecting Out of Sync errors

If your test results in the application being in an Out of Sync error state it is not by default detected by JMeter (because the response code is still HTTP/1.1 200 OK). To make an assertion for detecting this kind of error you should add a Response Assertion to your test plan. Right-click on the thread group and select Add → Assertions → Response Assertion. Configure the assertion to assert that the Text Response does NOT contain a pattern “Out of sync”.

Optional parameterization of the request

Sometimes, it is useful to parameterize the recorded requests. Parameterization of a request is easily done in JMeter:

  1. add a “User Defined Variables”-element into the first place of your Test Plan.

  2. Copy paste the whole parameter value of wanted UIDL-request into the newly made variable (e.g. PARAM1).

  3. Replace the value of the UIDL-request with the parameter reference (e.g. ${PARAM1}).

Start testing

Now, it is time to do the actual testing. Configure the thread group with proper ‘Number of Threads’ (e.g. 100) and set also the ‘Ramp-Up Period’ to some realistic value (e.g. 120). Then, add e.g. ‘Listener’ → ‘Graph Results’ to monitor how your application is performing. Finally, start the test from the Run → Start.

Stop on Error

When you are pushing your Vaadin application to the limits, you might get into a situation where some of the UIDL requests fail. Because of the server-driven nature of Vaadin, it’s likely that subsequent requests will cause errors like “Warning: Ignoring variable change for non-existent component“, as the state stored on the server-side is no longer in sync with the JMeter test script. In these cases, it’s often best to configure your JMeter thread group to stop the thread on sampler error. However, if you have configured your test to loop, you might want to still continue (and ignore the errors), if the next iteration will start all over again with fresh state.

Continuous Integration

If you want to integrate load testing in your CI, you can use this plugin.

You can read this for full integration with Jenkins :

JMeterServlet

In Vaadin 7 we recommend using the following or similar customized VaadinServlet.

Java

  1. package com.example.vaadin7jmeterservlet;
  3. import com.vaadin.server.ClientConnector;
  4. import com.vaadin.server.DeploymentConfiguration;
  5. import com.vaadin.server.ServiceException;
  6. import com.vaadin.server.VaadinRequest;
  7. import com.vaadin.server.VaadinService;
  8. import com.vaadin.server.VaadinServlet;
  9. import com.vaadin.server.VaadinServletService;
  10. import com.vaadin.server.VaadinSession;
  11. import com.vaadin.ui.Component;
  13. /**
  14.  * @author Marcus Hellberg (marcus@vaadin.com)
  15.  *  Further modified by Johannes Tuikkala (johannes@vaadin.com)
  16.  */
  17. public class JMeterServlet extends VaadinServlet {
  18.   private static final long serialVersionUID = 898354532369443197L;
  20.   public JMeterServlet() {
  21.     System.setProperty(getPackageName() + "." + "disable-xsrf-protection",
  22.             "true");
  23.   }
  25.   @Override
  26.   protected VaadinServletService createServletService(
  27.           DeploymentConfiguration deploymentConfiguration)
  28.           throws ServiceException {
  29.     JMeterService service = new JMeterService(this, deploymentConfiguration);
  30.     service.init();
  32.     return service;
  33.   }
  35.   private String getPackageName() {
  36.     String pkgName;
  37.     final Package pkg = this.getClass().getPackage();
  38.     if (pkg != null) {
  39.       pkgName = pkg.getName();
  40.     } else {
  41.       final String className = this.getClass().getName();
  42.       pkgName = new String(className.toCharArray(), 0,
  43.             className.lastIndexOf('.'));
  44.     }
  45.     return pkgName;
  46.   }
  48.   public static class JMeterService extends VaadinServletService {
  49.     private static final long serialVersionUID = -5874716650679865909L;
  51.     public JMeterService(VaadinServlet servlet,
  52.         DeploymentConfiguration deploymentConfiguration)
  53.         throws ServiceException {
  54.       super(servlet, deploymentConfiguration);
  55.     }
  57.     @Override
  58.     protected VaadinSession createVaadinSession(VaadinRequest request)
  59.         throws ServiceException {
  60.       return new JMeterSession(this);
  61.     }
  62.   }
  64.   public static class JMeterSession extends VaadinSession {
  65.     private static final long serialVersionUID = 4596901275146146127L;
  67.     public JMeterSession(VaadinService service) {
  68.       super(service);
  69.     }
  71.     @Override
  72.     public String createConnectorId(ClientConnector connector) {
  73.       if (connector instanceof Component) {
  74.         Component component = (Component) connector;
  75.         return component.getId() == null ? super
  76.             .createConnectorId(connector) : component.getId();
  77.       }
  78.       return super.createConnectorId(connector);
  79.     }
  80.   }
  81. }