Calculating the code coverage of integration tests

In our development structure we use an internally developed integration test system, based on Selenium, to test web application user interfaces. It works great. The Selenium tests run against the test environment (usually JBoss, but it can be any application server) and the tests are run periodically by the continuous integration tool, Jenkins. We can even automatically export test results with test steps to Confluence (our wiki) and link the test cases to their specific use cases.

But there was one thing missing: Calculating the code coverage of the integration tests.

We already used Cobertura for unit test coverage, so I started searching for the possibility of using Cobertura for the integration tests as well. And the conclusion is that it is possible:

1. First, build your web application, resulting in a .ear or .war file.
2. Let Cobertura instrument this artifact using the cobertura-instrument command line utility (Cobertura can instrument a .jar or .war, but not .war and .jar files in some .ear file, in this case, extract the .ear file first).
3. Move the cobertura.ser file generated by cobertura-instrument to the directory where your application server runs from. For me, using JBoss 4, this was the bin-directory.
4. Start the application server.
5. Run your Selenium tests against the environment with the instrumented application.
6. Stop the application server (when using JBoss, be sure jboss.shutdown.forceHalt is set to false, otherwise, the cobertura.ser file might not be written completely).
7. Use cobertura-report to generate the XML or HTML report.

And just like that, your integration tests coverage report is generated!

I did not manage to use the report in Jenkins because the Cobertura plugin from Jenkins thinks mojo has to run before a coverage report can be generated... I did manage however to use the coverage result in Sonar. Unfortunately, I could not add the integration tests coverage data to the project also containing the unit tests coverage data. That said, I was able to create a different project in Sonar containing the integration tests coverage using Sonar Runner. The sonar-project.properties file I used looked like this:

sonar.projectKey=my:key
sonar.projectName=Integration Tests
sonar.projectVersion=1.0

tests=src/test/java
sources=../project/src/main/java
sonar.dynamicAnalysis=reuseReports
sonar.cobertura.reportPath=../coverage-report/coverage.xml
sonar.skipDesign=true

There are other tools than Cobertura for calculating code coverage, like JaCoCo for example. JaCoCo might collaborate easier with Sonar when calculating integration tests coverage, but I did not try it.

Hopefully this will help you in determining your integration tests code coverage. Feel free to ask questions, share your experiences or comment!

Get in touch!