Friday 4 November 2011

iOS Code Coverage with Cobertura

Since I wrote this post, XCode has been updated somewhat, more concise xcodebuild commands can be found here:  http://jonboydell.blogspot.co.uk/2013/01/i-have-previously-written-couple-of.html

I am 100% sure that everyone who develops anything for the iOS platform is a Software Engineer and not a Bedroom Based Homebrewer then we all use Continuous Integration to ensure that our builds are never broken and to provide insightful metrics into the quality of our code.

There is a tool called gcovr (https://software.sandia.gov/trac/fast/wiki/gcovr) that can be used to turn the files generated by XCode into something more readable and something that, with the Jenkins Cobertura plugin, you can use to trigger build-fail thresholds in Jenkins and to click through your code base to find out where the unit test coverage is weakest.

gcovr is a python script that converts gcov (the tool that comes with XCode for generating code coverage files) output into a more readable XML format or into command line output.

Firstly

Download gcovr and copy it into somewhere useful on your filesystem.  Then set the following project build settings in XCode:  "Generate Test Coverage Files" to Yes and "Instrument Program Flow" to Yes.  This generates the .gcno and .gcda files needed for gcovr to generate something more readable.

Run gcovr
 
1. Change directory to your project folder (the one where your .xcodeproj file sits)
2. Run:

gcovr -r `pwd` -e .*${TESTTARGET}.* -x > coverage.xml


Where ${TESTTARGET} is the name of the tests target that XCode creates for you when you have the "include unit tests" option ticked on project creation.  This prevents the code coverage of the unit tests being reported.

3. Check the output coverage.xml to make sure that there's something in it that you recognise. (I say this as the best way to check it's worked is to just eyeball the file).


Integrate with Jenkins

My iOS CI works by executing a command line script in a Jenkins job. The script builds my project using some command much like this:

xcodebuild -sdk iphonesimulator -project myProject.xcodeproj -target myProjectTests -configuration Debug clean TEST_AFTER_BUILD=YES TEST_HOST=

You will have had to configure the myProjectTests target to produce GCOV code coverage data. Then to integrate the GCOVR step, add this to your build:

gcovr -r `pwd` --exclude '.*myProjectTests.*' --xml > myProjectDir/coverage.xml

n.b. It's having the correct setting for the -r option that makes the source code highlighting feature of the Cobertura plugin work properly.

Bosh! Now install the Cobertura plugin into Jenkins, enable it for your CI build, check the "Publish Cobertura Code Coverage Reports" and enter "**/coverage.xml" into the "Cobertura xml report pattern" box. Press Save. Run the build. You should now have a "Coverage Report" icon in your Jenkins job view which if you click on it will give you some headline data about your test code coverage plus clickable links to your source code which should give you a detailed breakdown per line of code on its coverage status.

4 comments:

  1. Thanks Jon, really helped out when setting up test coverage on our Jenkins node..
    I noticed that not all files are included in the statistics, a lot of our classes are not included in the report since they're not included in the test-target, only in the main application.
    A solution to this is to set the following values for the Debug-configuration in your main-target:
    Generate Test Coverage Files, Debug: $(GENERATE_COVERAGE)
    Instrument Program Flow, Debug: $(GENERATE_COVERAGE)

    You need to link the profile_rt library as well, this can be done by appending OTHER_LDFLAGS=-lprofile_rt to the xcodebuild command.

    So to generate coverage for all files:
    export GENERATE_COVERAGE=YES
    xcodebuild -target someTargetTests -configuration Debug OTHER_LDFLAGS=-lprofile_rt

    ReplyDelete
  2. Good point on the other linker flag. I had included "/Developer/usr/lib/libprofile_rt.dylib" using the +Frameworks options in XCode. I didn't know there was an LDFLAGS shortcut.

    ReplyDelete
  3. as you mentioned in a newer post http://jonboydell.blogspot.ch/2012/06/xcode-43-and-jenkins-code-test-coverage.html, the setup is now easier in Xcode 4.3 as you don't need to deal with the profile_rt library. Xcode manages it by itself (at last!).

    ReplyDelete
  4. How do I generate HTML report from gcovr coverage.xml using cobertura?

    ReplyDelete