Clang Static Analysis

Producing quality software is, or should be, the goal of any programmer.  Whether you're sat in your bedroom boshing out code left and right to get your smart idea off the ground, or whether you're sat where I am right now, in the offices of an digital advertising agency trying to get the most and best out of the guys sat next to me.

One good way of checking that the code you've written isn't full of unintended side effects is to run the static analyser (http://clang-analyzer.llvm.org/index.html) over it.  You can do this in XCode by using cmd+shift+B to build your project.  It will report any "infringements" that it feels you have made.  It is not flawless and may report false positives (here's a bit on that: http://xcodebook.com/2011/08/handling-false-positives-with-the-static-analyzer/).  You can also change your target settings to run the analyser automatically on build (the build option is "Run Static Analyzer").

The static analyser will run if you build your project from the command line using xcodebuild (and there's an option RUN_CLANG_STATIC_ANALYZER that can be set to YES if you want).  The warning appears within the compilation output seemingly cannot be converted to an error (even if you set "Treat Warnings as Errors" in your target settings).

There is a tool called scan-build that's part of LLVM that does the same thing but also produces readable HTML output.


The --status-bugs flag ensures that you get a non-0 exit code if any problems are found by the static analyzer (and the command echo $? prints out the exit code from the last command entered).

There's a Jenkins plugin (https://wiki.jenkins-ci.org/display/JENKINS/Clang+Scan-Build+Plugin) you can use where it'll fail the build if a certain number of problems are found.  However, I've never used this plugin and it hasn't been updated since April 2012 (which may or may not mean anything).

The LLVM project version of the static analyzer is the bleeding-edge, both in terms of development builds (you can get the tip of the tree) and stable builds.  Apple doesn't always update XCode when a new version of LLVM or the static analyser are released.

A Return to Unit Testing using xcodebuild

I have previously written a couple of posts on the testing of iOS applications by executing command line only commands. What I did:

xcodebuild -sdk iphonesimulator -project JonsProject.xcodeproj -target JonsProjectTest -configuration Debug TEST_AFTER_BUILD=YES

This used to fire up the simulator and execute the test schema. Now I get this:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

Partly I think that I might have lost my mind as the tests worked, I saw them work, they worked!  However, the build server on which all of this worked now does not, but then the scripts for the CI

have changed. So, which stopped working first? Who knows, but it's not adding to my positive mental health.

Anyway at least one of my colleagues has been suggesting that we use this run the simulator and execute the tests:

https://github.com/sgleadow/xcodetest

I am not sure that this is a wonderful idea as it appears to involve compiled in binaries. Which I don't think I agree with. You shouldn't have to alter your code to use what is a base "feature" of the IDE.

After a bit of poking around the script that gets executed there are these blocks of code:

RunTestsForApplication() {
    Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."
}


and 

if [ "${TEST_HOST}" != "" ]; then
    # All applications are tested the same way, by injecting a bundle.
    # The bundle needs to configure and run the tests itself somehow.
    RunTestsForApplication "${TEST_HOST}" "${TEST_BUNDLE_PATH}"

else
    # If no TEST_HOST is specified, assume we're running the test bundle.
   
    RunTestsForBundle "${TEST_BUNDLE_PATH}"
fi


Which kind of implies that RunTestsForApplication will be supported in the future and is somehow related to the setting of the TEST_HOST environment variable.  But what happens if you set that variable to blank?  Well, given this command:

xcodebuild -sdk iphonesimulator -project JonsProject.xcodeproj -target JonsProjectTests -configuration Debug  TEST_AFTER_BUILD=YES TEST_HOST=""

The unit tests run.  And fail, correctly, in my case.

error: -[JonsProjectTests testExample] : Unit tests are not implemented yet in Jons Password KeeperTests
Test Case '-[JonsProjectTests testExample]' failed (0.000 seconds).

Test Suite 'JonsProjectTests' finished at 2013-01-28 10:25:32 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.000) seconds

Test Suite '/build/Debug-iphonesimulator/JonsProjectTests.octest(Tests)' finished at 2013-01-28 10:25:32 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.001) seconds

But, but, but the simulator isn't open.  I remember, distinctly, that to get this lot working in the past, the simulator HAD to be open.

I am so losing my mind.

XCode 4.3 and Jenkins Code Test Coverage Updated

Since I wrote this post, XCode has been updated somewhat, more concise xcodebuild commands can be found here:  http://jonboydell.blogspot.co.uk/2013/01/i-have-previously-written-couple-of.html

Another trip to the US under my belt.  That makes five trips in the last twelve months.  The only thing I'll say on the matter is that flying to and from JFK-LHR is so much easier than flying PDX-SEATAC-LHR.  Five and a half hours on the way back is just a ridiculous fast flight time to get thousands of miles across the Atlantic.  That coupled with Silver BA Executive Club, almost guaranteed upgrades to Club on at least one leg of each flight makes the whole business almost bearable.  The only problem, of course, is that if the ride's bumpy it's bumpy for everyone from Cattle to First.

Anyways.

So, XCode 4.3 makes the generation of code coverage files so much easier and this updates my previous post on the topic.  Specifically, this is geared towards doing CI builds of XCode projects from the command line.


XCode:

1. Make sure you have a Test target set up in your project
2. Select it from your project settings TARGETS list
3. Go "Build Settings"
4. Set "Generate Test Coverage Files" and "Instrument Program Flow" to "Yes"

5. Optional - I don't use the "Test After Build" project setting in my main project targe to trigger the tests as I only want code coverage running on my central CI due to my desire to integrate exclusively with Cobertura.
 
Make sure that your CI build has the following command/s in it:

# Build and test

xcodebuild -sdk iphonesimulator -project MyProject.xcodeproj -target MyProjectTests -configuration Debug clean build TEST_AFTER_BUILD=YES

# Generate code coverage report.
ROOT=`pwd`
/path/to/gcovr -r $ROOT --exclude '.*Developer.*' --exclude '.*MyProjectTests.*' --xml > MyProject/coverage.xml


The final bit is my hook in for GCOVR to generate Cobertura reports.

Debugging Android Emulator HTTP/S Traffic

The project that I am involved in at work is very network centric and I found my team having to support a third party using the Android SDK to develop an application that used the services exposed by my project.  One of the most important tasks anyone on my team can do is debug the network requests being sent by the third party application to the network services.

The Android Emulator is not the friendliest in the universe.  But..

Download Charles - A very useful tool (for OSX)

Using Charles (I have a Mac)
1. Go - Proxy -> Proxy Settings -> Enable Transparent HTTP Proxying
2. Proxy -> Proxy Settings -> SSL -> Enable SSL Proxying (if that's what you need)
3. Proxy -> Proxy Settings -> SSL -> Add : Your service end point here!
4. Help -> Local IP Address : Write this down

From the Emulator
1. Open a Web Browser -> http://www.charlesproxy.com/charles.crt -> Save to Device -> Set up PIN (if you're doing SSL debugging)
2. Settings -> Wireless & Networks (Maybe -> More) -> Mobile Networks -> Access Point Names -> TelKila -> Proxy : The IP Address you wrote down
3. Settings -> Wireless & Networks (Maybe -> More) -> Mobile Networks -> Access Point Names -> TelKila -> Port : 8888 (unless you've changed the default port for Charles)
 
All Set!

iOS Code Coverage with Cobertura

Since I wrote this post, XCode has been updated somewhat, more concise xcodebuild commands can be found here:  http://jonboydell.blogspot.co.uk/2013/01/i-have-previously-written-couple-of.html

I am 100% sure that everyone who develops anything for the iOS platform is a Software Engineer and not a Bedroom Based Homebrewer then we all use Continuous Integration to ensure that our builds are never broken and to provide insightful metrics into the quality of our code.

There is a tool called gcovr (https://software.sandia.gov/trac/fast/wiki/gcovr) that can be used to turn the files generated by XCode into something more readable and something that, with the Jenkins Cobertura plugin, you can use to trigger build-fail thresholds in Jenkins and to click through your code base to find out where the unit test coverage is weakest.

gcovr is a python script that converts gcov (the tool that comes with XCode for generating code coverage files) output into a more readable XML format or into command line output.

Firstly

Download gcovr and copy it into somewhere useful on your filesystem.  Then set the following project build settings in XCode:  "Generate Test Coverage Files" to Yes and "Instrument Program Flow" to Yes.  This generates the .gcno and .gcda files needed for gcovr to generate something more readable.

Run gcovr
 
1. Change directory to your project folder (the one where your .xcodeproj file sits)
2. Run:

gcovr -r `pwd` -e .*${TESTTARGET}.* -x > coverage.xml


Where ${TESTTARGET} is the name of the tests target that XCode creates for you when you have the "include unit tests" option ticked on project creation.  This prevents the code coverage of the unit tests being reported.

3. Check the output coverage.xml to make sure that there's something in it that you recognise. (I say this as the best way to check it's worked is to just eyeball the file).


Integrate with Jenkins

My iOS CI works by executing a command line script in a Jenkins job. The script builds my project using some command much like this:

xcodebuild -sdk iphonesimulator -project myProject.xcodeproj -target myProjectTests -configuration Debug clean TEST_AFTER_BUILD=YES TEST_HOST=

You will have had to configure the myProjectTests target to produce GCOV code coverage data. Then to integrate the GCOVR step, add this to your build:

gcovr -r `pwd` --exclude '.*myProjectTests.*' --xml > myProjectDir/coverage.xml

n.b. It's having the correct setting for the -r option that makes the source code highlighting feature of the Cobertura plugin work properly.

Bosh! Now install the Cobertura plugin into Jenkins, enable it for your CI build, check the "Publish Cobertura Code Coverage Reports" and enter "**/coverage.xml" into the "Cobertura xml report pattern" box. Press Save. Run the build. You should now have a "Coverage Report" icon in your Jenkins job view which if you click on it will give you some headline data about your test code coverage plus clickable links to your source code which should give you a detailed breakdown per line of code on its coverage status.

XCode, iOS and Code Coverage

Since I wrote this post, XCode has been updated somewhat, more concise xcodebuild commands can be found here:  http://jonboydell.blogspot.co.uk/2013/01/i-have-previously-written-couple-of.html 

There are a lot of smart people who use the internet to publish the answers to problems that I encounter, mostly whilst at work, but sometimes when I want to catch a mouse or sew a button back onto my jeans. These people usually post, in one easily Googled page, step-by-step instructions to be repeated back much like a spell. Step 1 to Step X and your mouse is caught, your button sewn or a snippet of code copied and pasted into a project on which I am working from someones blog (attribution optional).

Today though, I done some discovering of my own and am now reasonably assured that I have got XCode running a code coverage tool of some form and producing some results in which I have some confidence.

So, what's the spell?

1. Create (if you haven't already) a testing target for your project
2. Change the target (just the testing target, not the whole project) options as follows:
2a. "Compiler for C/C++/Objective-C" to "Apple LLVM Compiler 3.0"
2b. "Generate Test Coverage Files" to "Yes"
2c. "Instrument Program Flow" to "Yes"
3. No other compiler or linker flags, and indeed, some linker flags may make the code coverage generator go wonky (specifically -all_load), I got a "402" error at compile time in one project and a break point error (at the same place) in another:

error: fopen$UNIX2003 called from function llvm_gcda_start_file

4. Add the following library to the "Build Phases" of the test target. That is:
4a. Go test target settings then "Build Phases" tab
4b. Go "Link Binary With Libraries"
4c. Go "+", then "Add Other.."
4d. Go "/Developer/usr/lib/libprofile_rt.dylib" which will add it to the list of things to link

Should work, spangle your "Test" schema. See what happens. This will generate a bunch of .gcda files in the build directory of your project. Which is hard to find. Try going "Organiser" then "Projects" (select your project), then click on the tiny grey arrow next to the "Derived Data" line (not the delete button). Then from the finder window, go "Build", "Intermediates", , "Debug-iphonesimulator", .build, "Objects-normal", .

If you're into a bit more than that, get the gcovr python script from here: https://software.sandia.gov/trac/fast/downloader (it's in the FAST library, under scripts, documentation here: https://software.sandia.gov/trac/fast) and run it from the ridiculous build directory above, will give you a pretty print of your code coverage. And if you're into a bit more than that have gcover generate xml and stuff it into Cobertura (gcovr --xml >/somewhere/useful/no/really.xml)

(I am using XCode 4.2 with iOS 5.0)