Wednesday 6 February 2013

Clang Static Analysis

Producing quality software is, or should be, the goal of any programmer.  Whether you're sat in your bedroom boshing out code left and right to get your smart idea off the ground, or whether you're sat where I am right now, in the offices of an digital advertising agency trying to get the most and best out of the guys sat next to me.

One good way of checking that the code you've written isn't full of unintended side effects is to run the static analyser (http://clang-analyzer.llvm.org/index.html) over it.  You can do this in XCode by using cmd+shift+B to build your project.  It will report any "infringements" that it feels you have made.  It is not flawless and may report false positives (here's a bit on that: http://xcodebook.com/2011/08/handling-false-positives-with-the-static-analyzer/).  You can also change your target settings to run the analyser automatically on build (the build option is "Run Static Analyzer").

The static analyser will run if you build your project from the command line using xcodebuild (and there's an option RUN_CLANG_STATIC_ANALYZER that can be set to YES if you want).  The warning appears within the compilation output seemingly cannot be converted to an error (even if you set "Treat Warnings as Errors" in your target settings).

There is a tool called scan-build that's part of LLVM that does the same thing but also produces readable HTML output.


The --status-bugs flag ensures that you get a non-0 exit code if any problems are found by the static analyzer (and the command echo $? prints out the exit code from the last command entered).

There's a Jenkins plugin (https://wiki.jenkins-ci.org/display/JENKINS/Clang+Scan-Build+Plugin) you can use where it'll fail the build if a certain number of problems are found.  However, I've never used this plugin and it hasn't been updated since April 2012 (which may or may not mean anything).

The LLVM project version of the static analyzer is the bleeding-edge, both in terms of development builds (you can get the tip of the tree) and stable builds.  Apple doesn't always update XCode when a new version of LLVM or the static analyser are released.

Monday 28 January 2013

A Return to Unit Testing using xcodebuild

I have previously written a couple of posts on the testing of iOS applications by executing command line only commands. What I did:

xcodebuild -sdk iphonesimulator -project JonsProject.xcodeproj -target JonsProjectTest -configuration Debug TEST_AFTER_BUILD=YES

This used to fire up the simulator and execute the test schema. Now I get this:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/Tools/RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

Partly I think that I might have lost my mind as the tests worked, I saw them work, they worked!  However, the build server on which all of this worked now does not, but then the scripts for the CI

have changed. So, which stopped working first? Who knows, but it's not adding to my positive mental health.

Anyway at least one of my colleagues has been suggesting that we use this run the simulator and execute the tests:

https://github.com/sgleadow/xcodetest

I am not sure that this is a wonderful idea as it appears to involve compiled in binaries. Which I don't think I agree with. You shouldn't have to alter your code to use what is a base "feature" of the IDE.

After a bit of poking around the script that gets executed there are these blocks of code:

RunTestsForApplication() {
    Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."
}



and 

if [ "${TEST_HOST}" != "" ]; then
    # All applications are tested the same way, by injecting a bundle.
    # The bundle needs to configure and run the tests itself somehow.

    RunTestsForApplication "${TEST_HOST}" "${TEST_BUNDLE_PATH}"

else
    # If no TEST_HOST is specified, assume we're running the test bundle.
   
    RunTestsForBundle "${TEST_BUNDLE_PATH}"
fi



Which kind of implies that RunTestsForApplication will be supported in the future and is somehow related to the setting of the TEST_HOST environment variable.  But what happens if you set that variable to blank?  Well, given this command:

xcodebuild -sdk iphonesimulator -project JonsProject.xcodeproj -target JonsProjectTests -configuration Debug  TEST_AFTER_BUILD=YES TEST_HOST=""

The unit tests run.  And fail, correctly, in my case.

error: -[JonsProjectTests testExample] : Unit tests are not implemented yet in Jons Password KeeperTests
Test Case '-[JonsProjectTests testExample]' failed (0.000 seconds).

Test Suite 'JonsProjectTests' finished at 2013-01-28 10:25:32 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.000) seconds

Test Suite '/build/Debug-iphonesimulator/JonsProjectTests.octest(Tests)' finished at 2013-01-28 10:25:32 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.001) seconds


But, but, but the simulator isn't open.  I remember, distinctly, that to get this lot working in the past, the simulator HAD to be open.

I am so losing my mind.