Next Studio : Testing CakePHP Projects Using Hudson

I'd thought, when typing essentially the title of this post into Google, that I'd readily find someone who'd already figured everything out. Unfortunately, there's mostly just questions.

Getting JUnit XML out of CakePHP

Cake's test results are available from the Cake Console:

$ cake/console/cake testsuite app all

This is initially promising, but unfortunately if you're to link this up with Hudson (and probably any other CI package) you'll want JUnit-style XML output from the test suite. We extend CakeBaseReporter to create an XML reporter which is more or less identical to SimpleTest's standard XmlReporter. Then, thanks to this excellent post, we transform SimpleTest XML output into something Hudson will understand. This is all bundled into a new Cake Shell which extends core TestSuiteShell in /vendors/shells/. The relevant invocation of testing and XSLT goes like so:

	$Reporter = new AppXmlReporter('utf-8', array(
		'app' => $this->Manager->appTest,
		'plugin' => $this->Manager->pluginTest,
		'group' => ($this->type === 'group'),
		'codeCoverage' => $this->doCoverage
	));
	ob_start();
	try {
		$result = $this->Manager->runAllTests($Reporter);
	} catch(Exception $e) {
		ob_get_clean();
		$this->out('Unit tests failed to run.');
		$this->_stop(1);
	}
	$xml = ob_get_clean();

	$xmlD = new DOMDocument();
	$xmlD->loadXML($xml);

	$xslt = new XSLTProcessor();
	$XSL = new DOMDocument();
	$XSL->load(dirname(__FILE__).DS.'xsl'.DS.'to-junit.xsl');
	$xslt->importStylesheet($XSL);
	$out = $xslt->transformToXML($xmlD);

Now with a call to Cake Console you should be spewing out XML test results. This should be very easy to wrap into a build task and thus have test results being exported during every build.

Getting XML into Hudson

Thankfully this part is a little easier. We've configured Hudson to run tests in downstream projects of our main project builds. The 'Execute Shell' build option is used with the following:

$WORKSPACE/cake/console/cake testreport app all

Then in Post-build Actions, you would like to [x] Publish JUnit test result report. Point Hudson at the XML which was output from the Console call above, which you'll need to be writing to disk somewhere in your build.

Aggregating Results in Hudson

The final, optional step is propagating test results back upstream to the parent project. We didn't find this strictly necessary, as when you have a single downstream test project, there isn't a huge benefit in aggregating downstream results to the parent. I was on a Hudson kick, however. Despite the notable absence of any useful error messages like "hey, aggregation isn't going to work because you aren't using fingerprinting", I pressed on.

Hudson might have an idea of heirarchy between projects with the upstream/downstream relationship, but it has no way of tying builds together between these projects, since each project can be built independently. Without the ability to draw relationships between specific builds of related projects, aggregating test results will not work. Switching on 'fingerprinting' is the key, which is the fancypants way of saying "please md5/sha/hash-of-choice the following files: ...". Hudson wants a file(s) which will change uniquely between each build to enable this feature. This is a little awkward for a CakePHP project, since there is no single changing artifact that a Java project might have (e.g. project.jar). We chose to write out a file with some identifying content during each build. This gives us the additional advantage of being able to track down builds running on production machines more easily by looking at the output, generated via shell script thusly:

FINGERPRINT_FILE="../fingerprint"
SVNPATH="`svn info ../. | grep URL | awk '{print $2}'`"
DATE="`date`"
BUILDER="`whoami`"
HOSTNAME="`hostname`"

# if fingerprint file exists and is a file, remove it
if [ -e "$FINGERPRINT_FILE" ] && [ -f "$FINGERPRINT_FILE" ]; then
	echo "Removing and recreating fingerprint file"
	rm "$FINGERPRINT_FILE"
	touch "$FINGERPRINT_FILE"
else
	echo "Creating new fingerprint file"
	touch "$FINGERPRINT_FILE"
fi

if [ -e "$FINGERPRINT_FILE" ] && [ -f "$FINGERPRINT_FILE" ]; then
	svn info "$SVNPATH" >> "$FINGERPRINT_FILE"
	echo "Time of Build: $DATE" >> "$FINGERPRINT_FILE"
	echo "User Running Build: $BUILDER" >> "$FINGERPRINT_FILE"
	echo "Hostname: $HOSTNAME" >> "$FINGERPRINT_FILE"
else
	echo "Failed to create fingerprint file for output"
	exit 1
fi

exit 0

Then under the Hudson project settings we ask Hudson to fingerprint this fingerprint file. This, in addition to the obvious step of checking 'Aggregate downstream test results' in the parent project, leads to proper aggregation of test results.