Context Navigation

Using Boost.Test

If you're developing with C++, boost is probably a familiar library. As it so happens, it's also a good match for being paired, quite easily, with bitten!

By roaming around the wiki (here: Data Storage) you can gather how bitten stores reports, and how test reports in particular are stored and used. For each test case a number of properties are stored, such as status and the fixture/test suite it belongs to. This data is fed to bitten by the <report> recipe command (Recipe Commands). We just have to make sure it receives it in a format that it understands.

Gathering the report

Fortunately for us, Boost.Test outputs test reports in XML format, if instructed to do so. Thus, the amount of work we have to carry out is reduced to specifying and applying a simple XSL transform.

But before we proceed to the needed transform we have to actually gather the XML data from boost. Suppose the program that runs your tests is called run_tests, then we can get an XML report by running that program as:

run_tests --report_format=XML --report_level=detailed

Or for more recent versions of boost (tested with 1.41):

run_tests --log_format=XML

The detailed report level is needed for the XSL transform provided in the next section. Furthermore, you have to somehow store this in a file, test_results.xml below. There are several to do this, which I leave up to you to figure out (hint: redirect stderr/cerr or read the boost docs).

Making it bitten-friendly

Now we need to apply the following transform on the report to make it understandable by bitten:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="xml" indent="yes"/> 
 <xsl:template match="/">
    <unit_tests>
       <xsl:apply-templates/>
    </unit_tests>
 </xsl:template>
 <xsl:template match="TestSuite/TestCase">
     <test>
        <status>
	  <xsl:if test="@result = 'passed'">success</xsl:if>
	  <xsl:if test="@result != 'passed'">failure</xsl:if>
	</status>
        <fixture>
	  <xsl:value-of select="../@name" />
        </fixture>
        <type>test</type>
        <name>
	  <xsl:value-of select="@name" />
        </name>
     </test>
  </xsl:template>
</xsl:stylesheet>

Comment from r.wilczek: Better use this version as it supports nested testsuites better and conforms to the documented test report format

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output method="xml" indent="yes" />
    <xsl:template match="/">
        <report category="test">
            <xsl:apply-templates />
        </report>
    </xsl:template>
    <xsl:template match="//TestSuite/TestCase">
        <test>
            <status>
                <xsl:if test="@result = 'passed'">success</xsl:if>
                <xsl:if test="@result != 'passed'">failure</xsl:if>
            </status>
            <fixture><xsl:value-of select="../@name" /></fixture>
            <name><xsl:value-of select="@name" /></name>
        </test>
    </xsl:template>
</xsl:stylesheet>

Use the <x:transform> recipe command to do this. An example on how to use it is below. You will, of course, have to put this in your repository somewhere.

Adjusting your recipe to make delicious Boost.Test soup

Here's a template for how to put all of this together in your build recipe:

<build xmlns:sh="http://bitten.cmlenz.net/tools/sh" xmlns:x="http://bitten.cmlenz.net/tools/xml">
  <step id="test" description="Test - run automated tests">
    <sh:exec executable="run_tests" output="test_results.xml" args="--log_level=nothing --report_format=XML --report_level=detailed" />
    <x:transform src="test_results.xml" dest="test_report.xml" stylesheet="test_results.xslt" />
    <report category="test" file="test_report.xml"/>
  </step>
</build>

Using this approach (using <sh:exec> to redirect the output to a file) your test log won't be very nice looking. You probably want to tweak this template to generate the XML data by other means and instruct it to output nice log-messages.

Appendix A: Redirecting the report data

By default, Boost.Test outputs the report to stderr and the log to stdout. Ideally, we only want to redirect the report data to our "test_results.xml" file.

Exactly how you should do this depends on how you use Boost.Test - i.e. if you supply your own main() function or let Boost.Test take care of that for you. In the latter case, this is one way of doing it:

//
// run_tests.cc
//

#define BOOST_AUTO_TEST_MAIN

#include <iostream>
#include <fstream>
#include <cassert>
#include <boost/test/auto_unit_test.hpp>


// Write cerr to file - fail and cry with no apparent error if we can't
// open the file :)
struct ReportRedirector
{
    std::streambuf *orig;
    std::ofstream out;

    ReportRedirector() : out("test_results.xml")
    {
        assert( out.is_open() );
        orig = std::cerr.rdbuf(out.rdbuf());
    }

    ~ReportRedirector()
    {
        std::cerr.rdbuf(orig);
    }
};

static ReportRedirector foo;

There might be other, prettier ways to do this - please contribute if you have a better solution.

Here's an alternative that IMHO is a little more elegant:

//
// run_tests.cc
//

#define BOOST_AUTO_TEST_MAIN

#include <iostream>
#include <fstream>
#include <cassert>
#include <boost/test/auto_unit_test.hpp>
#include <boost/test/results_reporter.hpp>

std::ofstream out;

struct ReportRedirector
{
    ReportRedirector()
    {
        out.open("test_results.xml");
        assert( out.is_open() );
        boost::unit_test::results_reporter::set_stream(out);
    }
};

BOOST_GLOBAL_FIXTURE(ReportRedirector)

This has been tested for boost 1.34.1.

Once the report has been properly redirected, you can update your recipe to include logging output:

<build xmlns:sh="http://bitten.cmlenz.net/tools/sh" xmlns:x="http://bitten.cmlenz.net/tools/xml">
  <step id="test" description="Test - run automated tests">
    <sh:exec executable="run_tests" args="--log_level=all --report_format=XML --report_level=detailed" />
    <x:transform src="test_results.xml" dest="test_report.xml" stylesheet="test_results.xslt" />
    <report category="test" file="test_report.xml"/>
  </step>
</build>