Edgewall Software

Changes between Initial Version and Version 1 of Build Output Processing


Ignore:
Timestamp:
Jul 22, 2005, 6:11:30 PM (19 years ago)
Author:
cmlenz
Comment:

Some thoughts on build output processing

Legend:

Unmodified
Added
Removed
Modified

  • Build Output Processing

    v1 v1  
     1= Bitten Build Output Processing =
     2
     3An important characteristic of any continuous integration system is how it processes the output of a build. Output can be provided as text messages written to the console by the build tool and the commands it executes, but also as files created by the build process. This page discusses the former.
     4
     5== Examples of Build Output ==
     6
     7The format of textual messages is specific to the build tool in use. Following are a couple of examples of output by build tools such as Make and Ant.
     8
     9=== Make ===
     10
     11For example, the output of a Make-driven build of a C project (in this case, [http://www.vim.org/ vim]) might look like this:
     12{{{
     13make[1]: Entering directory `/var/tmp/portage/vim-6.3.075/work/vim63/src'
     14gcc -c -I. -Iproto -DHAVE_CONFIG_H     -O2 -mcpu=i686 -fomit-frame-pointer   -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm  -I/usr/lib/perl5/5.8.6/i686-linux/CORE  -I/usr/include/python2.3 -pthread  -I/usr/lib/ruby/1.8/i686-linux  -o objects/buffer.o buffer.c
     15gcc -c -I. -Iproto -DHAVE_CONFIG_H     -O2 -mcpu=i686 -fomit-frame-pointer   -pipe -D_LARGEFILE_SOURCE -
     16...
     17gcc -c -I. -Iproto -DHAVE_CONFIG_H     -O2 -mcpu=i686 -fomit-frame-pointer   -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm  -I/usr/lib/perl5/5.8.6/i686-linux/CORE  -I/usr/include/python2.3 -pthread  -I/usr/lib/ruby/1.8/i686-linux  -o objects/if_python.o if_python.c
     18In file included from /usr/include/python2.3/Python.h:8,
     19                 from if_python.c:43:
     20/usr/include/python2.3/pyconfig.h:856:1: warning: "_POSIX_C_SOURCE" redefined
     21In file included from /usr/include/stdio.h:28,
     22                 from os_unix.h:21,
     23                 from vim.h:195,
     24                 from if_python.c:20:
     25/usr/include/features.h:190:1: warning: this is the location of the previous definition
     26...
     27  gcc   -rdynamic -Wl,-export-dynamic  -rdynamic   -L/usr/local/lib  -o vim objects/buffer.o objects/charset.o objects/diff.o objects/digraph.o objects/edit.o objects/eval.o objects/ex_cmds.o objects/ex_cmds2.o objects/ex_docmd.o objects/ex_eval.o objects/ex_getln.o objects/fileio.o objects/fold.o objects/getchar.o  objects/if_cscope.o objects/if_xcmdsrv.o objects/main.o objects/mark.o objects/memfile.o objects/memline.o objects/menu.o objects/message.o objects/misc1.o objects/misc2.o objects/move.o objects/mbyte.o objects/normal.o objects/ops.o objects/option.o objects/os_unix.o objects/pathdef.o objects/quickfix.o objects/regexp.o objects/screen.o objects/search.o objects/syntax.o  objects/tag.o objects/term.o objects/ui.o objects/undo.o objects/window.o  objects/if_perl.o objects/if_perlsfio.o objects/if_python.o objects/py_config.o  objects/if_ruby.o   objects/netbeans.o  objects/version.o       -lnsl   -lncurses -lgpm -ldl  -rdynamic  -L/usr/local/lib /usr/lib/perl5/5.8.6/i686-linux/auto/DynaLoader/DynaLoader.a -L/usr/lib/perl5/5.8.6/i686-linux/CORE -lperl -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc -L/usr/lib/python2.3/config -lpython2.3 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic  -Wl,-R -Wl,/usr/lib -L/usr/lib -L/usr/lib -lruby18 -lpthread -ldl -lcrypt -lm   
     28/usr/lib/python2.3/config/libpython2.3.a(posixmodule.o)(.text+0x4259): In function `posix_tempnam':
     29: warning: the use of `tempnam' is dangerous, better use `mkstemp'
     30/usr/lib/python2.3/config/libpython2.3.a(posixmodule.o)(.text+0x432e): In function `posix_tmpnam':
     31: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
     32}}}
     33
     34This text is the output from make, the compiler and the linker. Most of the output shows what the build is currently doing, but some of the messages are warnings (commonly written to the standard error stream). Not shown here are errors that completely terminate the build.
     35
     36=== Ant ===
     37
     38For a Java project built with [http://ant.apache.org/ Ant], the output is quite different. First, the output from each target is clearly separated from the previous target. Also, the output of every task is prefixed with the task name in square brackets.
     39
     40{{{
     41prepare:
     42    [mkdir] Created dir: /usr/local/src/commons-collections-3.1/build
     43
     44compile:
     45    [mkdir] Created dir: /usr/local/src/commons-collections-3.1/build/classes
     46    [javac] Compiling 267 source files to /usr/local/src/commons-collections-3.1/build/classes
     47    [javac] /usr/local/src/commons-collections-3.1/src/java/org/apache/commons/collections/CursorableLinkedList.java:1197: warning: org.apache.commons.collections.CursorableLinkedList in org.apache.commons.collections has been deprecated
     48    [javac] class CursorableSubList extends CursorableLinkedList implements List {
     49    [javac]                                 ^
     50    [javac] /usr/local/src/commons-collections-3.1/src/java/org/apache/commons/collections/CursorableLinkedList.java:1201: warning: org.apache.commons.collections.CursorableLinkedList in org.apache.commons.collections has been deprecated
     51    [javac]     CursorableSubList(CursorableLinkedList list, int from, int to) {
     52    [javac]                       ^
     53...
     54    [javac] 9 warnings
     55
     56jar:
     57    [mkdir] Created dir: /usr/local/src/commons-collections-3.1/build/classes/META-INF
     58     [copy] Copying 1 file to /usr/local/src/commons-collections-3.1/build/classes/META-INF
     59     [copy] Copying 1 file to /usr/local/src/commons-collections-3.1/build/classes/META-INF
     60    [mkdir] Created dir: /usr/local/src/commons-collections-3.1/build/conf
     61     [copy] Copying 1 file to /usr/local/src/commons-collections-3.1/build/conf
     62      [jar] Building jar: /usr/local/src/commons-collections-3.1/build/commons-collections-3.1.jar
     63
     64BUILD SUCCESSFUL
     65Total time: 9 seconds
     66}}}
     67 ''(this is from the build of Apache [http://jakarta.apache.org/commons/collections/ Commons Collections])''
     68
     69The output here also includes informational as well as status messages. Error messages are written to the standard error stream, but warnings go to standard output. The output from Ant can also be modified in a couple of ways: for example, there's an {{{-emacs}}} option for generating the output in a less verbose format that is better handled by Emacs. Also, a custom logger can be specified using the {{{-logger}}} option, which can be used to output messages in a completely different format. For exampe, Ant includes an {{{XmlLogger}}} class that logs the output in XML:
     70
     71{{{
     72<?xml version="1.0" encoding="UTF-8"?>
     73<?xml-stylesheet type="text/xsl" href="log.xsl"?>
     74
     75<build time="1 second">
     76        ...
     77        <task location="/usr/local/src/commons-collections-3.1/build.xml:394: " name="patternset" time="0 seconds"></task>
     78        <target name="clean" time="0 seconds">
     79                <task location="/usr/local/src/commons-collections-3.1/build.xml:95: " name="delete" time="0 seconds">
     80                        <message priority="info"><![CDATA[Deleting directory /usr/local/src/commons-collections-3.1/build]]></message>
     81                </task>
     82        </target>
     83</build>
     84}}}
     85
     86== Separating Output and Error Streams ==
     87
     88Without further knowledge about the build tool in use, the only option is to treat the output as a generic list of lines that go either to the standard output or the standard error stream. Messages on the error stream can be highlighted when presented, under the assumption that they represent warnings or errors.
     89
     90== Parsing the Output ==
     91
     92Richer handling of output can be provided if the output of the tool is known and consistent. This applies for example to Ant output in XML format. Based on an XML-formatted build log, Bitten could break up the output into targets, tasks and messages. Messages are always qualified with a “priority” in the XML log, which is one of “debug”, “verbose”, “info”, “warning” or “error”. This allows more fine grained handling of build messages than just differentiating between messages written to the output or error streams.
     93
     94 ''Note'': The {{{XmlLogger}}} option is employed by the CI system [http://cruisecontrol.sourceforge.net/ CruiseControl] to get the build output in a way that can later be transformed into a human-readable HTML rendering using XSLT.