Is there a free C mocking framework

Embedded unit tests and mocking with CMock


1 1 Embedded unit tests and mocking with CMock Simon Raffeiner, Leitwerk AG Appenweier, Germany Abstract Software development for embedded systems has mostly hardly changed in recent years. Although the concepts behind Agile Development, Test-Driven Development and Extreme Programming have also been transferred to embedded systems (see [Gre02], [Dow04] and [KBE06]), the potential in many projects is wasted. Unit testing and mocking are largely ignored. The problem arises from the misjudgment of many developers that software for embedded systems without an operating system is difficult to test because the necessary points of attack (automation, frameworks) are missing and the scarce resources make the use of larger test frameworks impossible. Instead, you usually limit yourself to system tests. The main topic of this document is the introduction to the CMock 1 Mocking Framework, which can be used in conjunction with the Unity 2 Unit Test Framework to generate white box tests for C programs. I. INTRODUCTION Basic knowledge of white box testing and unit testing in the field of software development is required. Chapter 2 provides an overview of the basic problems encountered when testing embedded systems. Chapter 3 briefly describes the history behind the creation of the CMock Framework, Chapter 4 the general functionality and Chapter 5 the internal design. Chapter 6 looks at the connection with Unity as an example, while Chapter 7 analyzes the resource consumption on different platforms (embedded and PC). The document concludes with Chapter 8, a summary of the limitations and a brief outlook into the future. II. TESTING EMBEDDED SYSTEMS Embedded systems differ fundamentally from conventional personal computers in terms of available resources, computing power and interaction possibilities for debugging purposes. If unit testing is to be used on a large scale - preferably for every non-trivial function in every non-trivial module - the execution of the partial good tests must be integrated into the tool chain used, run as quickly as possible and produce results that are as accurate as possible of the production system. Ideally, a developer must be able to start a test run at any time so that any errors that occur can be noticed immediately, in larger projects it can even be useful to set up a dedicated test system that automatically translates and tests every change to the source code and, if necessary, sounds an alarm. The most frequently used possible solutions are listed below: A special development system with a debugging interface that allows automated uploading and execution of a test binary. The results are transmitted back to the host PC via a suitable interface (RS-232, JTAG, etc.) or e.g. written to a defined memory address, which in turn can be read out via the debugging interface. This method takes the most time, but fulfills the requirement for realistic results and also allows testing of peripherals (timers, counters, etc.). An emulator on the development PC. This method is usually faster than using a development system, but the emulator must offer the same automation options (uploading the binary, executing and reading out the results) - which is not always the case. Also, many emulators are unable to emulate peripheral devices at all, or at least the emulation is not accurate. Compilation of the tests for the architecture of the developer PC (cross-compiling) and subsequent local execution. This is possible when programming in high-level languages ​​such as C, but also requires special attention to subtleties such as the length of data types. This method is the fastest and easiest to carry out, but does not allow any access to specific peripherals. The results must be

2 2 Fig. 1. Components in the Model Conductor Hardware pattern in terms of runtime, resource consumption etc. do not match those on the embedded system, but this way it is very possible to find errors in hardware-independent logic. The biggest difference between personal computers with graphical user interfaces and embedded systems is the lack of the presentation layer (view): Most embedded systems do not require any interaction with the user, even if input and display options are available (e.g. LEDs, LCD, buttons ) these are usually treated on the same level as sensors and relays. [KBE06] introduces the Model Conductor Hardware (MCH) pattern for embedded systems. It is essentially an adaptation of the well-known Model View Controller [BM00] pattern, but replaces the presentation layer that exists in desktop applications with a generic hardware layer. MCH dictates a modularization of all source codes in loosely coupled modules, whereby each module may only have sub-modules that can be sorted into one of the three categories: addressing the hardware (driver), saving / processing data (model) or decision logic (conductor) . Modules and sub-modules communicate via clearly defined interfaces - these are method calls that are defined in header files. As soon as an interface has been defined, all communication can only take place using the methods defined therein. This increases both testability and reusability drastically, modules can be exchanged as long as they fully meet the specification. Since modules have to work together in reality in order to fulfill a certain functionality, it is not easily possible to test a single module independently. In order to still make this possible, mock modules are used - minimal implementation of the interface without a functioning inner workings. Mocks are generated automatically and trained to simulate the specification of the interface (call parameters, return values). Since the module under test now only works with mocks that show exactly known behavior, the requirement of loose coupling is fulfilled again. The positive effects of modularization on the general code quality and potential reusability in other projects have already been proven in detail, [BM00] also shows only a very small overhead (if optimizing compilers are used). III. HISTORY CMock is a development of Atomic Objects, LLC 3 and comes from a project for Savant Automation 4 [FBKW07], one of the largest American manufacturers of driverless transport vehicles. At the end of 2005, Savant commissioned Atomic Objects to reprogram the firmware for two ARM9 5 -based control boards. Atomic Object relied on an agile 6 development process, but required a tool chain that offered all the options that the team had already used in the development of desktop and web applications and at the same time on a Microchip PIC controller with 256 bytes of RAM and 32 Kilobyte ROM worked. The project was successfully carried out and resulted in a number of developments, including the aforementioned Model Conductor Hardware Pattern [KBE06] and several frameworks (Argent, Unity and CMock). While it took nine months to develop the firmware for the first control board, the software for the second board (with similar complexity) could be completed in just four months thanks to process improvements. As part of a presentation (see [WV08]) for the Embedded Systems Conference Boston in October 2008, both Unity and CMock were released under an open source license. Greg Williams, Michael Karlesky and Mark VanderVoord, employees at Atomic Objects, continued the development up to the current version, the dominant architecture in embedded systems 6

3 3 IV. FUNCTIONALITY The CMock Framework consists of a series of scripts for the Ruby 7 interpreter language and generates source code for mock objects from header files written in the C programming language. Again, all outputs are C code. CMock specializes in mocking and therefore does not offer any functionality for unit tests, which usually requires the use of an additional framework. Generators for Unity, Ignore and CException 8 are already included. A. Distribution This document is based on the version released at the end of 2008 The source code archive contains the following directories: config configuration files docs documentation examples examples iar files for IAR systems Embedded Workbench Compiler for ARM architectures lib The actual framework test unit tests for CMock itself vendor Necessary libraries for the unit tests If CMock is integrated into a project, only the Ruby scripts from the subdirectory lib and the Unity Framework from the directory vendor are necessary. V. INTERNAL DESIGN CMock follows the design of most mocking frameworks: Offers a possibility to specify how often, with which call parameters and which return values ​​a method is called, store all data internally and thus imitate the method. If all call parameters are as expected, return the known return value, if not, report an error. Since C, unlike other environments such as Java does not support code manipulation at runtime and CMock is designed for embedded systems, all source code must be generated before the compilation process. This is done in three steps (see also Figure 2): Parser: Reads in all header files and extracts the necessary information Generator: Generates the necessary mock code 7 a common object-oriented scripting language 8 a simple exception mechanism for ANSI using plug-ins C Fig. 2. Data flow within CMock File Writer: An intermediate layer to the file system, writes the final result in files. Figure 3 shows the internal relationships between the modules schematically. Two additional, not described in detail modules provide configuration settings (config) and prefabricated C code parts for the generator (generator utils). A. The Parser The parser is in a single file, cmock header parser.rb. It extracts information from a header file and stores the following data in an internal has structure: Additionally integrated header files (can contain necessary data types) External variables, method names, modifiers (e.g. static), data types of return values ​​and call parameters, names of the call parameters Figure 4 shows an example header file for a temperature filter (included with the CMock distribution), Figure 5 shows the information extracted from it. The parser is currently reading the input file line by line, skipping unnecessary information (e.g. comments or blank lines). The remaining lines will initially be

4 4 Fig. 3. Internal dependencies between modules 1 #ifndef _TEMPERATUREFILTER_H 2 #define _TEMPERATUREFILTER_H 3 #include "Types.h" 4 5 void TemperatureFilter_Init (void); 6 float TemperatureFilter_GetTemperatureInCelcius (void); 7 float TemperatureFilter_ProcessInput (float temperature); 8 #endif // _TEMPERATUREFILTER_H Fig. 4. TemperatureCalculator.h: Example header file for a temperature filter module processes that refer to additional header files to be integrated, in the last step those with method declarations. The extraction is realized by regular expressions and works stably in practice. However, since the parser does not provide any plug-ins, an adaptation to a programming language other than C would only be possible by completely replacing the parser. B. The generator The source code is generated by a core (cmock generator.rb) and several plug-ins (cmock generator * .rb), which are managed by the plug-in manager (cmock plugin manager.rb). The core takes on the following tasks: Communication with the parser (transfer of the generated hash structure) Communication with the file writer (output of the final source code) Generation of the mock-headers with all required includes and external variables Generation of the global * Init (), * Destroy () and * Verify () methods Generation of the mock instance The generation of the mock methods is delegated to plug-ins to enable better integration with various unit test frameworks. Listing 6 shows the generated mock header for the temperature filter, listing 7 the associated mock instance. The actual methods of the now mocked interface are not part of the header file created, but the interface - in this case TemperatureFilter.h - is integrated at the beginning. The actual source code of the mock is not shown here for reasons of space, but it is

5 5 1 {: includes => ["types.h"],: externs => [],: functions => [{: modifier => "",: rettype => "void",: args => [] ,: var_arg => nil,: name => "TemperatureFilter_Init",: args_string => "void"}, {: modifier => "",: rettype => "float",: args => [],: var_arg = > nil,: name => "TemperatureFilter_GetTemperatureInCelcius",: args_string => "void"}, {: modifier => "",: rettype => "float",: args => [{: type => "float", : name => "temperature"}],: var_arg => nil,: name => "temperaturefilter_processinput",: args_string => "float temperature"}]} Fig. 5. Information extracted from the header in the internal representation 1 # ifndef _MOCKTEMPERATUREFILTER_H 2 #define _MOCKTEMPERATUREFILTER_H 3 #include "TemperatureFilter.h" 4 5 void MockTemperatureFilter_Init (void); 6 void MockTemperatureFilter_Destroy (void); 7 void MockTemperatureFilter_Verify (void); 8 9 void TemperatureFilter_Init_Expect (void); 10 void TemperatureFilter_GetTemperatureInCelcius_ExpectAndReturn (float toreturn); 11 void TemperatureFilter_ProcessInput_ExpectAndReturn (float temperature, float toreturn); 12 #endif Fig. 6. MockTemperatureCalculator.7. Excerpt from MockTemperatureCalculator.c: Generated mock instance no longer of interest because it is automatically generated from ready-made code parts. The mock instance is the heart of every mock module. It saves the following information: allocfailure, number of errors that occurred during memory reservation * Return CallCount (actual number of calls) and * Return CallsExpected (expected number of calls) for each method * Return (next element), Return Head (Start) and Return HeadTail ( End), pointer in an array with all trained return values ​​* $ variable (next element), * $ variable Head (start) and * $ variable HeadTail (end), pointer in an array with the expected call values ​​of each parameter of each method Mock methods are used to manage this data structure. * Init () sets

6 6 return all counters and arrays to their initial state, * Destroy () releases all reserved memory again, * Verify () uses methods from the Unit Test Framework used to compare all call counters with the actual values ​​and triggers an error if they are not equal. Expect (parameter) and * ExpectAndReturn (parameter, return value) work as usual from other Mock Frameworks: They increase the number of expected calls for the associated method by one, set the expected parameters and, in the case of * ExpectAndReturn (), the delivering return value in the mock instance. The actual methods specified in the interface, which are later called by the tested module, always have the same structure: Compare the parameters received with the target values ​​stored in the mock instance, manage the pointers in the arrays (e.g. jump to the next Data record), return of the known value if correct and triggering an error in the event of deviations. It should be noted at this point that the responsibility for correctly linking test drivers and mocks lies with the build system. Since mocks and real modules have identical method signatures, error messages relating to symbols that exist multiple times can otherwise occur. C. The File Writer The File Writer writes the generated source codes to temporary files and, after successful completion, moves them to the configured directories. Since there are only simple methods at this level, reference is made at this point to the cmock file writer.rb. D. Integration with Unity CMock works very closely with the Unity Framework. Although the code generation should actually be independent of the framework, Unity calls are automatically inserted in some parts. Listing 8 shows an example of the use of the generated mock in a test driver: The UsartModel module is responsible for converting the current temperature into a string for output on an LC display. The required temperature value is queried by method calls from the TemperatureFilter. The test driver initializes the mock, trains it for the required method calls 1 ruby ​​lib / cmock.rb -oconfig.yml src / uart.h src / sensor.h Fig. 9. Example: Call via the interpreter and return values ​​and then checks the final result delivered by the UsartModel. In this case, the temperature at the first call should be 25 degrees Celsius, the second call then tests an error condition (it is assumed that an infinitely negative temperature signals an error). By using the mock, the test is independent of real sensors and also independent of an actually existing UsartModel module.It is now even possible to develop and test this module if work on the other modules has not yet started - as long as the interface is stable and adhered to, a trained mock does not differ from a real module. VI. CALL CMock is usually integrated into a toolchain in order to automatically generate mock objects. Two modes are offered for this: The ruby.rb script can be called directly via the Ruby interpreter, but if Ruby scripts already exist, CMock also offers a class with several methods. Both modes are identical in functionality, a look at the ruby.rb script shows that an instance of the object is simply created there and the necessary methods are then executed. A. Call via the interpreter When called directly via the interpreter, all parameters are interpreted as paths to C header files or an optional configuration file. CMock then iterates over all paths and generates a mock for each header file. Listing 9 shows an example call. Configuration files must be in YAML 9 format. A complete list of all options can be found in the project documentation. If no configuration file is specified, the standard settings apply; the mocks are then stored separately in the mocks / subdirectory. 9 YAML Ain t Markup Language, a human-readable standard for serializing data

7 7 1 void testgetformattedtemperature (void) 2 {3 TemperatureFilter_Init (); 4 5 TemperatureFilter_GetTemperatureInCelcius_ExpectAndReturn (25.0f); 6 TEST_ASSERT_EQUAL_STRING ("25.0 C \ n", UsartModel_GetFormattedTemperature ()); 7 8 TemperatureFilter_GetTemperatureInCelcius_ExpectAndReturn (-INFINITY); 9 TEST_ASSERT_EQUAL_STRING ("Temperature sensor failure! \ N", UsartModel_GetFormattedTemperature ()); TemperatureFilter_Verify (); 12 TemperatureFilter_Destroy (); 13} Fig. 8. Use of the generated mock in a test driver for another module 1 cmock = (options) 2 cmock.setup_mocks (list) 3 cmock.generate_mock (source) Fig. 10. Ruby functions offered by CMock 1 cmock: 2 mock_path: mocks / 3 includes: 4 - Types.h 5 plugins: 6 - ignore B. Call from other Ruby scripts or via Rake CMock can be used as a native Ruby object from other Ruby scripts, Listing 10 shows the methods offered for this. The constructor accepts the same options that can be specified in the YAML configuration file. If no options are specified, the default settings apply. setup_mocks () generates mock objects for a list of input files, generate_mock () for a single file. VII. INTEGRATION IN EXISTING PROJECTS The CMock distribution and the sample files use the rake build system by default, but the integration into an existing tool chain is simple: only a build step has to be inserted before the compilation Mocks generated. The project wiki contains instructions 10 on how this can be configured in the widespread Eclipse development environment 11. Most projects have an already existing folder structure, so it makes sense to store the first few mocks and unit tests together with the associated modules in the same directory. This proves to be unnecessarily problematic in the 10 Eclipse IDE Integration 11 IDE, Integrated Development Environment Fig. 11. Example for a configuration section in YAML practice, however: Development environments with extended functionality such as e.g. Indexing, source code navigation and automatic completion get out of step in view of the now suddenly multiple existing methods (each once in the actual module and also in its mock), and the internal mock methods (Init (), Destroy (), * Expect () etc.). It is usually possible to exclude individual files from these functions, but this also increases the maintenance effort. If, on the other hand, module files, mocks and unit tests are each kept in their own folder, they only need to be hidden once. All CMock parameters, including the output path for the generated files, can be configured via a YAML file. Listing 11 shows an example of a configuration section that instructs CMock to forcibly include the Types.h header file, load the ignore plug-in and write the results to the mocks subdirectory relative to the current path. A larger project may require the use of several configuration files. VIII. RESOURCE CONSUMPTION Most embedded systems have very little memory for code (in-

8 8 Fig. 12. Code Instance Atmel AVR (Atmega8) Atmel AVR (Atmega32) Intel Intel i AMD x Resource consumption on different platforms internal / external flash memory or EEPROM) and data (internal / external RAM). If unit tests are to be carried out directly on the device or in an emulator, the memory allocation must be as small as possible so that as many tests as possible can be carried out from a single binary. This reduces the number of test binaries required and speeds up the process. The first column in Table 12 lists the size of the machine code generated from Listing 4 for different architectures (PC and Embedded), the second column the size of the empty mock instance. These values ​​do not change at runtime. In addition to these two values, the memory space required to save all call parameters and return values ​​is added at runtime: Each call of * Expect (parameters) or * ExpectandReturn (parameters, return) expands the internal arrays. The end result therefore depends on the data types to be stored and the number of trained values. In addition, the implementation of malloc () used does not have to reserve memory areas byte-by-byte, but can also work with larger blocks. The following C compilers were used: avr-gcc for Atmel AVR, SDCC # 5117 for Intel 8051, GCC for Intel i386, GCC for AMD x All compilers were instructed to keep the machine code as compact as possible via the corresponding call parameters. As can be seen from the table, the example is translated into a similar amount of machine code on all platforms, but different standards apply depending on the device: The Atmega8, for example, only has 8 kilobytes of flash memory and one kilo of RAM. This size is sufficient for a whole range of projects, for example an MP3 player was implemented with this controller. On the other hand, the memory is only sufficient for two or three mock objects with associated unit tests and necessary sub-systems (e.g. communication with the host). Some representatives of the 8051 only have 128 bytes of RAM, on these devices the use of unit tests and CMock is sometimes impossible because there is not enough memory available for training. Most larger projects, however, rely on environments with less restrictive limitations, such as the Atmega32 (32 kilobyte Flash, 2 kilobyte RAM) or more advanced models (ARM, MIPS, ColdFire etc.). Intel i386 and AMD x86 64 are listed as examples of cross-compiling on developer PCs where memory is typically not a problem. IX. LIMITATIONS AND CONCLUSIONS The content of this document shows that CMock meets the basic requirements for a mock framework for the C programming language. However, there may be several arguments against its use in embedded systems. CMock relies heavily on malloc (), realloc () and free () from the standard library. Not all runtime environments offer this functionality, in some projects its use is prohibited for security reasons - both in test drivers and in productive code. There is currently no solution for this. The strong dependency on Unity may create conflicts if another unit test framework is already being used in an existing project or Unity is not compatible with the project rules. Although the code generation should be handled entirely via plug-ins, this is not the case in all areas, so it is not possible to do without Unity completely at the moment. The CMock Framework is a relatively young project and has been actively developed since it was released. Since the use of unit tests and mocks in the embedded environment is currently not an issue in many companies, it will still have to be shown in the future whether CMock is the framework of choice or one of the competitors will conquer the field. REFERENCES [BM00] Andy Bower and Blair McGlashan, editors. Twisting The Triad - The evolution of the Dolphin Smalltalk MVP application framework. ESUG 2000 International Smalltalk Conference, [Dow04] Micah Dowty. Test driven development of embedded systems. University of Colorado at Boulder, March [FBKW07] Matt Flettcher, William Bereza, Mike Karlesky, and Greg Williams, editors. Evolving into Embedded Development. Agile 2007 Conference, August 2007.

9 9 [Gre02] James W. Grenning, editor. XP and Embedded Systems development. Object Mentor Inc., 5101 Washington Street, Suite 1108, Gurnee, IL USA, 1st edition, March [KBE06] Michael J. Karlesky, William I. Bereza, and Carl B. Erickson, editors. Effective Test Driven Development for Embedded Software. IEEE Electro / Information Technology Conference, [WV08] Greg Williams and Mark VanderVoord, editors. Embedded Feature Driven Design Using TDD and Mocks. Embedded Systems Conference Boston 2008, October 2008.