Code-Coverage Tool

The code-coverage tool provides software developers with a view of how much application code is exercised when a specific workload is applied to the application. To determine which code is used, the code-coverage tool uses Profile-Guided Optimization technology.

The major features of the code-coverage tool are:

• Visually presenting code coverage information for an application with a customizable code-coverage coloring scheme

• Displaying dynamic execution counts of each basic block of the application

• Providing differential coverage, or comparison, profile data for two runs of an application

The tool analyzes static profile information generated by the compiler, as well as dynamic profile information generated by running an instrumented form of the application binaries on the workload. The tool can generate the in HTML-formatted report and export data in both text-, and XML-formatted files. The reports can be further customized to show color-coded, annotated, source-code listings that distinguish between used and unused code.

The code-coverage tool can be used on IA-32, Intel® EM64T, and Intel® Itanium® architecture in a number of ways to improve development efficiency, reduce defects, and increase application performance:

• During the project testing phase, the tool can measure the overall quality of testing by showing how much code is actually tested.

• When applied to the profile of a performance workload, the code-coverage tool can reveal how well the workload exercises the critical code in an application. High coverage of performance-critical modules is essential to taking full advantage of the Profile-Guided Optimizations that Intel Compilers offer.

• The tool provides an option, useful for both coverage and performance tuning, enabling developers to display the dynamic execution count for each basic block of the application.

• The code-coverage tool can compare the profile of two different application runs. This feature can help locate portions of the code in an application that are unrevealed during testing but are exercised when the application is used outside the test space, for example, when used by a customer.

Code-coverage tool Requirements

To run the code-coverage tool on an application, you must have following items:

• The application sources.

• The .spi file generated by the Intel® compiler when compiling the application for the instrumented binaries using the -prof-genx (Linux*) or /Qprof-genx (Windows*) options.

• A pgopti.dpi file that contains the results of merging the dynamic profile information (.dyn) files, which is most easily generated by the profmerge tool. This file is also generated implicitly by the Intel® compilers when compiling an application with -prof-use (Linux) or /Qprof-use (Windows) options with available .dyn and .dpi files.

See Understanding Profile-guided Optimization and Example of Profile-guided Optimization for general information on creating the files needed to run this tool.

Running the Tool

In general, you must perform the following steps to use the code-coverage tool:

1. Compile the application using -prof-genx (Linux) or /Qprof-genx (Windows).
   
   This step generates an instrumented binary and the corresponding static profile information (pgopti.spi) file.

2. Run the instrumented application.
   
   This step creates the dynamic profile information (.dyn) file Each time you run the instrumented application, the compiler generates a unique .dyn file either in the current directory or the directory specified in by prof_dir.

3. Use the profmerge tool to merge all the .dyn files into one .dpi (pgopti.dpi) file.
   
   This step consolidates all runs and represents the total profile information for the application, generate an optimized binary, and create the dpi file needed by the code-coverage tool.
   
   You can use the profmerge tool to merge the .dyn files into a .dpi file without recompiling the application. The profmerge tool can also merge multiple .dpi files into one .dpi file using the profmerge -a option. Select the name of the output .dpi file using the profmerge -prof_dpi option.

Caution

The profmerge tool merges all .dyn files that exist in the given directory. Make sure unrelated .dyn files, which may remain from unrelated runs, are not present. Otherwise, the profile information will be skewed with invalid profile data, which can result in misleading coverage information and adverse performance of the optimized code.

4. Run the code-coverage tool passing arguments, as shown in the following sections.
   
   This step creates a report or exported data as specified. By default, the code-coverage tool places a single HTML file (CODE_COVERAGE.HTML) in the current directory. Open the file in a web browser to view the reports.

The tool uses the following syntax:

where -codecov_option is one or more optional parameters specifying the tool option passed to the tool. The available tool options are listed under Code-coverage tool Options (below). If you do not use any additional tool options, the tool will provide the top-level code coverage for the entire application.

Note

Windows* only: Unlike the compiler options, which are preceded by forward slash ("/"), the tool options are preceded by a hyphen ("-").

The code-coverage tool allows you to name the project and specify paths to specific, necessary files. The following example demonstrates how to name a project and specify .dpi and .spi files to use:

The tool can add a contact name and generate an email link for that contact at the bottom of each HTML page. This provide a way to send an electronic message to the named contact. The following example demonstrates how to add specify a contact and the email links:

This following example demonstrates how to use the tool to specify the project name, specify the dynamic profile information file, and specify the output format and file name.

Code-coverage tool Options

The tool uses the options listed in the table:

Visually Presenting Code Coverage for an Application

Based on the profile information collected from running the instrumented binaries when testing an application, the Intel® compiler will create HTML-formatted reports using the code-coverage tool. These reports indicate portions of the source code that were or were not exercised by the tests. When applied to the profile of the performance workloads, the code-coverage information shows how well the training workload covers the application's critical code. High coverage of performance-critical modules is essential to taking full advantage of the profile-guided optimizations.

The code-coverage tool can create two levels of coverage:

• Top level (for a group of selected modules)

• Individual module source views

Top Level Coverage

The top-level coverage reports the overall code coverage of the modules that were selected. The following options are provided:

• Select the modules of interest

• For the selected modules, the tool generates a list with their coverage information. The information includes the total number of functions and blocks in a module and the portions that were covered.

• By clicking on the title of columns in the reported tables, the lists may be sorted in ascending or descending order based on:

• • basic block coverage

  • function coverage

  • function name

By default, the code-coverage tool generates a single HTML file (CODE_COVERAGE.HTML) and a subdirectory (CodeCoverage) in the current directory. The HTML file defines a frameset to display all of the other generated reports. Open the HTML file in a web-browser. The tool places all other generated report files in a CodeCoverage subdirectory.

If you choose to generate the html-formatted version of the report, you can view coverage source of that particular module directly from a browser. The following figure shows the top-level coverage report.

The coverage tool creates a frame set that allows quick browsing through the code to identify uncovered code. The top frame displays the list of uncovered functions while the bottom frame displays the list of covered functions. For uncovered functions, the total number of basic blocks of each function is also displayed. For covered functions, both the total number of blocks and the number of covered blocks as well as their ratio (that is, the coverage rate) are displayed.

For example, 66.67(4/6) indicates that four out of the six blocks of the corresponding function were covered. The block coverage rate of that function is thus 66.67%. These lists can be sorted based on the coverage rate, number of blocks, or function names.  Function names are linked to the position in source view where the function body starts. So, just by one click, you can see the least-covered function in the list and by another click the browser displays the body of the function. You can scroll down in the source view and browse through the function body.

Individual Module Source View

Within the individual module source views, the tool provides the list of uncovered functions as well as the list of covered functions. The lists are reported in two distinct frames that provide easy navigation of the source code. The lists can be sorted based on:

• Number of blocks within uncovered functions

• Block coverage in the case of covered functions

• Function names

Setting the Coloring Scheme for the Code Coverage

The tool provides a visible coloring distinction of the following coverage categories: covered code, uncovered basic blocks, uncovered functions, partially covered code, and unknown code. The default colors that the tool uses for presenting the coverage information are shown in the tables that follows:

The default colors can be customized to be any valid HTML color name or hexadecimal value using the options mentioned for each coverage category in the table above.

For code-coverage colored presentation, the coverage tool uses the following heuristic: source characters are scanned until reaching a position in the source that is indicated by the profile information as the beginning of a basic block. If the profile information for that basic block indicates that a coverage category changes, then the tool changes the color corresponding to the coverage condition of that portion of the code, and the coverage tool inserts the appropriate color change in the HTML-formatted report files.

Note

You need to interpret the colors in the context of the code. For instance, comment lines that follow a basic block that was never executed would be colored in the same color as the uncovered blocks. Another example is the closing brackets in C/C++ applications.

Coverage Analysis of Module Subsets

One of the capabilities of the code-coverage tool is efficient coverage analysis of a subset of modules for an application.

You can generate the profile information for the whole application, or a subset of it, and then break the covered modules into different components and use the coverage tool to obtain the coverage information of each individual component. If only a subset of the application modules is compiler with the -prof-genx (Linux) or /Qprof-genx (Windows) option, then the coverage information is generated only for those modules that are involved with this compiler option, thus avoiding the overhead incurred for profile generation of other modules.

To specify the modules of interest, use the -comp option. This option takes the name of a file as its argument. The file must be a text file that includes the name of modules or directories you would like to analyze.

Note

Each line of component file should include one, and only one, module name.

For example:

Any module of the application whose full path name has an occurrence of any of the names in the component file will be selected for coverage analysis. For example, if a line of file component1 in the above example contains mod.cpp, then all modules in the application that have that name will be analyzed. The user can specify a particular module by passing more specific path information. For example, if the line contains /cmp1/mod1.cpp, then only those modules with the name mod.cpp will be selected that are in a directory named cmp1. If no component file is specified, then all files that have been compiled with -prof-genx (Linux) or /Qprof-genx (Windows) are selected for coverage analysis.  

Dynamic Counters

The coverage tool can be configured to generate the information about the dynamic execution counts. This ability can display the dynamic execution count of each basic block of the application and is useful for both coverage and performance tuning.

The custom configuration requires using the -counts option. The counts information is displayed under the code after a "^" sign precisely under the source position where the corresponding basic block begins.

If more than one basic block is generated for the code at a source position (for example, for macros), then the total number of such blocks and the number of the blocks that were executed are also displayed in front of the execution count. For example, line 11 in the code is an IF statement:

The coverage lines under code lines 11 and 12  contain the following information:

• The IF statement in line 11 was executed 10 times.

• Two basic blocks were generated for the IF statement in line 11.

• Only one of the two blocks was executed, hence the partial coverage color.

• Only seven out of the ten times variable n had a value of 0 or 1.

In certain situations, it may be desirable to consider all the blocks generated for a single source position as one entity.  In such cases, it is necessary to assume that all blocks generated for one source position are covered when at least one of the blocks is covered. This assumption can be configured with the -nopartial option. When this option is specified, decision coverage is disabled, and the related statistics are adjusted accordingly. The code lines 11 and 12  indicate that the print statement in line 12 was covered. However, only one of the conditions in line 11 was ever true. With the -nopartial option, the tool treats the partially covered code (like the code on line 11) as covered.

Differential Coverage

Using the code-coverage tool, you can compare the profiles from two runs of an application: a reference run and a new run identifying the code that is covered by the new run but not covered by the reference run. Use this feature to find the portion of the applications code that is not covered by the applications tests but is executed when the application is run by a customer. It can also be used to find the incremental coverage impact of newly added tests to an applications test space.

Generating Reference Data

Create the dynamic profile information for the reference data, which can be used in differential coverage reporting later, by using the -ref option. The following command demonstrate a typical command for generating the reference data:

The coverage statistics of a differential-coverage run shows the percentage of the code exercised on a new run but missed in the reference run. In such cases, the tool shows only the modules that included the code that was not covered. Keep this in mind when viewing the coloring scheme in the source views.

The code that has the same coverage property (covered or not covered) on both runs is considered as covered code. Otherwise, if the new run indicates that the code was executed while in the reference run the code was not executed, then the code is treated as uncovered. On the other hand, if the code is covered in the reference run but not covered in the new run, the differential-coverage source view shows the code as covered.

Running Differential Coverage

To run the code-coverage tool for differential coverage, you must have the application sources, the .spi file, and the .dpi file, as described in the  Code-coverage tool Requirements section (above).

Once the required files are available, enter a command similar to the following begin the process of differential coverage analysis:

Specify the path to the .dpi and .spi using the -spi and -dpi options.

Excluding Code from Coverage Analysis

The code-coverage tool allows you to exclude portions of your code from coverage analysis. This ability can be useful during development; for example, certain portions of code might include functions used for debugging only. The test case should not include tests for functionality that will unavailable in the final application.

Another example of code that can be excluded is code that might be designed to deal with internal errors unlikely to occur in the application. In such cases, not having a test case lack of a test case is preferred. You might want to ignore infeasible (dead) code in the coverage analysis. The code-coverage tool provides several options for marking portions of the code infeasible (dead) and ignoring the code at the file level, function level, line level, and arbitrary code boundaries indicated by user-specific comments. The following sections explain how to exclude code at different levels.

Including and Excluding Coverage at the File Level

The code-coverage tool provides the ability to selectively include or exclude files for analysis. Create a component file and add the appropriate string values that indicate the file and directory name for code you want included or excluded.  Pass the file name as a parameter of the -comp option. The following example shows the general command:

where file is the name of a text file containing strings that ask as file and directory name masks for including and excluding file-level analysis. For example, assume that the following:

• You want to include all files with the string "source" in the file name or directory name.

• You create a component text file named myComp.txt with the selective inclusion string.

Once you have a component file, enter a command similar to the following:

In this example, individual files name including the string "source" (like source1.c and source2.c) and files in directories where the name contains the string "source" (like source/file1.c and source2\file2.c ) are include in the analysis.

Excluding files is done in the same way; however, the string must have a tilde (~) prefix. The inclusion and exclusion can be specified in the same component file.

For example, assume you want to analyze all individual files or files contained in a directory where the name included the string "source", and you wanted to exclude all individual file and files contained in directories where the name included  the string "skip". You would add content similar to the following to the component file (myComp.txt) and pass it to the -comp option:

Entering the codecov -comp myComp.txt command with both instructions in the component file will instruct the tool to include individual files where the name contains "source" (like source1.c and source2.c) and directories where the name contains "source" (like source/file1.c and source2\file2.c ), and exclude any individual files where the name contains "skip" (like skipthis1.c and skipthis2.c) or directories where the name contains "skip" (like skipthese1\debug1.c and skipthese2\debug2.c).

Excluding Coverage at the Line and Function Level

You can mark individual lines for exclusion my passing string values to the -onelinedsbl option. For example, assume that you have some code similar to the following:

 If you wanted to exclude all functions marked with the comments INFEASIBLE or INF IA32, you would enter a command similar to the following.

You can specify multiple exclusion strings simultaneously, and you can specify any string values for the markers; however, you must remember the following guidelines when using this option:

• Inline comments must occur at the end of the statement.

• If the string is not part of an inline comment, the string value must be surrounded by quotation marks (").

An entire function can be excluded from coverage analysis using the same methods. For example, the following function will be ignored from the coverage analysis when you issue example command shown above.

Additionally, you can use the code-coverage tool to color the infeasible code with any valid HTML color code by combining the -onelinedsbl and -xcolor options. The following example commands demonstrate the combination:

Excluding Code by Defining Arbitrary Boundaries

The code-coverage tool provides the ability to arbitrarily exclude code from coverage analysis. This feature is most useful where the excluded code either occur inside of a function or spans several functions.

Use the -beginblkdsbl and -endblkdsbl options to mark the beginning and end, respectively, of any arbitrarily defined boundary to exclude code from analysis.  Remember the following guidelines when using these options:

• Inline comments must occur at the end of the statement.

• If the string is not part of an inline comment, the string value must be surrounded by quotation marks (").

For example assume that you have the following code:

The following example commands demonstrate how to use the -beginblkdsbl option to mark the beginning and the -endblkdsbl option to mark the end of code to exclude from the sample shown above.

Notice that you can combine these options in combination with the -xcolor option.

Exporting Coverage Data

The code-coverage tool provides specific options to extract coverage data from the dynamic profile information (.dpi files) that result from running instrumented application binaries under various workloads. The tool can export the coverage data in various formats for post-processing and direct loading into databases: the default HTML, text, and XML. You can choose to export data at the function and basic block levels.

There are two basic methods for exporting the data: quick export and combined export. Each method has associated options supported by the tool

• Quick export: The first method is to export the data coverage to text- or XML-formatted files without generating the default HTML report. The application sources need not be present for this method. The code-coverage tool creates reports and provides statistics only about the portions of the application executed. The resulting analysis and reporting occurs quickly, which makes it practical to apply the coverage tool to the dynamic profile information (the .dpi file) for every test case in a given test space instead of applying the tool to the profile of individual test suites or the merge of all test suites. The -xmlfcvrg, -txtfcvrg, -xmlbcvrg and -txtbcvrg options support the first method.

• Combined export: The second method is to generate the default HTML and simultaneously export the data to text- and XML-formatted files. This process is slower than first method since the application sources are parsed and reports generated. The -xmlbcvrgfull and -txtbcvrgfull options support the second method.

These export methods provide the means to quickly extend the code coverage reporting capabilities by supplying consistently formatted output from the code-coverage tool. You can extend these by creating additional reporting tools on top of these report files.

Quick Export

The profile of covered functions of an application can be exported quickly using the -xmlfcvrg, -txtfcvrg, -xmlbcvrg, and -txtbcvrg options. When using any of these options, specify the output file that will contain the coverage report. For example, enter a command similar to the following to generate a report of covered functions in XML formatted output:

The resulting report will show how many times each function was executed and the total number of blocks of each function together with the number of covered blocks and the block coverage of each function. The following example shows some of the content of a typical XML report.

In the above example, we note that function f0, which is defined in file sample.c, has been executed twice. It has a total number of six basic blocks, five of which are executed, resulting in an 83.33% basic block coverage.

You can also export the data in text format using the -txtfcvrg option. The generated text report, using this option, for the above example would be similar to the following example:

In the text formatted version of the report, the each line of the report should be read in the following manner:

Additionally, the tool supports exporting the block level coverage data using the -xmlbcvrg option. For example, enter a command similar to the following to generate a report of covered blocks in XML formatted output:

The example command shown above would generate XML-formatted results similar to the following:

In the sample report, notice that one basic block is generated for the code in function f0 at the line 11, column 2 of the file sample.c. This particular block has been executed only once. Also notice that there are two basic blocks generated for the code that starts at line 12, column 3 of file. One of these blocks, which has id = 1, has been executed two times, while the other block has been executed only once. A similar report in text format can be generated through the -txtbcvrg option.

Combined Exports

The coverage tool has also the capability of exporting coverage data in the default HTML format while simultaneously generating the text- and XML-formatted reports.

Use the -xmlbcvrgfull and -txtbcvrgfull options to generate reports in all supported formatted in a single run. These options export the basic-block level coverage data while simultaneously generating the HTML reports. These options generate more complete reports since they include analysis on functions that were not executed at all. However, exporting the coverage data using these options requires access to application source files and take much longer to run.