user1291510 user1291510 - 1 month ago 18
C++ Question

CMake testing causing error when tests fail

I've been looking around for a solution to this but cannot seem to find out why this happens. I use CMake for a small project and I wanted to add unit tests cmakes

add_test
. It works well when tests are passing but when they fail, I get an odd error message that I cannot interpret.

I've made a minimal example to illustrate what happens.

CMakeLists.txt

cmake_minimum_required (VERSION 2.6)

project (TestExample)

enable_testing()

add_executable(succeedExample succeed.cpp)
add_executable(failExample fail.cpp)

add_test(NAME succeedTest COMMAND succeedExample)
add_test(NAME failTest COMMAND failExample)


succeed.cpp

int main()
{
return 0;
}


fail.cpp

int main()
{
return 1;
}


I use Visual Studio 2015, and when building the RUN_TESTS target, I get the following error:

Severity Code Description Project File Line Suppression State
Error MSB3073 The command "setlocal
"C:\Program Files (x86)\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd
:VCEnd" exited with code 8. RUN_TESTS C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets 133


Yet if I change the fail.cpp to
return 0;
I get

1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
1> Test project C:/Users/.../projects/cmaketestexample/build
1> Start 1: succeedTest
1> 1/2 Test #1: succeedTest ...................... Passed 0.01 sec
1> Start 2: failTest
1> 2/2 Test #2: failTest ......................... Passed 0.02 sec
1>
1> 100% tests passed, 0 tests failed out of 2
1>
1> Total Test time (real) = 0.03 sec
========== Build: 1 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========


Can anyone point me towards a solution to fix this? Am I missing something obvious?

Thanks :)

EDIT

Visual studio version is 14.0 (Visual Studio 2015 community)

Cmake 3.5.2 - yet I set the version to 2.6 in the CMakeLists.txt.

EDIT2

running
ctest -C Debug
on command line:

Test project C:/Users/.../projects/cmaketestexample/build
Start 1: succeedTest
1/2 Test #1: succeedTest ...................... Passed 0.02 sec
Start 2: failTest
2/2 Test #2: failTest .........................***Failed 0.02 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) = 0.07 sec

The following tests FAILED:
2 - failTest (Failed)
Errors while running CTest


Changing fail.cpp to

#include <iostream>
int main()
{
std::cout << "Does it work now?" << std::endl;
return 1;
}


makes no difference.

EDIT3

A similar error occur on Ubuntu 14.04.4 LTS
using


  • cmake version 3.2.2

  • GNU Make 3.81

  • g++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4


Answer

I've given you code a try and what you see is the little script that CMake embeds the ctest call into. And ctest does fail - return a non-zero value - if any of tests it is executing does fail. And this behavior of ctest is by intention to stop any scripts/builds running the tests if encountering errors.

The possible error return codes of ctest are a combination of the following bits:

// provide some more detailed info on the return code for ctest
enum {
  UPDATE_ERRORS    = 0x01,
  CONFIGURE_ERRORS = 0x02,
  BUILD_ERRORS     = 0x04,
  TEST_ERRORS      = 0x08,
  MEMORY_ERRORS    = 0x10,
  COVERAGE_ERRORS  = 0x20,
  SUBMIT_ERRORS    = 0x40
};

So I see the following options:

  1. Ignore the strange error, the output you're looking for is just above the error message:

    1>------ Build started: Project: RUN_TESTS, Configuration: Debug Win32 ------
    1>  Test project C:/temp/StackOverflow/TestError/VS2015
    1>      Start 1: succeedTest
    1>  1/2 Test #1: succeedTest ......................   Passed    0.03 sec
    1>      Start 2: failTest
    1>  2/2 Test #2: failTest .........................***Failed    0.03 sec
    1>
    1>  50% tests passed, 1 tests failed out of 2
    1>
    1>  Total Test time (real) =   0.10 sec
    1>
    1>  The following tests FAILED:
    1>        2 - failTest (Failed)
    1>  Errors while running CTest
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: The command "setlocal
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: "C:\Program Files\CMake\bin\ctest.exe" --force-new-ctest-process -C Debug
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmEnd
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmErrorLevel
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: exit /b %1
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :cmDone
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd
    1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(133,5): error MSB3073: :VCEnd" exited with code 8.
    ========== Build: 0 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========
    
  2. Add your own custom target calling ctest on VS without a script, giving just one error:

    if (CMAKE_CONFIGURATION_TYPES)
        add_custom_target(
            RUN_CTEST
                COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIGURATION> --force-new-ctest-process 
                WORKING_DIRECTORY "${CMAKE_BINARY_DIR}"
        )
    endif()
    
  3. Add your own custom target command that does ignore the return value by moving the ctest call into a separate script (see references).

References

Comments