CMake: Project structure with unit tests

For questions 1 & 2, I would recommend making a library from your non-test files excluding main.cpp (in this case just src/sqr.cpp and src/sqr.h), and then you can avoid listing (and more importantly re-compiling) all the sources twice.

For question 3, these commands add a test called "MyTest" which invokes your executable "test" without any arguments. However, since you've added these commands to test/CMakeLists.txt and not your top-level CMakeLists.txt, you can only invoke the test from within the "test" subdirectory of your build tree (try cd test && ctest -N). If you want the test to be runnable from your top-level build directory, you'd need to call add_test from the top-level CMakeLists.txt. This also means you have to use the more verbose form of add_test since your test exe isn't defined in the same CMakeLists.txt

In your case, since you're running cmake in the root folder, your build tree and your source tree are one and the same. This is known as an in-source build and isn't ideal, which leads to question 4.

The preferred method for generating the build tree is to do an out-of-source build, i.e. create a directory somewhere outside of your source tree and execute cmake from there. Even creating a "build" directory in the root of your project and executing cmake .. would provide a clean structure which won't interfere with your source tree.

One final point is to avoid calling executables "test" (case-sensitive). For reasons why, see this answer.

To achieve these changes, I'd do the following:

CMakeLists.txt:

cmake_minimum_required (VERSION 2.8)
project (TEST)
add_subdirectory (src) 
add_subdirectory (test)
enable_testing ()
add_test (NAME MyTest COMMAND Test)

src/CMakeLists.txt:

add_library (Sqr sqr.cpp sqr.h)
add_executable (demo main.cpp)
target_link_libraries (demo Sqr)

test/CMakeLists.txt:

find_package (Boost COMPONENTS system filesystem unit_test_framework REQUIRED)
include_directories (${TEST_SOURCE_DIR}/src
                     ${Boost_INCLUDE_DIRS}
                     )
add_definitions (-DBOOST_TEST_DYN_LINK)
add_executable (Test test.cpp)
target_link_libraries (Test
                       Sqr
                       ${Boost_FILESYSTEM_LIBRARY}
                       ${Boost_SYSTEM_LIBRARY}
                       ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}
                       )

I like the example of @Fraser but would use the add_test command in the test/CMakeLists.txt and use enable_testing before add_subdirectory(test).

This way you can run your tests from the top-level build directory while specifying your tests in the test/CMakeLists.txt.

The result would look like this (I reused the example of @Fraser):

CMakeLists.txt

cmake_minimum_required (VERSION 2.8)
project (TEST)
add_subdirectory (src)

enable_testing ()
add_subdirectory (test)

src/CMakeLists.txt

add_library (Sqr sqr.cpp sqr.h)
add_executable (demo main.cpp)
target_link_libraries (demo Sqr)

test/CMakeLists.txt

find_package (Boost COMPONENTS system filesystem unit_test_framework REQUIRED)
include_directories (${TEST_SOURCE_DIR}/src
                     ${Boost_INCLUDE_DIRS}
                     )
add_definitions (-DBOOST_TEST_DYN_LINK)
add_executable (Test test.cpp)
target_link_libraries (Test
                       Sqr
                       ${Boost_FILESYSTEM_LIBRARY}
                       ${Boost_SYSTEM_LIBRARY}
                       ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}
                       )
add_test (NAME MyTest COMMAND Test)