How should I setup script-based source generation with CMake?

Question

I'm managing the build of a certain C++ repository using CMake. In this repository, and among other things, I have a bunch of .hpp files in a directory, and for each of these, I need to compile a generated bit of source code, which depends on its contents.

Naively, I would do this by generating a correspond .cpp file, and include that file in the source files of the CMake target for the library or executable I'm building. Now, since I don't actually need the source files themselves, I could theoretically just arrange for the compiler to get its source from the command-line instead.

My question: How would I set up this source generation and compilation, using CMake as idiomatically as possible?

Notes:

Assume I have, say, a bash script which can read the .hpp and generate the .cpp on the standard output.
CMake version 3.24 or whichever you like.
Should work on Unix-like operating systems, and hopefully on Windows-like OSes other than the fact that the bash script will fail.
Please comment if additional information is necessary to answer my question.

This is just a classic use of add_custom_command. Did you try that already / is something not working? — Alex Reinking
– Alex Reinking, Commented Aug 7, 2022 at 23:48
@AlexReinking: I was assuming whether there was something higher-level than add_custom_command; and whether I should actually generate files or just generate sources for the compiler to eat. — einpoklum
– einpoklum, Commented Aug 8, 2022 at 19:14

Alex Reinking · Accepted Answer · 2022-08-08 12:39:20Z

1

Let's assume for the sake of portability that you have a Python script, rather than a bash script that manages your code generation. Let's say that it takes two arguments: the source .hpp file and the destination .cpp file. We'll assume it is in your source tree under ./tools/codegen.py.

Now let's assume that your .hpp files are in ./src/genmod for "generated module" because the sources for these headers are generated by codegen.py.

Finally, we'll assume there's a final executable target, app, with a single source file, ./src/main.cpp.

Here's a minimal build that will work for this, with some step-by-step discussion.

We start with some boring boilerplate.

cmake_minimum_required(VERSION 3.24)
project(example)

This probably works on earlier versions, I just haven't tested it, so YMMV. Now we'll create the executable target and link it to our generated sources preemptively. Note that the dependent target does not need to exist before calling target_link_libraries.

add_executable(app src/main.cpp)
target_link_libraries(app PRIVATE genmod)

Now we'll find a Python interpreter and write down an absolute path to the codegen tool.

find_package(Python3 REQUIRED)
set(codegen_py "${CMAKE_CURRENT_SOURCE_DIR}/tools/codegen.py")

Next we'll construct the list of input headers. I'm imagining there are three: A.hpp, B.hpp, and C.hpp.

set(input_headers A.hpp B.hpp C.hpp)
list(TRANSFORM input_headers PREPEND src/genmod/)

I used list(TRANSFORM) here to save some typing. Now we'll just create an object library called genmod, which will "hold" the objects for the generated C++ files.

add_library(genmod OBJECT)

And now comes the real meat. For each of the headers, ...

foreach (header IN LISTS input_headers)

we'll construct absolute paths to the header and generated source files ...

    string(REGEX REPLACE "\\.hpp$" ".cpp" gensrc "${header}")
    set(header "${CMAKE_CURRENT_SOURCE_DIR}/${header}")
    set(gensrc "${CMAKE_CURRENT_BINARY_DIR}/${gensrc}")

and then write a custom command that knows how to call codegen.py. We specify the outputs, command arguments, and dependencies. Don't forget to include the generator script as a dependency, and never forget to pass VERBATIM to ensure consistent, cross-platform, argument quoting.

    add_custom_command(
        OUTPUT "${gensrc}"
        COMMAND Python3::Interpreter "${codegen_py}" "${header}" "${gensrc}"
        DEPENDS "${header}" "${codegen_py}"
        VERBATIM
    )

Finally, we attach this source to genmod.

    target_sources(genmod PRIVATE "${gensrc}")
endforeach ()

We can test this build using Ninja's dry-run feature to make sure the commands look correct.

$ cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /home/reinking/.venv/default/bin/python3.9 (found version "3.9.2") found components: Interpreter 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/reinking/test/build

$ cmake --build build -- -nv
[1/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/A.hpp /home/reinking/test/build/src/genmod/A.cpp
[2/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/B.hpp /home/reinking/test/build/src/genmod/B.cpp
[3/8] cd /home/reinking/test/build && /home/reinking/.venv/default/bin/python3.9 /home/reinking/test/tools/codegen.py /home/reinking/test/src/genmod/C.hpp /home/reinking/test/build/src/genmod/C.cpp
[4/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/A.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/A.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/A.cpp.o -c /home/reinking/test/build/src/genmod/A.cpp
[5/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/B.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/B.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/B.cpp.o -c /home/reinking/test/build/src/genmod/B.cpp
[6/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/genmod.dir/src/genmod/C.cpp.o -MF CMakeFiles/genmod.dir/src/genmod/C.cpp.o.d -o CMakeFiles/genmod.dir/src/genmod/C.cpp.o -c /home/reinking/test/build/src/genmod/C.cpp
[7/8] /usr/bin/c++   -O3 -DNDEBUG -MD -MT CMakeFiles/app.dir/src/main.cpp.o -MF CMakeFiles/app.dir/src/main.cpp.o.d -o CMakeFiles/app.dir/src/main.cpp.o -c /home/reinking/test/src/main.cpp
[8/8] : && /usr/bin/c++ -O3 -DNDEBUG  CMakeFiles/genmod.dir/src/genmod/A.cpp.o CMakeFiles/genmod.dir/src/genmod/B.cpp.o CMakeFiles/genmod.dir/src/genmod/C.cpp.o CMakeFiles/app.dir/src/main.cpp.o -o app   && :

And indeed we can see that the commands are what we'd naturally expect them to be.

edited Aug 8, 2022 at 12:39

answered Aug 8, 2022 at 0:11

Alex Reinking

20.7k8 gold badges68 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

einpoklum Over a year ago

Won't this generate the sources only at build time? Rather than config time?

Alex Reinking Over a year ago

Where in your question did you specify that you wanted the source files generated at config time? In any case, I wouldn't do that... incrementally building just one file would force the regeneration of all of them, not to mention all the unrelated work. And doing it at configure time locks you out of parallelism.

einpoklum Over a year ago

I didn't... which is why I gave an upvote. But - generation at config time could also in theory only happen if the file is missing. If in no other way, then by checking for the sources' existence. Anyway, I eventually went the Python route with a bash script as fallback if Python is missing. Perhaps you should add one should make it a python2-and-3 compatible script for maximum portability.

Alex Reinking Over a year ago

Sure you could check timestamps and whatnot at configure time, but you can't avoid all the other work your configure step might do, plus the generated build system would do that work for you anyway. Let each part do what it's good at. With Python 2 well past end of life, it is no longer worthy of consideration.

Alex Reinking Over a year ago

I wrote find_package(Python3 REQUIRED) and used the target Python3::Interpreter, so it will use whichever interpreter it finds that is some version of Python 3, not 2. Maybe that's a system interpreter, maybe it's a conda environment or venv.

|

Collectives™ on Stack Overflow

How should I setup script-based source generation with CMake?

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related