Better CMake Part 6 -- Superbuilds w/ ExternalProject

[@kebabimpaler]

I have used external project for many many years. I had no idea that it was capable of doing the half of what you just outlined thank you lol.

[@silverqx]

I'm at 2:58 in your video and I can tell already that it's a super helpful video thx 👌

[@alexiswinters9364]

I'd really like to see a video about how to generate a CMakeConfig file so that I can use a project in downstream CMake packages.

[@jeffersonamstutz]

Sounds good! I think I’ll do that as my next topic!

[@krofikS4]

Thanks for great content! Please keep going. It would interest me, how can I setup CMake such as when I 'make' VS Solution, these follow same structure as if I created them by hand. And by that I mean: setting build directories (for release and debug), showing actual folder structure inside project explorer. Is there any such way to specify layout for VS Solution?

[@jeffersonamstutz]

Unfortunately I’m not well versed in vanilla VS - I only have built VS projects generated by CMake as I primarily develop code on Linux.

[@Kaszanas]

Hello! This is great information. Thank You. There is not much info online that is easily accessible on how to handle dependencies like that. I would like to be able to include different dependencies and handle them automatically like shown in this example. Is the CMake code published anywhere? I do not necessarily understand later on how one is supposed to include the built dependencies into the current project.

[@literaltechbros5833]

Really great stuff! My only question is, how do you refer to dependencies when you're writing one of the subprojects? For example, if I added build subproject(NAME mySubproject...DEPENDS ON rkcommon) would I automatically be able to include header files from the rkcommon project in my mySubproject project? Or would there be more to it?

[@jeffersonamstutz]

Each project should use find_package() to go find dependent projects. The superbuild dependency tracking is about making sure each project is built in the correct order -- it doesn't actually affect the underlying project's CMake itself. In this case, all projects which use rkcommon each do their own find_package(rkcommon) and then link exported rkcommon targets, which is where things like include paths and library files get incorporated.

[@arkadymagomedov1700]

The "*_ROOT" variables you set manually are required for the find_package, right? I am trying to figure out how to compile a complex library (the one that has external dependencies that have other external dependencies, etc...) to web assembly using emscripten tools. Most libraries rely on find_package. But for emscripten to work all libraries must be compiled with emscripten (duh...).

[@jeffersonamstutz]

Yes, in the multi-project example you see TBB_ROOT specified as a CMake argument to each project to find the TBB that was extracted locally. As for emscripten, I haven't used that toolchain before so I'm not familiar enough with it to know how one should approach dependencies. I would hope that there's a way to find_package() libraries built with emscripten (?), but I'm not sure how much native CMake can handle that beyond header-only interface libraries. Sounds like an important problem to solve! One thing I've been experimenting with lately are "source libraries" (not sure if there's an official name for it). The idea is that you can use target_sources(${TARGET} INTERFACE sourceA.cpp sourceB.cpp ...) on an CMake interface library target, which will then propagate the cpp files themselves downstream to targets which link it. This is problematic if you don't control all the dependencies, but is definitely an option if you are able. If anything, you could create a superbuild of external projects with your own custom CMakeLists.txt for each project that creates a "source library" of each one manually. However, because there isn't really a build step for these types of "source only" libraries, you could just do it with FetchContent instead all in the furthest downstream consuming project's CMake build.

[@arkadymagomedov1700]

@@jeffersonamstutz I do not think it is feasible with FetchContent as it does not set the *_ROOT variable. So if you need a library A which depends on library B, you need to first download and compile library B. Library A uses find_package(B) which would try to look in system folders, which is not desired. FetchContent does not set *_ROOT variables. This guy invented a wheel kzbin.info/www/bejne/g5fYc4Zvo5qSl7s&lc=UgwlVI4oxNHL8Os3fft4AaABAg.9RcmsnQR_6E9RcqDPaKY30 And there is this proposal as well: gitlab.kitware.com/cmake/cmake/-/issues/21687

[@stevenbroshar7948]

What's a superbuild? ... is that your word for a build that uses ExternalProject?

[@illiadubrovin8995]

superbuild should be first think you hear and learn about cmake.........

[@stevenbroshar7948]

Please don't start with the docs. CMake docs are TERRIBLE! (like most product docs)

[@danielt5542]

Hi @jeffersonamstutz, how can I link a library built from ExternalProject_Add into an executable? the target name given by ExternalProject_Add is of type UTILITY and cannot be linked directly using target_link_libraries(app libA)

[@jeffersonamstutz]

To get external dependencies into your build you will need find_package(), but ExternalProject runs at build time which is after when CMake would search for it during configuration. The best thing to do is to write your project that builds only itself and maintain a separate CMake superbuild that can string together ExternalProject invocations. You certainly could mix these in the same directory structure by only invoking your executable’s CMake from a parent folder containing the superbuild. You would do this by having the ExternalProject invocation bypass the download step and point directly to your local source directory. Be careful though - it is bad practice to always assume overly rigid dependency handling by requiring that you can only use the deps built locally.