Libraries: A First Step Toward Standard C++ Dependency Management - Bret Brown & Bill Hoffman 2023

cppcon.org/
CppCon 2023 Early Access: cppcon.org/early-access
Access All 2023 Session Videos Ahead of Their Official Release To KZbin. At least 30 days exclusive access through the Early Access system. Videos will be released to the CppCon channel on a schedule of one video per business day, with initial releases starting in November.
---
Plenary: Libraries - A First Step Toward Standard C++ Dependency Management - Bret Brown & Bill Hoffman - CppCon 2023
github.com/CppCon/CppCon2023
Prebuilt libraries have existed for decades… they even predate C++! After all these years, techniques to use prebuilt libraries are still ad hoc and difficult to maintain. A root cause of this variety of techniques is the variety of things that are C++ libraries: header-only libraries, statically-linked archives, dynamically-linked binaries, and so on. The consuming projects need to build against these libraries in consistent ways or risk unproductive workflows - and potentially, even catastrophic failure in production environments. This lack of convergence creates enormous interoperability problems across broad portions of the worldwide programming ecosystem, not just the C++ parts of it.
This talk will explore the complexities of defining what is a “C++ library.” It will then present the joint work of Kitware, Bloomberg, and others toward a preliminary design for creating initial standards for dependency management in C++ - metadata files to describe prebuilt libraries. A roadmap for maturing the design will also be shared, including proposing a standard definition for C++ libraries, building on previous proposals such as P1313: Package Specification (wg21.link/P1313).
This talk is intended for anyone who produces, maintains, or consumes C++ libraries. Special knowledge of C++ tooling, build systems, or package managers is not required.
---
Bill Hoffman
Mr. Hoffman is a founder of Kitware and currently serves as Chairman of the Board, Vice President, and Chief Technical Officer (CTO). He is the original author and lead architect of CMake, an open source, cross-platform build and configuration tool that is used by hundreds of projects around the world, and he is the co-author of the accompanying text, Mastering CMake. Using his 20+ years of experience with large software systems development, Mr. Hoffman is also a major technical contributor to Kitware’s Visualization Toolkit, Insight Toolkit, and ParaView projects.
Bret Brown
Bret Brown is the lead of the C++ Infrastructure team for Bloomberg's Developer Experience department where he focuses on build systems, packaging standards, compilation toolchain support, and other ecosystem aspects for C++. As part of that role, he is also active in the ISO C++ Tooling Study Group (SG-15).
Bret likes making authoring and maintaining C++ codebases simpler and more intuitive by treating projects more like cattle and less like pets. He is especially interested in the software development lifecycle, development automation, modern build systems, packaging, code transformation, software governance, and code analysis.
Bret worked in embedded C++ and safety critical C++ for previous employers.
__
Videos Filmed & Edited by Bash Films: www.BashFilms.com
KZbin Channel Managed by Digital Medium Ltd: events.digital-medium.co.uk
#cppcon #programming #cpp

Looks like a useful first step towards a saner C/C++ package management. Kudos.

[@dbardbar]

Great talk. Sounds like a good idea, and definitely a step in the right direction. After spending many years in C++, and then many years in Java, it is so sad to see the state C++ is in, regarding packaging and usage of 3rd-party code. This is a solved problem in Java with Gradle. It is so liberating to just go and use a library, without spending weeks figuring out how to integrate it with your build system. I think this is the #1 thing that would make me think once, twice and thrice before choosing C++ for a new project. Modules are step in the right direction, but far from enough. IMHO, this is the most important thing the C++ community should focus on, much more than all the (great) language features introduced in recent years.

[@driedurchin]

I think the main counter argument (and I don't really know how I feel about it) is that according to some major C++ players (Google), you shouldn't, full stop, include code blindly using a package manager. If you are going to depend on third-party code, you should audit it, review it, test it, and integrate it wholesale into your VCS. Arguably this is the safe thing to do, but it just isn't practical to move fast.

[@__Brandon__]

​@@driedurchinyou can audit code and have a standard package manager. Look how rust and python, they are good examples of what C++ packages could be

[@kuhluhOG]

@@driedurchin having a system to easily include dependencies and actually auditing and reviewing it doesn't cancel each other out in practice people often behave like that, but that's quite frankly a different can of worms

[@TheDoomista]

I don't understand the statement "you cannot do headeronly with modules, you need some kind of structured information". Why can't I do headeronly with modules? Just put all of the code inside a single .ixx file, export symbols you need, it should work the pretty much the same way, doesn't it?

[@franciscomalmeida]

It will work fine you don't mind exposing the code, but keep in mind that the parser will have to go through the entire module source to build the list of exports each time, depending on imports it can slow down your build time considerably (yes, it can be worse than headers).

[@franciscomalmeida]

Module based source trees need a well defined hierarchy for optimal build times. If well structured build times are extremely fast, but if not, you won't see the gains. This is one of the reasons I usually recommend those who can to just design libraries with named modules (wrap old libraries if you must to isolate them). Header modules are proving too troublesome and honestly a poor middle ground solution.

[@gregthemadmonk]

Modules are not as "drop-in" as headers: after being dropped it, they need to be processed by a build system to resolve module dependencies, compiled into a BMI + an object file and only then could be used by other files. The compile times get significantly better, but you essentially NEED a build step for libraries now where all header-only solutions wanted you to do is copy-paste a file into your project.

[@joaopires51]

I believe the issue is that there's no way to describe the relationship between the name of a module and the file(s) that compose that module. Also, the name of the file and the name of the module can be different. i.e foo.ixx can export the module bar. At the moment, the only way to find the name of the module is to parse the code files. Again, I believe that this is how cmake works on a single project, i.e. all files are parsed to discover what modules they use and what modules they export. But this solution doesn't scale to all the libraries that you have installed on your system. So, some sort of metadata on each of the libraries is needed to keep track of that relationship

[@TheDoomista]

@@franciscomalmeida Funny, I didn't get notification on these replies. I assumed that modules will be faster than headers, because they essentially act as precompiled headers (I am ignoring the price for recompilation that you will not pay for third party library). So if a module is imported from more than one place, the compilation should be faster over normal header, even if the module puts all of the code in the interface file.

[@richcole157]

Hmm, what about gn? It is short sighted to think your build system is language specific. You can be declarative, but allow the build system to run things it built to provide additional information but it should be turtles all the way down. Then it is about well managed interfaces and build steps that take inputs and produce outputs. I think trying to reuse a multiplicity of package managers is a mugs game. The system should be able to build from scratch and it should pull binaries to avoid having to rebuild. It should have the notion of a workspace with some locally checked out packages. It should also have version sets for continuous build and deployments.

[@schekla]

Technical details: kzbin.info/www/bejne/f6jYc42mgbqkbrM

[@avgStan1234]

Nix is a purely functional package manager with over 80,000 packages. Why has something like Nix not been evaluated and studied for this C++ project/s?

[@danielelupo5224]

The idea seems good, but instead of creating a new standard/stuff/whatever, it's better to try to use and improve something that already exists, like vcpkg. I know that developers and researchers like to create their own solution instead of using or adapt existing ones, but I'm pretty sure that it will only increase fragmentation. VHS killed Betamax and become a standard, Git killed Mercurial and become a standard (and some company yet use Subversion not because it's better, but 'cause of the change friction), and so on. Just take a package manager that became the standard and kill other ones. Developers that created losing package manager will be angry, but it's better that trying to create an upper layer for integrating different package manager, build systems and so on.

[@kuhluhOG]

There is just the problem that there is no package manager which won in the C++ world. Sure, there are some big ones (vcpkg, Conan, arguably some functionality of CMake), but none have in use a big enough lead compared to the others to be considered a winner.

[@abdulshabazz8597]

The presenter's view a cpp library/dependancy manager as a coordinating file, similar to the way cmake is used, but... I'm beginning to realize a truly scale invariant flexible dependency manager will eventually evolve into an Operating System...

[@__Brandon__]

Pip, Gradle, and Cargo work just fine. We could get something atleast as useful into standard C++ within a decade

[@USGrant21st]

I think the build management went the wrong direction from the outset. What we see is not the solution, it's going down the same historical rabbit hole. We need all build management facilities to be condensed into a C++ API and become a part of the standard library, and compilation, packaging, dependency management, etc. would be just another C++ application.

[@yaroslavpanych2067]

Ehm, did you read what you just have written? I have a program, let's say "calculator". In what world anything in source code need to know or decide how exactly whole program is being built? What has standard library to do with build process? Who told you that standard library is available? Build stuff, dependency resolution stuff is out of scope of any language, fullstop.

[@jayenvanz]

@@yaroslavpanych2067have you tried Rust?

[@AnthonyDentinger]

IIRC, this is where they are going, but you can’t standardize something until you know you have a complete solution that works with the zillions of use-cases of C++. Remember that packages have been very diverse up to now, either because various contexts developped their own sets of conventions, or because different contexts just needed something different. E.g., some libraries need to be built with specific versions of the glibc; how would we express that? It’s not that these concerns cannot be answered; it’s just that it takes time to figure stuff out accorss the entire ecosystem, so you need to start with making third-party projects and see what experience that yields.

[@llothar68]

Just had to go through 80 build file written in make and cmake just last week to patch them so the SO libraries are not setting a version number because Android can't resolve symbols in a versioned dynamic library. And no patchelf is not enough to remove the versioning. Just mentioning to tell you i know a little bit about this things. I can say that this approach is still not helping and not the breakthrough we all hope for. If you do a deep analysis, so come back that libraries are just a list of file names and definitions. Thats all we need. The general export information (interchange between build systems and build scripts) must be simple and basic like like assembler code. When we have the filepaths and the necessary code generating cflags we can build them with different variants (like different sanitizers). It is not the library writers or the build script job to find libraries. Thats out of scope for the spec. Just tell use that we need to add HAS_LIB_SATAN to the files when we are possessed. All the dozens of different things that they are doing in a cmake build script are nice for the library developer but we have to separate it from the delivery.

[@mapron1]

That is 100% what CPS is for if you missed the point.

[@Tuniwutzi]

I'm not quite sure I understand you correctly, but I think you're viewing this only from the perspective of a vendor of a specific library. This proposal is attempting to standardize the way library vendors describe their deliveries such that library consumers can find and use them easily. Files + Flags may be enough for you library, but in the general case more things are needed. A format like this must also describe the dependencies of the delivered library (including which versions it's compatible with). Different variants (debug, release, header-only, ...), optional components and supported environment (hardware, OS, ...) are also very useful. The scope of the spec is: what does the consumer of a library need to know to be able to use it.

[@rasitsimsek9400]

The C++ is based mainly on C libraries, which is the weakest part of modern C++. There seems to be no progress under the hood.

[@MoAlyousef]

This sounds largely like .pc files, the only difference is the format (json).