CppCon 2017: James McNellis “Everything You Ever Wanted to Know about DLLs”

CppCon.org
-
Presentation Slides, PDFs, Source Code and other presenter materials are available at: github.com/CppCon/CppCon2017
-
If you build software for Windows, you use DLLs, and it’s likely that you may build DLLs of your own. DLLs are the primary mechanism for packaging and encapsulating code on the Windows platform. But have you ever stopped to think about how DLLs work? What goes into a DLL when you build it, what happens when you link your program with a DLL, or how do DLLs get located and loaded at runtime? Many of us build and use DLLs without fully understanding them. In this session, we’ll give an in-depth introduction to DLLs and how they work.
We’ll begin by looking at what’s in a DLL-the kinds of things a DLL can contain and the basic data structures that are used-and the benefits and drawbacks of packaging code in a DLL. We’ll look at how DLLs are loaded, including the details of how the loader locates DLLs and maps them into the process; how dependencies are resolved among DLLs; and DLL lifetime and how DLLs get unloaded. We’ll also look at how DLLs get built, including what makes DLLs “special,” what goes into an import library, and how the linker uses import libraries. Finally, we’ll look at several other miscellaneous topics, including how DLLs interact with threads and thread-local storage, and mechanisms for solving or mitigating the dreaded “DLL hell.”
-
James McNellis: Microsoft, Senior Software Engineer
James is a senior engineer on the Windows Debugger team at Microsoft, where he works on the Time Travel Debugging (TTD) reverse debugging toolkit. Prior to joining the Debuggers team in 2016, he was a member of the Visual C++ team, where he was responsible for the Microsoft C Runtime (CRT) and C Standard Library implementation. Passionate about all things related to C++, he is a frequent speaker at C++ conferences around the world and is a former top contributor on StackOverflow. He can be found on Twitter at @JamesMcNellis.
-
Videos Filmed & Edited by Bash Films: www.BashFilms.com
*-----*
Register Now For CppCon 2022: cppcon.org/registration/
*-----*

[@llothar68]

Please do a "Everything You Ever Wanted to Know about PDB's"

[@Dhairyasd]

There is a difference between a good teacher and a regular lecturer. A good teacher realizes the process of learning of the audience and presents the topic in just the exact order of how the audience is interpreting all of the topics. While the just-another-lecturer will present all of his topics randomly across the lecture. This was an amazing talk from an amazing teacher. Really opened up my mind and taught me a lot about DLLs. When you learn so much about a single topic in such limited time, you just can’t stop thinking about who taught you about that subject. Thank you for the talk and I feel forever indebted to the speaker for his effort in providing me with such valuable information. Thank you!

[@rationalcoder]

If you wan't to comment about this being useless b/c it is windows specific, note that a "DLL" _is_ Windows specific, so I'm not sure why you even clicked on the video. I find that this type of platform-specific detail enlightening, as Windows can be a bit impenetrable sometimes. For example, until this video, I didn't have a good explanation for why executables link to __imp_ symbols. Now I do. That kind of information alone makes this a good talk. EDIT: The name aside, "DLL" _is_ Windows specific in this context. Anything else is nitpicking. You knew what you were signing up for. The platform agnostic term is "shared library."

[@SE45CX]

I do love his specific exemplified talk. It communicates much clearer and faster than ambiguous general talk.

[@FindecanorNotGmail]

No, "dynamically linked library" is a generic term older than Windows 1.0. BTW. "DLL" was also used for Symbian, which used different file formats.

[@VivekYadav-ds8oz]

Even then, this is interesting because concepts do carry over to .so files in Linux, like how it indirectly calls the DLL functions via the __imp_ thing you talked about.

[@victornoagbodji]

almost done watching this talk. such a great talk and amazing presenter. need more from the visual studio team : )

[@khatharrmalkavian3306]

I know this is an old post, but check out Microsoft's "Channel 9" website if you want to hear from that lot.

[@0xF33D]

If only this talk was half a year ago! :) Highly enjoyed it anyway, though. Very concise and on point! Knowing how hard these topics are - I've spent weeks researching my problem, and it's not even covered here, it just so happened that I needed to know how DLLs work, - I'd like to get more of these videos in the future. Thanks!

[@VPNuser]

I just finished watching this video on KZbin about everything for DLLs, and I wanted to express my appreciation and praise. I am truly grateful to you for sharing such a practical tutorial that has been incredibly helpful for my software development work. Your explanations were clear and concise, and I now have a complete understanding of how to load DLLs and incorporate them into my programs. This is a crucial skill for me, and your video provided many valuable tips and suggestions. Thank you once again for sharing your knowledge and delivering an excellent presentation!

[@gast128]

Nice talk with lots of new details for me. The last question addresses the issues of exporting C++ which is okay as long as you stick to compiler versions and options. The only omission of the talk is the normal way of specifying import and export attribute in a header which solves many of the maintenance issues.

[@matthews4795]

Thank you so much for this talk! I've always wondered how DLLs are structured.

[@jonathanemery9557]

I was trying to learn about relocation and this talk made it stupid simple. Amazing presentation!

[@hl2mukkel]

Highly interesting talk! I enjoyed this one :)

[@314dez]

THE BEST EXPLANATION EVER!!!!!!!!!!!!!!!!!!

[@ahmadalastal5303]

Excellent talk about DLLs, well done James

[@bonbonpony]

Nice :) Now someone has to make such video for shared objects on Linux.

[@Frozander]

Now I can finally delve into the mysterious realm of Windows specific programs.

[@mohammadalaaelghamry8010]

That is the right content. Thanks. It's greatly useful.

[@Leonardo_A1]

great presentation in a effitient time ... please create more of this . CU Leonardo

[@AbacateSexy]

Shame he didn't have time for TLS and DLL Hell. I usually hate watching lectures but this was a joy to listen to

[@jackninja1]

Very good presentation!

[@okaminess]

Very helpful. Thanks, James.

[@abrrrtoooasd]

Great video, learned a lot. Thanks.

[@MobilTemp]

Amazing man!

[@corpsitism]

Thank you Jay Mac!

[@Fetrovsky]

"...and I don't know why anyone would use this..." I actually like this option better than the rest.

[@georganatoly6646]

as a tool writer new to targeting win32/PE this was fantastic

[@bersi3306]

This guy is awesome! Funny stuff.

[@karloes0]

10:18 I think PE start address is stored in 2 bytes: 3C (low byte) and 3D (high byte). But in this example 3D is 0x00 so it is correct.

[@User-cv4ee]

Great talk! To the point and precise. I was shocked by the recommendation not to export C++ functions/classes across boundaries. What is one supposed to do then? Only C?

[@DatMilu2K]

I loce this guy. :D I always try to disassemble my binaries with notepad. xD

[@FindecanorNotGmail]

He mentions "The loader" _many_ times in the talk but does not mention what the loader is. On MacOS and Linux, the loader of dynamic libraries is first loaded into the program's address space and then _it_, actually being the program, loads all libraries itself before calling main(). Does MS Windows work in the same way?

[@bit2shift]

No, the loader is an integral part of the kernel, not a separate program.

[@pianoblader4263]

I'm sure the loader is in kernel mode, but on windows the ntdll.dll is first loaded into the process and is used to create stack/ heap and load dlls into virtual address space. Someone correct me if I'm wrong.

[@cortexauth4094]

Windows Internals Vol. 1 describes the "Image Loader" in great depth

[@ecosta]

That was much better than I expected. But I can't stand that "A:\" prompt... I can almost hear a 5 1/4" floppy disk spinning...

[@multiHappyHacker]

wow this guy talks really fast, lots of very interesting stuff.

[@taichu5625]

my two weeks searching and stackoverflow contains in this video

[@zhaoli2984]

Nice talk

[@AlexSmith-fs6ro]

Is it possible to define a variable in a dll that can be shared across processes that load up the dll?

[@davinciclone7277]

Yes if it's read only.

[@jorgellanque7704]

No words

[@eitantal726]

16:51 you set RAX = 0x70002000 . This is a pointer to char*. But the DLL isn't actually mapped to this address. When you RET, you end up with a pointer to something you didn't want. Your pointer (in RAX) should equal 0x90002000. What am I missing here? is LEA doing a remapping?

[@eitantal726]

Answer: The disassembly interpretation is misleading. The actual opcode doesn't assign an absolute value, but a relative one. In the disassembly, it just looks like an absolute address. This is the real disassembly: lea rax,[rip+0xff9] # 0x1000 LEA does not remap. LEA is just a glorified ADD, that unlike add, can accept different registers. ADD cannot take EIP as an operand

[@user-hn4vs6wx6t]

👏👏👏

[@cortexauth4094]

12:00 it's similar to how IA-32e is tho : D

[@tohopes]

0:23 bait and switch fucker. *gets up and leaves

[@tohopes]

this is great info btw. very dense and well organized presentation. Microsoft should have done this years ago but I'm glad that Mr. McNellis has done this in such a nice form. even the questions are good.

[@deckard5pegasus673]

3:49 Actually this is mostly false. Dlls most times are not sharing code, and each process has it's own copy of the dll. Which is actually quite crazy, because the whole idea of dlls in the first place was to share code, but microsoft ended up deciding it was a security risk.

[@txcpnae]

Take a shot every time he says DLL 😅😅

[@DFsdf3443d]

why extern "c"?

[@Ameraygo]

extern C won't mangle your exported functions symbols. If you call GetProcAdress to get imported function pointer you need to provide the function symbol. If you use extern C then function AddNumber() will have AddNumber() symbol and you can get it by GeTProcAddress("AddNumber"). If you didn't use extern C then AddNumber function would have it's symbol mangled randomly to something like YZ@Add*5Number. The person using the dll would need to do GetProcAddress("YZ@Add*5Number"). Those symbols could change in different .dll releases so a user would have to change their code

[@MrM12LRV]

Nice explanation :) Just want to clarify: the name is mangled based on the function's signature; it is not a random process. Since C has no knowledge of C++ constructs such as namespaces, classes, function overloading, the C++ compiler will have to mangle the names in a predefined way so that the function can be uniquely identified. This predefined way allows C++ compilers the ability to reconstruct the function's signature from just the symbol and find the function.

[@bonbonpony]

Another thing worth adding is that when you know the pattern of this mangling (and there _is_ a pattern, it's not "random", and there are standards for that, e.g. the Itanium ABI documents), you can actually call C++ functions across the DLL boundaries, export C++ classes from a DLL, etc. Especially when you use the same compiler/linker for both the executable and the library (because their ABIs need to match precisely in order for this to work).

[@FindecanorNotGmail]

C++ name mangling is compiler-specific, but on each major OS there has in practice been one compiler that has set the standard. On Windows, it is MS VC++ (the name mangling seen in dumps in this example). Borland and Watcom compilers used different mangling but those are not popular anymore. GCC has used different formats in different versions. I think clang uses VC++ format on Windows and the latest GCC format on Unix/Linux systems.

[@bit2shift]

@@FindecanorNotGmail GCC and Clang use the Itanium C++ ABI. On Windows, you can use that ABI on Clang by using the MinGW toolchain.

[@huyvole9724]

look like DLL injection man :)) you watched it?

[@StEvUgnIn]

What most JNI classes hide from you

[@CamiloDiazBogotaDC]

this presentation is too dense to process mentally speaking., :/ almost got a head ache. I had to watched it multiple times to understand most of it.

[@LaurentLaborde]

7:51 how to be flagged as potentially harmful by a malware analyzer :D

[@Dziaji]

I started watching this, then i realized that he was right. I probably don’t want to watch this.

[@tobiaskarl4939]

A:/ load from old diskette ... *shrubb* .... *shrubb* wait for loading 1.44 MB 🤣

[@sent4dc]

That's a downvote from me for skipping the most interesting stuff at the end. He went thru the simple stuff and then skipped lesser known stuff like delay loading, thread-local-storage, etc. Why do we have to limit this by an hour? So annoying!

[@antonnadolskiy3063]

Slide on 26:16 - this won't work because of DLL name mangling. Even Microsoft doesn't know how their things works, blah..

[@mitchellyism]

extern "C" tho

[@schulmastery]

rekt

[@ArmanAli-dn9ou]

usually the people who do the actual work don't come for presentation...

[@kcvinu]

I am using D programming language and we have a thing like pragma(mangle, "name_of_the_func"). You can make sure your function name is not mangled to something else.

[@retropaganda8442]

James McNellis probably is going to enjoy using UNIX. I think he's loosing his time at Microsoft.

[@jankodedic3130]

I once programmed in DLL, not recommend

[@TheEVEInspiration]

How about link the libs directly into your executable whenever there is no run-time reason to do otherwise? I always did things this way and its better at everything. The amount of code shareable outside of OS specific things, between concurrent applications/processed is small and not worth the overhead of dynamic linking at run-time. And application specific DLLs are almost never needed unless you have a plug-in system that allows others to inject extensions. Not a situation your typical application be like, in other words, DLLs are virtually never needed, unless someone fucked up bad. Also note that most applications use only very tiny portions of the libs they need. A linker can remove the dead waste during final executable creation and optimize memory layout. With dynamic loading you cannot get rid of this overhead.

[@bonbonpony]

Well, if you have shitloads of disk space and memory to waste, go on. But I'd rather like to have the same code kept only once in my memory and on my disk, to be shared among executables, even if it costs a couple of CPU cycles on each call. Calls across the module boundaries aren't something that one would do in a tight loop anyway.

[@stefangeorgeclaudiu]

If you have millions of clients and every new version/security fix you have to upload 10MB executable instead of 1MB DLL that is one hell of a bill for network traffic and unsatisfied customers that will either stop updating or uninstall the application.

[@catalinvasile9081]

I work regularly with 1-2 GB libraries and statically linking them takes tens of minutes. Using DLLs drops their size by 100x (since the DLL itself is linked, so duplicated template symbols are removed). So huge link time optimizations. Another reason is when using QT: statically linking them in 10 applications results in 2GB worth of exe files. With DLLs you all the files (10 exes + QT DLLS) are around 100MB total

[@bit2shift]

@@TheEVEInspiration DLLs allow global objects inside themselves to be safely and cleanly destroyed even when the main program exits abruptly.

[@stephenkamenar]

dll are so dumb. why can't i include the dll in my exe. i don't want all these stupid dlls

[@MrAlgebrain]

#pragma comment is used by boost

[@garychap8384]

And this is at CPPCon why? I hope to give a talk next year on making perfect smoothies using the Nuitribullet blender.

[@alexandrupana6037]

Are you trying really hard to be an idiot, or does it come naturally to you? Go and read what CppCon is about: "Presentations by the C++ community: What do embedded systems, game development, high frequency trading, and particle accelerators have in common? C++, of course! Expect talks from a broad range of domains focused on practical C++ techniques, libraries, and tools." and "CppCon’s goal is to encourage the best use of C++ while preserving the diversity of viewpoints and experiences, but other than that it is non-partisan and has no agenda." Now go fuck yourself :)

[@DatMilu2K]

Alexandru Pana Very well said my friend. :)

[@garychap8384]

Alexandru, why so triggered? I expressed an opinion that this talk has nothing to do with C++. Hardly a reason to tell someone to "go fuck themselves" I defend my comment. Mainly because I'm tired of going to events (I spend about £4000 a year chasing industry cons) and finding about a tenth of it relevant, and the rest just filler, off-topic platform stuff you get at the vendor conferences and copious advertorial nonsense. You have a different point of view... and you know what? I'm totally cool with that. Hence no expletives, insults or anger. Hell, I'd even buy you a beer. As an severely autistic male, it's not very often that I'm the one who has to point out a humour deficit : ) Thanks for that XD

[@alexandrupana6037]

GaryChap, appologies for my crude outburst, I always assume I'm replying to trolls when I comment on youtube. My issue with your point of view is that you assume everything must be useful to you personally, and that if it's not, then it's not related to C++ or useful to anyone else. Moreover, DLLs are pretty closely related to C++, whereas "Nuitribullet blender" not so much. By asserting that this doesn't belong to Cpp Con, you're basically telling people who find it interesting that they don't belong to Cpp Con. Which I believe is rude and wrong.

[@origamibulldoser1618]

I don't think explaining the usefulness or purpose of dynamically linked libraries to a CppCon audience is necessary.

[@TheEVEInspiration]

They aren't useful at all for the vast majority of applications. The memory "advantage" he mentioned simply ain't there, the idea is purely rooted in theory.

[@bonbonpony]

Even if it's not useful to you or me, because we already know this stuff, it doesn't mean it can't be useful to someone else who didn't know how dynamic linking works. I was programming in C++ for a couple of years already until I learnt about dynamic linking.

[@nullplan01]

@@TheEVEInspiration Where do you get this from? He claims at the end that the read-only pages will be shared between processes, and since Linux implements the exact same thing with mmap(), I have no reason to doubt him. I know it is feasible and indeed done on other platforms, and he claims this will be done in Windows as well. And he works for M$, so might as well believe him.

[@kalindimitrow2087]

useless coz OS specific

[@hl2mukkel]

you wouldn't say this if it were for your platform

[@quinson93]

useful on specific OS

[@kalindimitrow2087]

it's cppcon not specific OS con c++ is multiplatform language

[@AaronKoolenBourke]

There's tonnes of talks at CPPCon that discuss things that are for linux only

[@kalindimitrow2087]

If it OS specific it is bad talk it's not part of the c++ standard and as such it shouldn't be part of this conference (this include linux talks mac os talk and etc)