Before Main: How Executables Work on Linux

[@lsx3.1416]

Please create a series about this! It's VERY insightful for us, not only as programmers but as Linux admins :)

[@doublepmcl6391]

Indeed, also this is making me more interested in Rust too :-D

[@pepzi_]

Yes, this would make a super interesting series!

[@lawrencedoliveiro9104]

1:01:31 The kernel is an ELF file, but it doesn’t have an interpreter or any shareable library dependencies. It has a format that a bootloader like GRUB knows how to load. Kernel modules are also ELF files. Again, they are a special flavour of ELF, that the module loader built into the kernel knows how to handle.

[@lawrencedoliveiro9104]

23:55 Debugger symbols are different from linker symbols. Debugger symbols have type information attached. And there are debugger symbol entries for local variables on call frames, which the linker doesn’t need to know about. Also debugger tables include line-number information for relating tracebacks to source lines.

[@lawrencedoliveiro9104]

1:20:26 This is called a “transfer vector”. It’s an area separate from the code, which is patched by the loader to link separate dynamic libraries together while keeping the code area untouched and therefore shareable.

[@adammontgomery7980]

I caught about 30% of this, but I'll go do some research and come back. This is why I love KZbin; you've earned a subscription!

[@lawrencedoliveiro9104]

1:05:05 The interpreter name is in the INTERP segment, which is also mapped by the .interp section in the ELF file.

[@lawrencedoliveiro9104]

27:26 There are some differences in the low-level interface to the Linux kernel on different processor architectures. For example, the dispatch numbers assigned to system calls are different. libc hides all that from you, and lets you write portable code. If you bypass it, then you have to deal directly with those sorts of headaches. libc is written in C (and I think some assembler). Since it runs entirely in userland, there is nothing it can do that you can’t do directly, if you really want to.

[@shocb]

Hey Ryan -- this was awesome. Thanks, I learned a lot ! Can I make a small suggestion? The actual work you're showing us is only using about 60% of your screen real estate. It would be great if you maximize it to full when showing something to put focus on it. Also I think this is a good LPT when working on your own too :D

[@lawrencedoliveiro9104]

21:06 The error is happening because ld is language-agnostic. When invoked via the C compiler, the latter includes “-lc” so libc is included in the link; without an explicit option mentioning a library, ld won’t include the library.

[@lawrencedoliveiro9104]

56:25 This is actually extensible in Linux, through the “binfmt” mechanism. It lets you register handlers for any kind of executable format. For example, most scripting languages are supported through that “#!” mechanism, but Java .jar files don’t have such a marker, so you need to add a special handler for those.

[@lawrencedoliveiro9104]

1:27:52 Actually, no. The user stack is a userland ABI thing. The kernel has its own stack, but that’s separate. x86 has the concept of a stack register at the hardware level, but POWER/PowerPC does not, for example.

[@lawrencedoliveiro9104]

19:48 Those symbols __libc_csu_init and __libc_csu_init are defined in the static version of libc (/usr/lib/x86_64-linux-gnu/libc.a on my system), not the dynamic version. They are also in /usr/lib/x86_64-linux-gnu/libc_nonshared.a, which I think is used together with the shared version of libc.

[@lawrencedoliveiro9104]

55:25 It calls fork() to create a new process, and it is the child process that does the exec().

[@lawrencedoliveiro9104]

53:54 The “??” is the routine name, which is unknown (no debugger symbol for it). The “()” means the routine has an empty argument list.

[@UFO_researcher]

A shared object elf is a binary in the purest sense. The dynamically linked libraries are to allow them to be upgradable. Also, statically linking makes a binary more than 10 MB, it is considered bad practice to statically link. Libraries in Linux are compiled similar to binaries, they have a .so extension (shared object). In Windows they are called .dll (dynamically linked library) files.

[@TehGettinq]

Amazing content, thanks a lot it was a real treat, learned a lot. Looking forward to other videos from both of you, Amos should definitely start a youtube (or whatever other video streaming platform he prefers) channel, would love to watch his stuff too.

[@lawrencedoliveiro9104]

1:08:08 Or just use gcc with the “-static” option. Produces larger executables, though.

[@lawrencedoliveiro9104]

1:00:52 Note that ldd only shows direct dependencies, not dependencies of dependencies. If there are any failures resolving any dependencies (direct or indirect), you will get a “file not found” error trying to run your executable. And then you sit there baffled, because you know you typed the name of the executable correctly, so why can it not be found?

[@lawrencedoliveiro9104]

1:26:28 ELF is a more advanced format than, say, Windows PE or DLL files. It doesn’t need hacks like “binding” to improve loading speed.

[@pinch-of-salt]

Please do more videos like this! rustlang and cool tools!

[@lawrencedoliveiro9104]

1:15:42 Use the “-g” option in gcc to build an executable with debug symbols.

[@lawrencedoliveiro9104]

59:02 libcanberra is a high-level interface to GUI audio feedback.

[@lawrencedoliveiro9104]

1:02:44 file seems to say that every ELF file is “dynamically linked”. Maybe there is a special flavour of ELF known as “statically linked”, which Linux does not use.

[@lawrencedoliveiro9104]

No, I just used gcc with the -static option, and it produced an ELF file that identifies as “statically linked”.

[@yumimato]

It would be helpful to include links in the description.

[@RyanLevicksVideos]

I've added some links! Will make a habit of this for the future.

[@lawrencedoliveiro9104]

50:37 Try “!ls”. Or if you want to temporarily suspend your GDB session and get back to the parent shell prompt, do CTRL/Z.

[@lawrencedoliveiro9104]

14:09 So removing the default unwind handling saved maybe 40-50K at most.

[@lawrencedoliveiro9104]

58:51 By the way, did you know that ldd is a shell script? Have you looked at how it works?

[@lawrencedoliveiro9104]

34:16 What would be “the purest sense of the word”?

[@jkjensen]

This was great but it was really hard to hear Amos :(

[@lawrencedoliveiro9104]

10:51 Why isn’t all that in a shared library?

[@lawrencedoliveiro9104]

1:01:05 Notice there is no filename listed for that “linux-vdso.so.1” shared library. It doesn’t actually exist in any file: it is built into the kernel. That is where system call entry points are to be found.

[@lawrencedoliveiro9104]

6:17 Why do you need to do a search of your history? Just keep pressing up-arrow.

[@lawrencedoliveiro9104]

36:11 And as a result, Windows is prone to “DLL hell”.

[@lawrencedoliveiro9104]

17:14 Tip: for outputting simple literal strings, use “puts” or “fputs” instead of “printf” or “fprintf”. Why?

[@mahmoudfarouk2759]

fasterthanli.me/series/making-our-own-executable-packer The link for those who want it.

[@lianghuayu240]

How to min size on windows?

[@lawrencedoliveiro9104]

52:38 But this program is itself an ELF file.

[@lawrencedoliveiro9104]

50:49 Try “bt -past-main”.

[@danielegger6460]

Not using cargo-binutils to inspect your binaries? zt, zt, zt...

[@slowerthanstone]

Haha, I wasn't sure Ryan had them installed and didn't want to spend valuable stream time installing them!

[@lawrencedoliveiro9104]

1:12:57 The length of the string.

[@lawrencedoliveiro9104]

36:00 If you want a good presentation on segments and sections and ELF generally, try this kzbin.info/www/bejne/pHSUhmSCf7Z_btE .

[@lawrencedoliveiro9104]

32:11 That “V” is for “five”.

[@lawrencedoliveiro9104]

58:10 Or use “file -L”.

[@thewelder3538]

A perfect demonstration of why Rust is just useless. My executable that does the same as this Rust executable is 140 bytes and I wasn't even trying.

[@lawrencedoliveiro9104]

7:18 Just type “!vi”.

[@lawrencedoliveiro9104]

1:03:11 Guess how the ldd script works.

[@lawrencedoliveiro9104]

56:54 Found one: /sbin/ldconfig .

[@lawrencedoliveiro9104]

1:20:56 That one.