Debugging with Core Dumps

[@fbafelipe7666]

keep in mind that sending core dumps may have privacy implications. For example, if Excel or Word crash and send a full core dump to microsoft, it would be sending the document the user was viewing/editing as a side effect (since it is loaded on memory), it is also true for any program data manipulate user files/data.

[@karlforshaw]

Another great video, and a massive help for all of us who love coding in C but haven't had much of a chance to use it professionally. Thanks Jacob!

[@JacobSorber]

I like C. It's pretty low-level, and it occasionally feels like chopping down a tree with a really sharp pocket knife, but from a language-standpoint it's pretty simple. The complications mostly come from the underlying system, which it doesn't hide from you-not the language.

[@RakeshKumar-vm9xo]

@@JacobSorber True..

[@thewavewarden3315]

"You could fill up your disk pretty quick, especially with how frequently your code crashes." ROASTED

[@kenwallace6493]

Mile-a-minute, dense, chocked full of stuff I missed or never knew. Wish I'd had a prof this knowledgeable when I went to college. Great stuff, Jacob!

[@dineshbharti4454]

Being a part of organisation as a QA, I find this video very helpful. Explained in a way which is very easy to understand. Thank you and keep sharing the knowledge.

[@chasedoe2594]

That explosion break my ear lol

[@JacobSorber]

Sorry. I hope it heals. Segfaults are serious. :)

[@hsaidinsan6345]

Your comment saved my ears 2 seconds before the Big Bang Thanks a lot 😅

[@YuanweiLao]

Same here

[@thepinback]

Nice video! But of the most helpful things with core dumps is the ability to walk the call stack in gdb.

[@User_Masuri]

There isn’t a better video than this in the world!!

[@mayuparge]

@Jacob Sorber... one of the best video on core dump analysis ...Thanks Jacob

[@JacobSorber]

My pleasure. Glad you liked it!

[@abhisheksa6635]

Professional grade explanation, much appreciated.

[@samjohnson5044]

Thank you for this! I have a GTK widget that crashes every other leap year, will examine with the core dump.

[@GeekyMino]

Hi Jacob , this was a great video but I still have a question. When you deploy your code you are usually using release build which is optimized and probably won't have the debug symbols and further more the target might not have a lot of memory. So how can we use this core dump in such scenario?

[@lyingcat9022]

Good question. Sad I see no answer:(

[@amritagrawal6681]

Superb... Gonna watch all of his video... Thanks a lot sir...

[@JacobSorber]

You're welcome. All the best.

[@yannisran7312]

Loved the analogy and animation

[@JacobSorber]

Thanks. We had fun making it.

[@matsim0]

IAs someone who writes code for embedded systems and uses mainly python for tasks on my PC, this was _really_ interesting, because I didn't ever do a core dump - I mean, normally I don't even have a file system 😅

[@andrewseaman2840]

Thanks. I enjoyed your video. I'm trying to run RLCraft (a modpack for Minecraft) and have fairly rudimentary coding skills and knowledge. Anyway, the crash report that I got when I loaded the launcher said, "Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again." So that's how I found your video. I still don't know what to do to fix the issue but I do now understand what it's talking about. Thanks

[@axlslak]

love the analogy with space travel. coredump are precious things. :)

[@PrashantMarshal]

For a detailed discussion, you could have talked about the "??" problem when debugging with coredump files.

[@JacobSorber]

Good idea for a future video. Thanks.

[@BARaaz04]

Great video... Thank you, Professor

[@stelluspereira]

Thankyou for helping the ‘C’ community

[@moseschristopherb]

How to deal with coredumps that are generated/dumped in the target which is running on a different architecture? Assume, -> Host is x86 and Target is armhf -> Target doesn't have gdb/memory to debug on Target itself.

[@swedishpsychopath8795]

I always thought that a core dump was something you had to submit to the CPU manufacturer (Intel/Amd/etc) or to NASA so that technicians in white lab coats could read the machine code that was in the CPU at the moment after the program crashed using an open CPU with lots of wires connected to it and where they looked at the contents of the memory with an electron microscope. But then it turns out that anyone can look at the program code with a simple debugger. Why couldn't they say this 40 years ago when I started as a programmer?

[@SimonJentzschX7]

great video!, Since you usually don't release with debug-symbols, I wonder how useful the coredump may be in this case. How can you extract anything without debug-symbols?

[@JacobSorber]

Good question (and good idea for a future video). Debugging without debug symbols is definitely more painful. You can still look at the assembly. You can also release with debug symbols (accepting the downsides). Or, you can generate two versions of the binary (one with and one without debug symbols) and then use the debuggable version with the core dump. For release builds, I think I favor the last one.

[@SimonJentzschX7]

@@JacobSorber Right, But as far as I understand it, if I put in Debug-Symbols I cannot use the optimize-flags.

[@shushens]

Most of them are sending minidumps, along with other possible values. It would be lovely if you made a video on minidumps as well.

[@JacobSorber]

I'll see what I can do.

[@MinusGix]

Thanks! Your videos are very informative, and have helped me a lot.

[@JacobSorber]

Glad they're helpful. Let me know if there are other topics you think I should cover.

[@MinusGix]

​@@JacobSorber A video on the Linux (POSIX?) LD_PRELOAD environment variable would be interesting. Solving circular includes (such as with include guards) would be a useful video for people writing code. (Assuming you haven't already covered these, haven't yet watched all of the videos. They're quite good and to the point.)

[@JacobSorber]

@@MinusGix I have covered LD_PRELOAD (see the videos on shims) and include guards (fairly recent video on headers).

[@MinusGix]

​@@JacobSorber Ah, ok. I had been watching the videos from oldest to newest to make sure I watched them in order, and hadn't gotten to them yet. There is inline asm (unsure if clang has it, but GCC does), but that's rather obscure and most people don't use it. Videos about common C patterns, such as common ways of structuring your programs would be interesting. (Such as writing code that uses flags, and why you would do such a thing.) Could go into common ways that C++ library methods are written using C. This bit me when I recently tried using the Windows Direct2D api as the way it's structured to translate to C isn't obvious (the docs are all for C++ as well..) but I don't know if that's a common way of translating C++ to C or not. There is writing a less-featured clone of a linux utility (perhaps ls?) and comment about the design decisions you make as you write it. That would be different from your usual videos though. Since you seem to do embedded fairly often (?) you could talk about writing efficient code, and how the compiler also at times knows better than you. Examining the assembled output of simple C programs to see how there can be differences. (godbolt.org is quite nice for such a thing, but this would also be a good way to also show some linux utilities)

[@karthik2705]

One of my binary crashed. And the bt is pointing to (static) library part I tried compiling library with debug sumbols enabled. But the bt is pointing to wrong code How to debug this scenario ?

[@tobb10001]

So... If I set the limit to a finite number and the core dump gets truncated... how useful is it gonna be than? Can I be sure to at least have the stack in it then?

[@MichelCarroll]

kill -SEGV 4787 "Oops, wrong pro..." *universe explodes*

[@simonmaracine4721]

This was very helpful. Thank you!

[@GabinoHernandez-kt9js]

I LOVE YOUR VIDEOS!!!!!

[@GrozaRobertGenaro]

great, but an issue, i have a mac, running vscode on its terminal, when i tried gdb but it says "bash gdb command not found"

[@JacobSorber]

Yeah, gdb doesn't come installed by default on MacOS. Also, Apple favors clang and lldb, and has done some things that make it tricky to get gdb working well on MacOS.

[@GrozaRobertGenaro]

Jacob Sorber yeah, lldb worked

[@netanelkomm5636]

That intro is NOT a movie💀😭

[@nischayranjan3728]

"core" file is not being generated in my linux. I went through all the same commands, as it is explained in the video

[@valekprometey]

Can I unlimit -c for certain processes, but not for all?

[@sunnybambooflute]

thanks for sharing the knowledge, btw we are using the same slack notify sound. lol

[@namespace242]

very helpful , thank you !

[@albertomedinarobredo]

thanks matthew

[@AnnatarTheMaia]

You didn't explain how to compile the program to contain all the debugging information so that the core file could be debugged in situ on a remote system where the source code to step through would not be available, for example at a client's site.

[@gauravsaini5549]

very Informative, Thanks for the video!

[@JacobSorber]

You're welcome!

[@surj8749]

Great video

[@XenoTravis]

If you set it to like half the size of the full core dump is will that possibly not include the info you want or maybe not let the debugger run properly? I don't understand what information it absolutely needs.

[@JacobSorber]

Good question. If the core dump is larger than the limit, it won't save any of it (rather than saving part of the memory). It's all or nothing, because it would be difficult to know what memory is most helpful.

[@XenoTravis]

@@JacobSorber so what is the use of adding a limit if it won't use the limit?

[@almosteasy9590]

@@XenoTravis maybe it means.. if the size is lesser than the limit then only make core dump, otherwise forget it.

[@unsafecast3636]

My linux system has core dumps to unlimited by default. I'm using Arch

[@bibloc5293]

Are you sitting on the floor or some small chair? I am looking for ergonomic sitting positions as debugging can take hours or days.

[@aniltadikamalla4236]

I have some problem in usage of GDB. can you please help me out here?

[@NeZversSounds]

Wait, that means the user finding bug will need to tinker with the machine (assuming can get admin privileges) then try to crash it again to actually get a core dump. Sounds like too many loops to jump to help the developer to fix the app. Then how those apps with crash reports can get it without that?

[@redcrafterlppa303]

It's only an assumption. But I guess programs with crash reports run a small runtime that catches segfaults and saves them before the operating system receives and trashes them.

[@sobad123]

great video! thank u

[@shivanaikrachanaikar2843]

i have core dump file how can i analyze it and which tool i need to download

[@MrDimonsky]

My problem under Ubuntu - no core dumped generated. I tried everything already..

[@Victor_Marius]

Are coredumps available in python?

[@aidenpearce5724]

is there any video about makefile that you uploaded ?

[@JacobSorber]

Yeah, I have a few. kzbin.info/www/bejne/l2nQgX54rbZshNE

[@aidenpearce5724]

@@JacobSorber thx jacob, hope you can make series about make more than 60 seconds 💪💪💪

[@hdqhdjxiwj]

sad little zero made my day

[@first-thoughtgiver-of-will2456]

what is this std bloatware you speak of I write my HAL in my applications

[@MECHANISMUS]

There's hope...

[@activelivingchallenger4298]

my school won't let me use ulimit -c to anything other than 0 on their machines. So I just used git and used it on my local machine. Hope that helps anyone.

[@exploringlife9323]

i didn't find core file after dump

[@mehmetdemir-lf2vm]

rand()%255 should have been rand()&255 or rand()%256 :)

[@sverkeren]

Not if the purpose was to generate random number from 0 to 254. :-)

[@dnbndu]

Feeling sad for the one who was flying 🥲

[@JacobSorber]

Yeah, seg faults are sometimes scary and sad. But, I have it on good authority that he bailed out in time.

[@dnbndu]

@@JacobSorber now I can sleep comfortably. 🙏

[@leokiller123able]

Poor Lego guy :(

[@JacobSorber]

I know. Tragic.

[@recepbashir7216]

nice

[@WilliamRaezer]

"Taint analysis" , "core dump".

[@kapilverma2668]

How to write our own memory leak detector?

[@JacobSorber]

One way is to use a shim to intercept malloc, calloc, realloc, and free calls, and then keep track of what blocks have been allocated and their sizes, and then check to see what blocks aren't freed at the end.

[@kapilverma2668]

@@JacobSorber Is it possible with MSVC on windows?

[@BetaChri5]

@@kapilverma2668 Yes

[@kapilverma2668]

@@BetaChri5 How

[@m4l490n]

lol, that was an overly dramatic intro. Oh boy, core files are certainly big. I have a small program that generates a 71MB core file.

[@DipietroGuido]

"Now, when this program eventually crashes, as we knew it would, it says that it 'Core dumped'. It means the computer dumped the content of the process' memory when it crashed. Except for the fact that it really didn't. I'm sorry to break it to you, but sometimes computers lie." - Jacob the Sorber, 1984, University of Wakanda

[@synchro-dentally1965]

3:00 The dump is a lie...

[@franciscomelojunior2535]

Starts at 2:00

[@PrashantMarshal]

A good introductory video, but a bad waste of 1 minute at the start. Cheers!