Analyzing Linux kernel crash dumps - The one tutorial that has it all

Hi all,

Welcome to the longest, most thorough tutorial on Linux kernel crash analysis ever. This is the sixth and final article in the long series. We learned about LKCD and Kdump, we had specific guides for CentOS and openSUSE and we had a detailed crash usage guide.

Today, we escalate this further. How much? Well, a lot. This tutorial will teach you how to use the crash utility to analyze Linux kernel crash memory cores, including detailed analysis of crash reports, using cscope to search for functions in C sources, recompiling and making objects with symbols, using objdump to disassemble objects, submission of crashes to developers, numerous examples, and more. We will ever write a faulty kernel module and then dissect it.

I promise you, you have never seen or read something like this. Digital Nirvana, Dedoimedo style.

http://www.dedoimedo.com/computers/crash-analyze.html


Table of Contents

1. Necessary reading
2. Analyzing the crash report - First steps
3. Getting warmer
3.1. Fedora example
3.2. Another example, from the White Paper
3.3. Kernel Page Error
3.4. Status check
4. Getting hot
4.1. Backtrace
4.2. Call trace
4.3. Instruction pointer
4.4. Code Segment (CS) register
4.5. Fedora example, again
4.6. backtrace for all tasks
4.7. Dump system message buffer
4.8. Display process status information
5. Super geeky stuff
5.1. Source code
5.2. cscope
5.3. Disassemble the object
5.3.1. Trivial example
5.3.2. objdump
5.3.3. Moving on to kernel sources
5.3.4. Intermediate example
5.3.5. Difficult example
5.4. Alternative solution (debug kernel)
6. Next steps
6.1. Kernel crash bug reporting
6.2. Google for information
6.3. Crash analysis results
6.3.1. Single crash
6.3.2. Hardware inspection
6.3.3. Reinstallation & software changes
6.3.4. Submit to developer/vendor
7. Official documentation
8. References
9. Conclusion


Cheers,
Mrk