1. 内核Panic现场还原的核心思路
内核Panic是Linux系统最严重的故障之一,它意味着内核检测到了无法恢复的错误状态。要有效还原Panic现场,我们需要掌握以下几个关键环节:
1.1 内核Oops与Panic的区别
Oops是内核遇到非致命错误时的警告机制,系统可能继续运行;而Panic则是内核主动触发的紧急停止。两者都会生成包含关键信息的日志,但Panic意味着系统立即终止。在实际生产环境中,Panic往往伴随着硬件异常、内存损坏或关键数据结构破坏等严重问题。
1.2 关键日志采集方法
当系统发生Panic时,第一时间应该保存以下信息:
code复制[ 321.456789] Kernel panic - not syncing: Fatal exception in interrupt
[ 321.456790] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.0-135-generic #152
[ 321.456791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 321.456792] Call Trace:
[ 321.456793] <IRQ>
[ 321.456794] dump_stack+0x6d/0x9a
[ 321.456795] panic+0x101/0x2e3
[ 321.456796] ? _raw_spin_unlock_irqrestore+0x16/0x40
[ 321.456797] oops_end+0x0/0x50
这些日志通常通过以下方式获取:
- 串口控制台输出(最可靠)
- /var/log/kern.log
- dmesg命令输出
- 崩溃转储工具(如kdump)
提示:在生产环境中,务必配置串口重定向或网络日志服务器,因为本地日志系统在Panic时可能不可用。
1.3 栈帧解析实战
以典型的内核Oops为例,我们来看如何解析调用栈:
code复制[ 123.456789] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[ 123.456790] IP: [<ffffffff81234567>] my_module_func+0x17/0x30 [faulty]
[ 123.456791] PGD 0
[ 123.456792] Oops: 0000 [#1] SMP
[ 123.456793] CPU: 0 PID: 1234 Comm: insmod Tainted: G OE 4.15.0-112-generic
[ 123.456794] RIP: 0010:[<ffffffff81234567>] [<ffffffff81234567>] my_module_func+0x17/0x30 [faulty]
[ 123.456795] RSP: 0018:ffff88003d5c3e08 EFLAGS: 00010246
[ 123.456796] RAX: 0000000000000000 RBX: ffff88003d7e2000 RCX: 0000000000000000
[ 123.456797] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff88003d7e2000
[ 123.456798] RBP: ffff88003d5c3e28 R08: 0000000000000000 R09: 0000000000000000
[ 123.456799] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa0005000
[ 123.456800] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 123.456801] FS: 00007f8b5f5f9700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 123.456802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 123.456803] CR2: 0000000000000010 CR3: 000000003d40a000 CR4: 00000000000006f0
[ 123.456804] Call Trace:
[ 123.456805] [<ffffffffa0005011>] faulty_write+0
