前言

最近在适配 ubuntu系统，记录一下其crash的安装。

一、apt 安装dbgsym vnlinux

# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiversedeb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiversedeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list

$ sudo apt install ubuntu-dbgsym-keyring
$ sudo apt-get update

# apt -y install linux-image-$(uname -r)-dbgsym
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:linux-image-unsigned-5.15.0-122-generic-dbgsym
The following NEW packages will be installed:linux-image-5.15.0-122-generic-dbgsym linux-image-unsigned-5.15.0-122-generic-dbgsym
0 upgraded, 2 newly installed, 0 to remove and 42 not upgraded.
Need to get 1,061 MB of archives.
After this operation, 6,917 MB of additional disk space will be used.
Get:1 http://ddebs.ubuntu.com jammy-updates/main amd64 linux-image-unsigned-5.15.0-122-generic-dbgsym amd64 5.15.0-122.132 [1,061 MB]

二、使用.ddeb包安装dbgsym vnlinux

也可以直接在 http://ddebs.ubuntu.com/pool/main/l/linux/下载对应的.ddeb包：

dpkg -i linux-image-unsigned-5.15.0-122-generic-dbgsym_5.15.0-122.132_amd64.ddeb

三、dbgsym发行版

在http://ddebs.ubuntu.com/pool/main/l/linux/查看支持的dbgsym vmlinux：

发行版	内核版本
14.04	3.13.0
16.04	4.4.0
18.04	4.15.0
20.04	5.4.0
22.04	5.15.0
24.04	6.8.0

其他内核版本，请参考：http://ddebs.ubuntu.com/pool/main/l/

四、crash调试

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy# uname -r
6.5.0-18-generic

我这里用ubuntu22.04 的6.5.0-18-generic内核调试为例：

/** 64-bit SYSCALL instruction entry. Up to 6 arguments in registers.** This is the only entry point used for 64-bit system calls.  The* hardware interface is reasonably well designed and the register to* argument mapping Linux uses fits well with the registers that are* available when SYSCALL is used.** SYSCALL instructions can be found inlined in libc implementations as* well as some other programs and libraries.  There are also a handful* of SYSCALL instructions in the vDSO used, for example, as a* clock_gettimeofday fallback.** 64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11,* then loads new ss, cs, and rip from previously programmed MSRs.* rflags gets masked by a value from another MSR (so CLD and CLAC* are not needed). SYSCALL does not save anything on the stack* and does not change rsp.** Registers on entry:* rax  system call number* rcx  return address* r11  saved rflags (note: r11 is callee-clobbered register in C ABI)* rdi  arg0* rsi  arg1* rdx  arg2* r10  arg3 (needs to be moved to rcx to conform to C ABI)* r8   arg4* r9   arg5* (note: r12-r15, rbp, rbx are callee-preserved in C ABI)** Only called from user space.** When user can change pt_regs->foo always force IRET. That is because* it deals with uncanonical addresses better. SYSRET has trouble* with them due to bugs in both AMD and Intel CPUs.*/SYM_CODE_START(entry_SYSCALL_64)UNWIND_HINT_ENTRYENDBRswapgs/* tss.sp2 is scratch space. */movq	%rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)SWITCH_TO_KERNEL_CR3 scratch_reg=%rspmovq	PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rspSYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)ANNOTATE_NOENDBR/* Construct struct pt_regs on stack */pushq	$__USER_DS				/* pt_regs->ss */pushq	PER_CPU_VAR(cpu_tss_rw + TSS_sp2)	/* pt_regs->sp */pushq	%r11					/* pt_regs->flags */pushq	$__USER_CS				/* pt_regs->cs */pushq	%rcx					/* pt_regs->ip */
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)......

SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)pushq	%rax					/* pt_regs->orig_ax */PUSH_AND_CLEAR_REGS rax=$-ENOSYS/* IRQs are off. */movq	%rsp, %rdi/* Sign extend the lower 32bit as syscall numbers are treated as int */movslq	%eax, %rsi/* clobbers %rax, make sure it is after saving the syscall nr */IBRS_ENTERUNTRAIN_RETcall	do_syscall_64		/* returns with IRQs disabled */......

看其vmlinux：

 cat /boot/System.map-6.5.0-18-generic | grep entry_SYSCALL_64_after_hwframe
ffffffff82200078 T entry_SYSCALL_64_after_hwframe

ffffffff82200078:       50                      push   %rax
ffffffff82200079:       57                      push   %rdi
ffffffff8220007a:       56                      push   %rsi
ffffffff8220007b:       52                      push   %rdx
ffffffff8220007c:       51                      push   %rcx
ffffffff8220007d:       6a da                   push   $0xffffffffffffffda
ffffffff8220007f:       41 50                   push   %r8
ffffffff82200081:       41 51                   push   %r9
ffffffff82200083:       41 52                   push   %r10
ffffffff82200085:       41 53                   push   %r11
ffffffff82200087:       53                      push   %rbx
ffffffff82200088:       55                      push   %rbp
ffffffff82200089:       41 54                   push   %r12
ffffffff8220008b:       41 55                   push   %r13
ffffffff8220008d:       41 56                   push   %r14
ffffffff8220008f:       41 57                   push   %r15
ffffffff82200091:       31 f6                   xor    %esi,%esi
ffffffff82200093:       31 d2                   xor    %edx,%edx
......
ffffffff822000d1:       90                      nop
ffffffff822000d2:       90                      nop
ffffffff822000d3:       90                      nop
ffffffff822000d4:       90                      nop
ffffffff822000d5:       90                      nop
ffffffff822000d6:       90                      nop
ffffffff822000d7:       90                      nop
ffffffff822000d8:       90                      nop
ffffffff822000d9:       90                      nop
ffffffff822000da:       90                      nop
ffffffff822000db:       90                      nop
ffffffff822000dc:       90                      nop
ffffffff822000dd:       90                      nop
ffffffff822000de:       90                      nop
ffffffff822000df:       90                      nop
ffffffff822000e0:       90                      nop
ffffffff822000e1:       e8 9a 4a f1 ff          call   0xffffffff82114b80
ffffffff822000e6:       90                      nop
ffffffff822000e7:       90                      nop
ffffffff822000e8:       90                      nop
ffffffff822000e9:       90                      nop
ffffffff822000ea:       90                      nop

到了ffffffff822000d1处，全是 nop指令：

# cat /boot/System.map-6.5.0-18-generic | grep '\<do_syscall_64\>'
ffffffff82114b80 T do_syscall_64

而我看crash：

crash> dis entry_SYSCALL_64_after_hwframe
0xffffffff97400078 <entry_SYSCALL_64_after_hwframe>:    push   %rax
0xffffffff97400079 <entry_SYSCALL_64_after_hwframe+1>:  push   %rdi
0xffffffff9740007a <entry_SYSCALL_64_after_hwframe+2>:  push   %rsi
0xffffffff9740007b <entry_SYSCALL_64_after_hwframe+3>:  push   %rdx
0xffffffff9740007c <entry_SYSCALL_64_after_hwframe+4>:  push   %rcx
0xffffffff9740007d <entry_SYSCALL_64_after_hwframe+5>:  push   $0xffffffffffffffda
0xffffffff9740007f <entry_SYSCALL_64_after_hwframe+7>:  push   %r8
0xffffffff97400081 <entry_SYSCALL_64_after_hwframe+9>:  push   %r9
0xffffffff97400083 <entry_SYSCALL_64_after_hwframe+11>: push   %r10
0xffffffff97400085 <entry_SYSCALL_64_after_hwframe+13>: push   %r11
0xffffffff97400087 <entry_SYSCALL_64_after_hwframe+15>: push   %rbx
0xffffffff97400088 <entry_SYSCALL_64_after_hwframe+16>: push   %rbp
0xffffffff97400089 <entry_SYSCALL_64_after_hwframe+17>: push   %r12
0xffffffff9740008b <entry_SYSCALL_64_after_hwframe+19>: push   %r13
0xffffffff9740008d <entry_SYSCALL_64_after_hwframe+21>: push   %r14
0xffffffff9740008f <entry_SYSCALL_64_after_hwframe+23>: push   %r15
0xffffffff97400091 <entry_SYSCALL_64_after_hwframe+25>: xor    %esi,%esi
0xffffffff97400093 <entry_SYSCALL_64_after_hwframe+27>: xor    %edx,%edx
0xffffffff97400095 <entry_SYSCALL_64_after_hwframe+29>: xor    %ecx,%ecx
0xffffffff97400097 <entry_SYSCALL_64_after_hwframe+31>: xor    %r8d,%r8d
0xffffffff9740009a <entry_SYSCALL_64_after_hwframe+34>: xor    %r9d,%r9d
0xffffffff9740009d <entry_SYSCALL_64_after_hwframe+37>: xor    %r10d,%r10d
0xffffffff974000a0 <entry_SYSCALL_64_after_hwframe+40>: xor    %r11d,%r11d
0xffffffff974000a3 <entry_SYSCALL_64_after_hwframe+43>: xor    %ebx,%ebx
0xffffffff974000a5 <entry_SYSCALL_64_after_hwframe+45>: xor    %ebp,%ebp
0xffffffff974000a7 <entry_SYSCALL_64_after_hwframe+47>: xor    %r12d,%r12d
0xffffffff974000aa <entry_SYSCALL_64_after_hwframe+50>: xor    %r13d,%r13d
0xffffffff974000ad <entry_SYSCALL_64_after_hwframe+53>: xor    %r14d,%r14d
0xffffffff974000b0 <entry_SYSCALL_64_after_hwframe+56>: xor    %r15d,%r15d
0xffffffff974000b3 <entry_SYSCALL_64_after_hwframe+59>: mov    %rsp,%rdi
0xffffffff974000b6 <entry_SYSCALL_64_after_hwframe+62>: movslq %eax,%rsi
0xffffffff974000b9 <entry_SYSCALL_64_after_hwframe+65>: xchg   %ax,%ax
0xffffffff974000bb <entry_SYSCALL_64_after_hwframe+67>: mov    $0x48,%ecx
0xffffffff974000c0 <entry_SYSCALL_64_after_hwframe+72>: mov    %gs:0x1fbd0,%rdx
0xffffffff974000c9 <entry_SYSCALL_64_after_hwframe+81>: mov    %edx,%eax
0xffffffff974000cb <entry_SYSCALL_64_after_hwframe+83>: shr    $0x20,%rdx
0xffffffff974000cf <entry_SYSCALL_64_after_hwframe+87>: wrmsr
0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp    0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
0xffffffff974000d3 <entry_SYSCALL_64_after_hwframe+91>: int3
0xffffffff974000d4 <entry_SYSCALL_64_after_hwframe+92>: int3
0xffffffff974000d5 <entry_SYSCALL_64_after_hwframe+93>: int3
0xffffffff974000d6 <entry_SYSCALL_64_after_hwframe+94>: int3
0xffffffff974000d7 <entry_SYSCALL_64_after_hwframe+95>: int3
0xffffffff974000d8 <entry_SYSCALL_64_after_hwframe+96>: int3
0xffffffff974000d9 <entry_SYSCALL_64_after_hwframe+97>: int3
0xffffffff974000da <entry_SYSCALL_64_after_hwframe+98>: int3
0xffffffff974000db <entry_SYSCALL_64_after_hwframe+99>: int3
0xffffffff974000dc <entry_SYSCALL_64_after_hwframe+100>:        int3
0xffffffff974000dd <entry_SYSCALL_64_after_hwframe+101>:        int3
0xffffffff974000de <entry_SYSCALL_64_after_hwframe+102>:        int3
0xffffffff974000df <entry_SYSCALL_64_after_hwframe+103>:        int3
0xffffffff974000e0 <entry_SYSCALL_64_after_hwframe+104>:        int3
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>:        call   0xffffffff97314b80 <do_syscall_64>

在0xffffffff974000d1处nop指令变成了jmp：

0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp    0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
......
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>:        call   0xffffffff97314b80 <do_syscall_64>

与UNTRAIN_RET指令有关。

（2）

/** Save all registers in pt_regs. Return GSBASE related information* in EBX depending on the availability of the FSGSBASE instructions:** FSGSBASE	R/EBX*     N        0 -> SWAPGS on exit*              1 -> no SWAPGS on exit**     Y        GSBASE value at entry, must be restored in paranoid_exit** R14 - old CR3* R15 - old SPEC_CTRL*/
SYM_CODE_START(paranoid_entry)ANNOTATE_NOENDBRUNWIND_HINT_FUNCPUSH_AND_CLEAR_REGS save_ret=1ENCODE_FRAME_POINTER 8/** Always stash CR3 in %r14.  This value will be restored,* verbatim, at exit.  Needed if paranoid_entry interrupted* another entry that already switched to the user CR3 value* but has not yet returned to userspace.** This is also why CS (stashed in the "iret frame" by the* hardware at entry) can not be used: this may be a return* to kernel code, but with a user CR3 value.** Switching CR3 does not depend on kernel GSBASE so it can* be done before switching to the kernel GSBASE. This is* required for FSGSBASE because the kernel GSBASE has to* be retrieved from a kernel internal table.*/SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14/** Handling GSBASE depends on the availability of FSGSBASE.** Without FSGSBASE the kernel enforces that negative GSBASE* values indicate kernel GSBASE. With FSGSBASE no assumptions* can be made about the GSBASE value when entering from user* space.*/ALTERNATIVE "jmp .Lparanoid_entry_checkgs", "", X86_FEATURE_FSGSBASE/** Read the current GSBASE and store it in %rbx unconditionally,* retrieve and set the current CPUs kernel GSBASE. The stored value* has to be restored in paranoid_exit unconditionally.** The unconditional write to GS base below ensures that no subsequent* loads based on a mispredicted GS base can happen, therefore no LFENCE* is needed here.*/SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbxjmp .Lparanoid_gsbase_done.Lparanoid_entry_checkgs:/* EBX = 1 -> kernel GSBASE active, no restore required */movl	$1, %ebx/** The kernel-enforced convention is a negative GSBASE indicates* a kernel value. No SWAPGS needed on entry and exit.*/movl	$MSR_GS_BASE, %ecxrdmsrtestl	%edx, %edxjs	.Lparanoid_kernel_gsbase/* EBX = 0 -> SWAPGS required on exit */xorl	%ebx, %ebxswapgs
.Lparanoid_kernel_gsbase:FENCE_SWAPGS_KERNEL_ENTRY
.Lparanoid_gsbase_done:/** Once we have CR3 and %GS setup save and set SPEC_CTRL. Just like* CR3 above, keep the old value in a callee saved register.*/IBRS_ENTER save_reg=%r15UNTRAIN_RET_FROM_CALLRET
SYM_CODE_END(paranoid_entry)

查看vmlinux：

# cat /boot/System.map-6.5.0-18-generic | grep '\<paranoid_entry\>'
ffffffff822013b0 T paranoid_entry

ffffffff822013b0:       56                      push   %rsi
ffffffff822013b1:       48 8b 74 24 08          mov    0x8(%rsp),%rsi
ffffffff822013b6:       48 89 7c 24 08          mov    %rdi,0x8(%rsp)
ffffffff822013bb:       52                      push   %rdx
ffffffff822013bc:       51                      push   %rcx
ffffffff822013bd:       50                      push   %rax
ffffffff822013be:       41 50                   push   %r8
ffffffff822013c0:       41 51                   push   %r9
ffffffff822013c2:       41 52                   push   %r10
ffffffff822013c4:       41 53                   push   %r11
ffffffff822013c6:       53                      push   %rbx
ffffffff822013c7:       55                      push   %rbp
ffffffff822013c8:       41 54                   push   %r12
ffffffff822013ca:       41 55                   push   %r13
ffffffff822013cc:       41 56                   push   %r14
ffffffff822013ce:       41 57                   push   %r15
ffffffff822013d0:       56                      push   %rsi
ffffffff822013d1:       31 f6                   xor    %esi,%esi
ffffffff822013d3:       31 d2                   xor    %edx,%edx
......
ffffffff82201484:       90                      nop
ffffffff82201485:       90                      nop
ffffffff82201486:       90                      nop
ffffffff82201487:       90                      nop
ffffffff82201488:       90                      nop
ffffffff82201489:       90                      nop
ffffffff8220148a:       90                      nop
ffffffff8220148b:       90                      nop
ffffffff8220148c:       90                      nop
ffffffff8220148d:       90                      nop
ffffffff8220148e:       90                      nop
ffffffff8220148f:       90                      nop
ffffffff82201490:       90                      nop
ffffffff82201491:       90                      nop
ffffffff82201492:       90                      nop

到了ffffffff82201484处都是nop指令。

crash查看：

crash> dis paranoid_entry
0xffffffff974013b0 <paranoid_entry>:    push   %rsi
0xffffffff974013b1 <paranoid_entry+1>:  mov    0x8(%rsp),%rsi
0xffffffff974013b6 <paranoid_entry+6>:  mov    %rdi,0x8(%rsp)
0xffffffff974013bb <paranoid_entry+11>: push   %rdx
0xffffffff974013bc <paranoid_entry+12>: push   %rcx
0xffffffff974013bd <paranoid_entry+13>: push   %rax
0xffffffff974013be <paranoid_entry+14>: push   %r8
0xffffffff974013c0 <paranoid_entry+16>: push   %r9
0xffffffff974013c2 <paranoid_entry+18>: push   %r10
0xffffffff974013c4 <paranoid_entry+20>: push   %r11
0xffffffff974013c6 <paranoid_entry+22>: push   %rbx
0xffffffff974013c7 <paranoid_entry+23>: push   %rbp
0xffffffff974013c8 <paranoid_entry+24>: push   %r12
0xffffffff974013ca <paranoid_entry+26>: push   %r13
0xffffffff974013cc <paranoid_entry+28>: push   %r14
0xffffffff974013ce <paranoid_entry+30>: push   %r15
0xffffffff974013d0 <paranoid_entry+32>: push   %rsi
0xffffffff974013d1 <paranoid_entry+33>: xor    %esi,%esi
0xffffffff974013d3 <paranoid_entry+35>: xor    %edx,%edx
......
0xffffffff97401484 <paranoid_entry+212>:        jmp    0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401486 <paranoid_entry+214>:        int3
0xffffffff97401487 <paranoid_entry+215>:        int3
0xffffffff97401488 <paranoid_entry+216>:        int3
0xffffffff97401489 <paranoid_entry+217>:        int3
0xffffffff9740148a <paranoid_entry+218>:        int3
0xffffffff9740148b <paranoid_entry+219>:        int3
0xffffffff9740148c <paranoid_entry+220>:        int3
0xffffffff9740148d <paranoid_entry+221>:        int3
0xffffffff9740148e <paranoid_entry+222>:        int3
0xffffffff9740148f <paranoid_entry+223>:        int3
0xffffffff97401490 <paranoid_entry+224>:        int3
0xffffffff97401491 <paranoid_entry+225>:        int3
0xffffffff97401492 <paranoid_entry+226>:        int3
0xffffffff97401493 <paranoid_entry+227>:        ret
0xffffffff97401494 <paranoid_entry+228>:        int3
0xffffffff97401495 <paranoid_entry+229>:        int3
0xffffffff97401496 <paranoid_entry+230>:        int3
0xffffffff97401497 <paranoid_entry+231>:        int3

到了0xffffffff97401484：

0xffffffff97401484 <paranoid_entry+212>:        jmp    0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401493 <paranoid_entry+227>:        ret

也是nop指令变成了jmp指令。
这与UNTRAIN_RET_FROM_CALL指令有关。

参考资料

https://ebpf.top/post/ubuntu_kdump_crash/