1，安装crash 

sudo apt install linux-crashdump
sudo apt-get install kexec-tools
sudo apt install crash

sudo apt-get install texinfo
sudo apt-get upgrade/update

查看linux系统版本， uname -a

lark@ubuntu:~$ uname  -a
Linux ubuntu 5.15.0-124-generic #134~20.04.1-Ubuntu SMP Tue Oct 1 15:27:33 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

x86-arm64(arm6-64版本），开发板使用arm64版本。 

下载vmtools:Index of /pool/main/l/linux (ubuntu.com),下载下面的uname -a 对应的版本。

http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-unsigned-5.15.0-124-generic-dbgsym_5.15.0-124.134_amd64.ddeb

sudo dkpg -i  linux-image-unsigned-5.15.0-124-generic-dbgsym_5.15.0-124.134_amd64.ddeb

方法二:

sudo apt install ubuntu-dbgsym-keyring 

sudo apt-get update

sudo apt -y install linux-image-$(uname -r)-dbgsym

The following additional packages will be installed:
  linux-image-unsigned-5.15.0-124-generic-dbgsym
The following NEW packages will be installed:
  linux-image-5.15.0-124-generic-dbgsym linux-image-unsigned-5.15.0-124-generic-dbgsym 

安装完成后查看boot目录：

lark@ubuntu:~/Desktop$ cd /usr/lib/debug/
boot/      .build-id/ lib/       usr/   

lark@ubuntu:~$ sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-124-generic  

 挂死.

lark@ubuntu:/var/crash/202411060815$ sudo crash -d /usr/bin/gdb /usr/lib/debug/boot/vmlinux-5.15.0-124-generic /var/crash/202411060815/dump.202411060815 
[sudo] password for lark: 

crash 7.2.8
Copyright (C) 2002-2020  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [768MB]: patching 146450 gdb minimal_symbol values

please wait... (patching 146450 gdb minimal_symbol values) Segmentation fault

 怀疑gdb 和crash版本不一致导致，下面根据crash跟新gdb.

sudo apt purge crash

git clone https://github.com/crash-utility/crash.git

cd crash

make -j8

sudo make install

crash --version 

更新后正常进入crash界面.

lark@ubuntu:~/crash$ sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-124-generic /var/crash/202411060815/dump.202411060815 

crash 8.0.5++
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
Type "apropos word" to search for commands related to "word"...

      KERNEL: /usr/lib/debug/boot/vmlinux-5.15.0-124-generic
    DUMPFILE: /var/crash/202411060815/dump.202411060815  [PARTIAL DUMP]
        CPUS: 2
        DATE: Wed Nov  6 08:14:55 PST 2024
      UPTIME: 01:49:53
LOAD AVERAGE: 0.05, 0.11, 0.31
       TASKS: 398
    NODENAME: ubuntu
     RELEASE: 5.15.0-124-generic
     VERSION: #134~20.04.1-Ubuntu SMP Tue Oct 1 15:27:33 UTC 2024
     MACHINE: x86_64  (2208 Mhz)
      MEMORY: 2 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 19284
     COMMAND: "bash"
        TASK: ffff8d1dcce5cc80  [THREAD_INFO: ffff8d1dcce5cc80]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash> 

查看挂死堆栈。 

crash> bt
PID: 19284    TASK: ffff8d1dcce5cc80  CPU: 0    COMMAND: "bash"
 #0 [ffffb558c4873c28] machine_kexec at ffffffffb1092400
 #1 [ffffb558c4873c88] __crash_kexec at ffffffffb11a1cbf
 #2 [ffffb558c4873d50] panic at ffffffffb1ce768d
 #3 [ffffb558c4873dd0] sysrq_handle_crash at ffffffffb180291a
 #4 [ffffb558c4873de0] __handle_sysrq.cold at ffffffffb1d3d69c
 #5 [ffffb558c4873e28] write_sysrq_trigger at ffffffffb1803408
 #6 [ffffb558c4873e40] proc_reg_write at ffffffffb1434407
 #7 [ffffb558c4873e60] vfs_write at ffffffffb1392366
 #8 [ffffb558c4873e98] ksys_write at ffffffffb13946e7
 #9 [ffffb558c4873ed8] __x64_sys_write at ffffffffb139478a
#10 [ffffb558c4873ee8] x64_sys_call at ffffffffb10050ec
#11 [ffffb558c4873ef8] do_syscall_64 at ffffffffb1d8b074
#12 [ffffb558c4873f10] x64_sys_call at ffffffffb1006acf
#13 [ffffb558c4873f20] clear_bhb_loop at ffffffffb1e018c5
#14 [ffffb558c4873f30] clear_bhb_loop at ffffffffb1e018c5
#15 [ffffb558c4873f50] entry_SYSCALL_64_after_hwframe at ffffffffb1e00124
    RIP: 00007f186ecea297  RSP: 00007fff9fcffa78  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f186ecea297
    RDX: 0000000000000002  RSI: 0000558fbec146c0  RDI: 0000000000000001
    RBP: 0000558fbec146c0   R8: 000000000000000a   R9: 0000000000000001
    R10: 0000558fbe95e017  R11: 0000000000000246  R12: 0000000000000002
    R13: 00007f186edc96a0  R14: 00007f186edc54a0  R15: 00007f186edc48a0
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b