Debug kernel panics _ Embedded System Blog.pdfUnab

文件名称: Debug kernel panics _ Embedded System Blog.pdf

所属分类: Linux

开发工具:

文件大小: 307kb

下载次数: 0

上传时间: 2019-07-03

提供者: devwa******

下载 (307kb)

不能下载？报告错误

详细说明：Unable to handle kernel paging request at virtual address 内核奔溃问题debug方法2019/73 Debug kernel panics Embedded System Blog Here is the kernel backtrace where Kernel is crashing at"add_ range function. (Actually nearest function symbol to the crash). Let's analyze step by step 1. Crash occurs at below location as per Back Trace PC is at add range +0x14/0x6c 2. Grep/find add range in System map file and note down symbol name address ie. 80049f28 #T inux-Kernel grep add range System map 80049f28 T add range 3. Replace add range symbol name address in add range+0x"=80049f28 +0X14=80049F3C 4 80049F3C should be same as pc address in the back trace wow it is same so it means that Kernel version i am using and on which issue is reported is same(also depends on same. config settings). Lets move to next step 5. run objdump on vmlinux to get the disassembly and detail on objdump program and vmlinux can be fetched from below hyperlinks objdump: is a program for displaying various information about object files For instance, it can be used as a disassembler to view executable in assembly form vmlinux: is a statically linked executable file that contains the Linux kernel in one of the object file formats supported by linux, the vmlinux file might be required for kernel debugging, symbol table generation or other operations #objdump -D-S --show-raw-insn --prefix-addresses line-nurmbers vmlinux> okjdunp 6. Find"add range"in vmlinux objdump and look for PC address calculated above i.e. 80049F3C 80049F3C e5903004 ldr #4] https://web.archiveorg/web!20170723011824/https://sanjeevsharmaengg.wordpresscom/tag/debug-kernel-panics/ 3/11 2019/73 Debug kernel panics Embedded System Blog 7. The crash point can be identified below r3,r0,#4]=x0+4=02120c0+4=02120bc4 /replace ro with ro register value from the Back Trace 8. Wow! This is same as fault address Unable to handle kernel paging request at virtual address 02120bc4 Conclusion: Here rO is pointing to invalid address and from disassembly found out where ro is pointing and found out why ro is pointing to invalid address Using gDb to find the location where your kernel panicked or oopsed a quick and easy way to find the line of code where your kernel panicked or oopsed is to use GDB list command. You can do this as follows Let's assume your panic/oops message says something like 174.5070841 Stack 174.507163 ce0bd8ac0000000800000000ce4a7e90~039ce30ce0bd8ac 0718b04c07185a0 [174.507380 ce4a7eao c0398f22 ceobd8ac c0718b04 ce4aeb0 c037deee e obd8eo ce obd8ac [174.5075971 ce4aleco c037dfe0 c07185a0 ceObdBac ce4a7ed4 c037d353 ceobd8ac ceobd8ac 174.5078881Ca-1 Trace: 174.508125 [1? sd remove+0x20/0x70 https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 4/11 2019/73 Debug kernel panics Embedded System Blog 174.508235 [1? scsi bus remcve+0x32/0x40 174.508326 []? device release driver+0x3e/0x70 74.508421 [c037dfe0>]? device release driver+0x20/0x40 [174.508514 [1? bus remove device+0x73/0x90 174.508606 [l? device del+Oxef/0x150 174.5086931 1? scsi remove device+0x47/0x80 174.508786 [1? scsi remove device+0x22/0x40 [174.508877 [1? scsi remove target+0x94/0xd0 174.508969 [1? remove child+Cx0/0x20 174.509060 []? remove child+0x17/0x20 174.509148 []? device for each child+0x38/0x60 [174.50924- [l scsi remove target+0x2f/0x60 [174.5093931 [1? iscsi unbind session+0x77/0xa0 [scsi transport iscsi] [174.509699 []? run workqueue+0x6e/0x140 174.50980- [1 iscsi unbind session +0x0/0xa0 Iscsi transport. i scsi J 174.509977 [1? worker thread+0x88/0xe0 174.510047 [? autoremove wake function+0x0/0x40 Lets say you want to know what line of code represents sd remove+0X20/0X70 cd to your directory of your kernel tree and run gdb on the"o"file which has the function sd remove in this case in sd o, and use the gdb " list"command, (gdb) list (function +Oxoffset), in this case function is sd remove( and offset is 0x20, and gdb should tell you the line number where you hit the panic or oops. This worked reliably for most cases #t gdb sdo gdb)list *(sd remove+0x20) 0x1650 is in sd remove (Kernel/linux-xxx/drivers/scsi/sd. c: 2125) 2120 static int sd remove(struct device *dev) https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 5/11 2019/73 Debug kernel panics Embedded System Blog 2121 2122 struct scsi disk sdkpi chronize full( 2125 sdkp dev get drvdata(dev) 2126 olk queue prep rq(sdkp->device->request queue prep fn) device del(&sckp->dev)i 2128 del gendisk(sckp->disk)i d shutdown(dev) so dev get drvdatao)is the function where crash has een happened and lets analyze why d t. drvdata(d :ashi ng Disassembling the kernel Cross tools are needed The utility The main utility used to do it is the objdump one arim-none linux /*If We have object code handy then, we can disassemble the individual object file alsc like objdump -s panic. o gdb on vmlinux One can disassemble a built kernel using gdb on the vmlinux image. This is useful when one gets a kernel Oops message and a stack dump-one can then disassemble the object code and see where the Oops is occurring. For example https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 6/11 2019/73 Debug kernel panics Embedded System Blog ffarm-none-linux-gnueabi-gdb -silent vmlinux #disassemble printk Dump of assembler code for function printk: Oxffffffff8023dce0 : S0xd8, rsp Oxffffffff8023dce7 : lea 0xe0(rsp),rax Oxffffffff8023dcef : mov rsi, 0x28(rsp) Oxffffffff8023dcf4 : mov 各rsp,8rSi Oxffffffff8023dcf7 : mov rdx, 0x30(rsp) Oxffffffff8023dcfc : mov Rcx, 0x38(rsp) Oxffffffff8023ddo1 : mov rax, 0x8(rsp) Oxffffffff8023ddo6 : lea 0x20(rsp),rax Oxffffffff8023ddob : mov r8,0x40(8ySp) Oxffffffff8023dd10 : mov r9, 0x48(rsp) Oxffffffff8023dd1 5 : movl $0x8,(rsp) Oxffffffff8023ddlc : movl 0×30,0x4(IsP) Oxffffffff8023dd2 4 : mov 各rax,0x10(rsp) Oxffffffff8023dd29 : calla Oxffffffff8023d980 Oxffffffff8023dd?e : add $0xd8,rsp Oxffffffff8023dd35 : reta End of assembler dump How to Interpret Assembly Language( EABl C function call mapping to ARM registers) First of all we should disassemble the kernel function with either oBjdumP utility or using gdb on vmlinux kernel image by referring above mention section in Document. For example here is the dis-assembly of add range kernel function where i will demonstrate how this all works this will be different depending on how the compiler optimizes but it should give an idea #gdb disassemble add range Dump of assembler code for function add range 0x8004c4d8<+0> mov 0x8004c4dc<+4>: push r4,r5,r6,r7,r11 r12,1x,pc} 8004c4e0<+8> su上 11,r12,# 0x×8004C4e4<+12>: ldrd 6,[r11,#4] 0x8004c4e8<+16>: ldrd r4,[r11,#12] 0x8004c4ec<+20>: r7,r5 0x8004c4f0<+24>: camped r6,r4 0×8004c4f4<+28>: bcs ×5004≈510 https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 7/11 2019/73 Debug kernel panics Embedded System Blog add range+56 0x8004c4f8<+32> cimp 1 0x8004c4fc<+36>: 1s11 r3,Y2,#4 x8004c500<+40> addl- 0×8004c504<+44>: add I 0×8004C508 trait r6,[r0,r3] 0x8004c50c<+52>: strait 4,[r1,#8] 0x8004c510<+56>: mov r0,2 0x8004c514<+60>: 1dm sp,(y4,r5,x6,r7 cH End of assembLer dump Corresponding Kernel c function int add range(struct range * range, int az, int nr range, 164 start, 164 end) if (start > en return n- range / Out of slots: * i f (nr range > az) return n- range; range inr range, star starti range [nr range], end mr rang return nr range Lets analyse the first 3 lines which is more or less same/common across all functions and here r12=IP(Intra-Procedure-call scratch register), r11=FP(Frame pointer). The FP keeps track of the variables from function to function. It is a frame on the stack of the function. please explore basic frame layout for more detail. so in simple words SP is where the stack is and FP is where the stack was like Pc and LR register 0x8004c4d8<-0>: Inov r12, sp /*get a copy of 0x8004c4dc<+4> push {r4,r5,r6,r7,r11 r12, lr, pc] /*Save the frame, link register, program counter and other Register cn to the stack 0x8004c4e0<-8>: r11 r12, 4/*Set the new frame pointer x/ https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 8/11 2019/73 Debug kernel panics Embedded System Blog The next 2 instruction passing 4 bytes and 12 bytes from the Frame pointer nto the r6 and r4 registers for an function call and in other words values stores at r1 1+#4 would be stored in ro and values stores at r11+#12 would be stored in r4 Note: LDRD is used to store double word instruction however content also will be loaded into r7 and r5 register. This function call is dealing with 64 bit data so 64 bit data is manipulated in stack only 0x8004c4e4×+12>: ldrd r6,[r11,#4] 0×8004c4e8<+16> l drd 4,[r11,#12] Note: The first four registers rO-r3 are used to pass argument values into a subroutine and to return a result value from a function. so Ro=range, R1=az, R2=nr range, R3=start, R4=end The next instructions can easily mapped with c code Note: Underlying mapping somewhat different from the normal C to Assembly conversion mapping because here 64-bit value is being passed in Function call argument which is u64 start and u64 end and to deal with 64 bit data it has to be stored in register pair and can be retrived using ldrd instruction from stack using frame pointer 0×8004c4eC<-20> F r7 ′x5 /* first instruction compare r and r5 register i. e store 32 bit LSB for start end whose value is stored in stack 0x8004c40<+24>: cmpeg r6, r4 /*This next instruction performs an comparison only if the result of ab I7 r5 instruction found true(i.e 0x8004c4f4<-28>: bcs 0x8004c510 < add range+56> 0x8004c4f8+32> cmp r2, r1 /*This instruction compare values stored in resisters r? and rl which are passed argument values i.e nr range and responding C code is f (start > end) return n- range; /○ ut of slots:x/ if (nr range a2) return n range; https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 9/11 2019/73 Debug kernel panics Embedded System Blog Lets move to next instructions 0x8004c4f # 0×8004c500<-40> addl t ,r2,#1 0×8004c504<-44 ddl 1,x0,r3 0x8004c508<-48>: stxd1七 r6,[r0,r3 0x8004c50c<+52>: strdlt r4 [rl, #81 Corresponding c code ange[nr range]. start starc; range inr range] end end; mr range 0x8004c510<-56>: noV ro r2/*move r2 content into ro register which can be return back and As I said ro-R3 are also used tc hold return value from function 0x8004c514<-60>: l dm p,{r4,r5,r6,x7,x11 sD, pc)/*LDM is used to lcad multiple instructions and similar to pop stack instruction Corresponding ode is return nr range; Here are the ARM Register definition for your reference and please keep in mind these register when you are mapping c function to aRm registers and more detailed information can be found here Note In addition to above technique's, don't forget to visit Tour of ARM Assembly(http:www.coranac.com/tonc/text/asm.htm)whichwillhelpyou to understand following's in deeper detail and after going through it, I Bet you, you should be able to produce some nice ARM assembly, or at least be able to read it well enough General assembly The arM instruction set References 1. Procedure Call Standard for the ArM Architecture https://web.archiveorg/web/20170723011824/https://sanjeevsharmaengg.wordpress.com/tag/debug-kernel-panics/ 10/11