Analysis of a kernel crash

Here is the kernel crash looks like :-

<1>[   49.034499] Unable to handle kernel NULL pointer dereference at virtual address 00000004
<1>[   49.034512] pgd = ed0a4000
<1>[   49.034519] [00000004] *pgd=00000000
<0>[   49.034533] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
<4>[   49.034540] Modules linked in: cmemk bcmdhd snd_usb_audio snd_usbmidi_lib snd_rawmidi snd_hwdep atmel_mxt_ts input_polldev usb_f_fs usb_f_ncm dabridge u_ether usb_storage xhci_plat_hcd xhci_hcd dwc3 libcomposite udc_core configfs dwc3_omap extcon_harman phy_ti_pipe3 phy_omap_usb2 phy_omap_control traceBuf faceplate tun cfg80211 nxp7120i2c loop pvrsrvkm(O)
<4>[   49.034641] CPU: 0 PID: 104 Comm: surfaceflinger Tainted: G        W  O    4.4.45+ #2
<4>[   49.034649] Hardware name: Generic DRA74X (Flattened Device Tree)
<4>[   49.034657] task: eda95b00 ti: edb40000 task.ti: edb40000
<4>[   49.034671] PC is at drm_atomic_helper_check_modeset+0x10c/0x9bc
<4>[   49.034681] LR is at drm_mode_equal+0xb8/0xd4
<4>[   49.034688] pc : [<c045c2f0>]    lr : [<c0479e6c>]    psr: 600f0013
<4>[   49.034688] sp : edb41cb0  ip : edb41c88  fp : edb41cfc
<4>[   49.034695] r10: 00000000  r9 : d160e520  r8 : c0bb528c
<4>[   49.034702] r7 : c0bb528c  r6 : 00000000  r5 : d1048800  r4 : d53dbb80
<4>[   49.034707] r3 : d160e300  r2 : 00000000  r1 : c0bb528c  r0 : 00000001
<4>[   49.034714] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
<4>[   49.034721] Control: 10c5387d  Table: ad0a406a  DAC: 00000051
<4>[   49.034727]
<4>[   49.034727] PC: 0xc045c270:
<4>[   49.034733] c270  e1520003 0a000005 e5983000 e3130010 1a0001bf e5d53006 e383300a e5c53006
<4>[   49.034765] c290  e5943000 e5932208 e2866001 e1520006 caffffdd e5941018 e3510000 c305728c
<4>[   49.034795] c2b0  c3a06000 c34c70bb da000051 e594301c e7939106 e3590000 0a000049 e5972000
<4>[   49.034824] c2d0  e305128c e5943020 e34c10bb e3120010 e50b1030 e793a106 1a0001dd e59922bc
<4>[   49.034854] c2f0  e59a3004 e5920004 e1500003 0a000012 e3500000 0a000006 eb00557e e5943014
<4>[   49.034883] c310  e7933100 e5d32006 e3822008 e5c32006 e59a3004 e3530000 0a00011f e1a00003
<4>[   49.034912] c330  eb005574 e5943014 e7933100 e5d32006 e3822008 e5c32006 e59a3004 e3530000
<4>[   49.034941] c350  0a000115 e59921d8 e592300c e3530000 0a000127 e1a00009 e1a0100a e12fff33
<4>[   49.034971]
<4>[   49.034971] LR: 0xc0479dec:
<4>[   49.034977] 9dec  189da9f8 e595103c e3510000 0a000010 e594703c e3570000 0a000019 e3a00cca
<4>[   49.035007] 9e0c  e3430b9a ebfd469d e1a01007 e1a08000 e3a00cca e3430b9a ebfd4698 e1580000
<4>[   49.035036] 9e2c  0a000006 e1a00006 e89da9f8 e3a00001 e89da9f8 e594303c e3530000 1a00000a
<4>[   49.035066] 9e4c  e5942068 e5953068 e0223003 e313091f 1a000007 e1a00005 e1a01004 ebfffb4f
<4>[   49.035095] 9e6c  e89da9f8 e1a00007 e89da9f8 e1a00001 e89da9f8 e3a00000 e89da9f8 e1a0c00d
<4>[   49.035123] 9e8c  e92ddff0 e24cb004 e24dd00c e52de004 e8bd4000 e5903000 e1a07000 e50b1030
<4>[   49.035152] 9eac  e593312c e3530001 0a00003c e1a09007 e5b9403c e1540009 e5946000 e1a08004
<4>[   49.035182] 9ecc  0a000034 e597a030 e2875030 e155000a 1a000003 ea000023 e59aa000 e155000a
<4>[   49.035211]
<4>[   49.035211] SP: 0xedb41c30:
<4>[   49.035217] 1c30  00000001 c0244278 c045c2f0 c045c2f0 600f0013 ffffffff edb41c94 c0bb528c
<4>[   49.035249] 1c50  edb41cfc edb41c60 c010d324 c01011f4 00000001 c0bb528c 00000000 d160e300
<4>[   49.035278] 1c70  d53dbb80 d1048800 00000000 c0bb528c c0bb528c d160e520 00000000 edb41cfc
<4>[   49.035307] 1c90  edb41c88 edb41cb0 c0479e6c c045c2f0 600f0013 ffffffff 00000051 c0479dc0
<4>[   49.035336] 1cb0  024000c0 c049e130 edb41cdc edb41cc8 c0160d28 c01eabe8 ee801e80 c0bb528c
<4>[   49.035365] 1cd0  edb41d1c d53dbb80 ee1dec00 d53dbb80 00000001 c0bb528c d53db300 d53db380
<4>[   49.035394] 1cf0  edb41d14 edb41d00 c045cbc0 c045c1f0 c045cba0 00000001 edb41d64 edb41d18
<4>[   49.035423] 1d10  c04832f0 c045cbac edb41d3c edb41d28 c0160d28 c01eabe8 ee801f00 00076f08
<4>[   49.035452]
<4>[   49.035452] IP: 0xedb41c08:
<4>[   49.035457] 1c08  eef53610 edb40000 00000001 ee801d80 edb41c3c edb41c28 c0160d28 c01eabe8
<4>[   49.035488] 1c28  ef16df40 c02441dc 00000001 c0244278 c045c2f0 c045c2f0 600f0013 ffffffff
<4>[   49.035527] 1c48  edb41c94 c0bb528c edb41cfc edb41c60 c010d324 c01011f4 00000001 c0bb528c
<4>[   49.035568] 1c68  00000000 d160e300 d53dbb80 d1048800 00000000 c0bb528c c0bb528c d160e520
<4>[   49.035615] 1c88  00000000 edb41cfc edb41c88 edb41cb0 c0479e6c c045c2f0 600f0013 ffffffff
<4>[   49.035653] 1ca8  00000051 c0479dc0 024000c0 c049e130 edb41cdc edb41cc8 c0160d28 c01eabe8
<4>[   49.035695] 1cc8  ee801e80 c0bb528c edb41d1c d53dbb80 ee1dec00 d53dbb80 00000001 c0bb528c
<4>[   49.035729] 1ce8  d53db300 d53db380 edb41d14 edb41d00 c045cbc0 c045c1f0 c045cba0 00000001
<4>[   49.035758]
<4>[   49.035758] FP: 0xedb41c7c:
<4>[   49.035764] 1c7c  c0bb528c c0bb528c d160e520 00000000 edb41cfc edb41c88 edb41cb0 c0479e6c
<4>[   49.035792] 1c9c  c045c2f0 600f0013 ffffffff 00000051 c0479dc0 024000c0 c049e130 edb41cdc
<4>[   49.035822] 1cbc  edb41cc8 c0160d28 c01eabe8 ee801e80 c0bb528c edb41d1c d53dbb80 ee1dec00
<4>[   49.035853] 1cdc  d53dbb80 00000001 c0bb528c d53db300 d53db380 edb41d14 edb41d00 c045cbc0
<4>[   49.035882] 1cfc  c045c1f0 c045cba0 00000001 edb41d64 edb41d18 c04832f0 c045cbac edb41d3c
<4>[   49.035911] 1d1c  edb41d28 c0160d28 c01eabe8 ee801f00 00076f08 edb41d7c ee1dec00 c0244aa0
<4>[   49.035939] 1d3c  d53dbb80 ee1dec00 d1048800 00000001 ee1ded00 edae8000 d53dbb80 edb41d7c
<4>[   49.035969] 1d5c  edb41d68 c048376c c0483158 b6679000 00000000 edb41e2c edb41d80 c04848b4
<4>[   49.035999]
<4>[   49.035999] R1: 0xc0bb520c:
<4>[   49.036004] 520c  00000001 00000000 c0bb5214 c0bb5214 00000000 00000000 00000000 00000000
<4>[   49.036033] 522c  00000000 00000000 00000000 00000000 00000000 00000000 eea751c0 00000000
<4>[   49.036061] 524c  00040004 ee8b3480 00000000 00000000 00000000 00000000 00380038 ee13a5b0
<4>[   49.036090] 526c  ee13a5b0 00000001 00000000 00000000 00000000 00000000 ee240a90 00000001
<4>[   49.036118] 528c  00000000 ee6b9330 ee18d8c0 00000000 00000000 00000000 00000000 00000000
<4>[   49.036147] 52ac  00000000 00000000 00000000 00000001 00000000 c0bb52c0 c0bb52c0 00000000
<4>[   49.036177] 52cc  00000000 00000000 00000000 00000001 00000000 c0bb52e0 c0bb52e0 00000000
<4>[   49.036205] 52ec  00000000 00000000 00000000 00000001 00000000 c0bb5300 c0bb5300 00000000
<4>[   49.036234]
<4>[   49.036234] R3: 0xd160e280:
<4>[   49.036240] e280  00000000 00000000 00000000 00000000 00000018 00000028 00000000 00000000
<4>[   49.036269] e2a0  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036297] e2c0  00000000 f9375000 00004000 00000002 d160aec0 00000003 00000000 c0434614
<4>[   49.036326] e2e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036355] e300  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036383] e320  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036412] e340  d160ef40 d160e344 d160e344 00000000 00000000 d16cf180 00000014 00000000
<4>[   49.036441] e360  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036470]
<4>[   49.036470] R4: 0xd53dbb00:
<4>[   49.036475] bb00  00000001 edaaf700 5f434344 4944454d 41465f41 45545f56 552d5453 5f314253
<4>[   49.036504] bb20  49545241 5f535453 2e564146 00666366 00000000 00000000 00000000 00000000
<4>[   49.036534] bb40  00000001 00000000 5f703267 75646f6d 765f656c 445f706d 4d5f4343 41494445
<4>[   49.036563] bb60  3037395f 6c615f31 2e6d7562 0063786c 00000000 00000000 00000000 00000000
<4>[   49.036593] bb80  ee1dec00 00000000 cd7d7000 d1336100 d53db380 d53db300 00000001 d160e500
<4>[   49.036625] bba0  d160e300 edb41de4 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036659] bbc0  00000001 00000000 5f434344 4944454d 41465f41 45545f56 552d5453 5f314253
<4>[   49.036693] bbe0  55424c41 465f534d 662e5641 00006663 00000000 00000000 00000000 00000000
<4>[   49.036725]
<4>[   49.036725] R5: 0xd1048780:
<4>[   49.036730] 8780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036759] 87a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036787] 87c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036816] 87e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.036848] 8800  ee1dfc00 00000101 00000003 00000000 00000000 00000000 00000000 dededede
<4>[   49.036877] 8820  00000000 30383231 38363778 00000000 00000000 00000000 00000000 00000000
<4>[   49.036907] 8840  00000000 00000000 00000048 0000f488 00000500 00000510 00000514 00000521
<4>[   49.036936] 8860  00000000 00000300 00000303 00000305 00000326 00000000 1900000a 00000000
<4>[   49.036965]
<4>[   49.036965] R7: 0xc0bb520c:
<4>[   49.036970] 520c  00000001 00000000 c0bb5214 c0bb5214 00000000 00000000 00000000 00000000
<4>[   49.036999] 522c  00000000 00000000 00000000 00000000 00000000 00000000 eea751c0 00000000
<4>[   49.037028] 524c  00040004 ee8b3480 00000000 00000000 00000000 00000000 00380038 ee13a5b0
<4>[   49.037056] 526c  ee13a5b0 00000001 00000000 00000000 00000000 00000000 ee240a90 00000001
<4>[   49.037085] 528c  00000000 ee6b9330 ee18d8c0 00000000 00000000 00000000 00000000 00000000
<4>[   49.037113] 52ac  00000000 00000000 00000000 00000001 00000000 c0bb52c0 c0bb52c0 00000000
<4>[   49.037141] 52cc  00000000 00000000 00000000 00000001 00000000 c0bb52e0 c0bb52e0 00000000
<4>[   49.037170] 52ec  00000000 00000000 00000000 00000001 00000000 c0bb5300 c0bb5300 00000000
<4>[   49.037199]
<4>[   49.037199] R8: 0xc0bb520c:
<4>[   49.037204] 520c  00000001 00000000 c0bb5214 c0bb5214 00000000 00000000 00000000 00000000
<4>[   49.037238] 522c  00000000 00000000 00000000 00000000 00000000 00000000 eea751c0 00000000
<4>[   49.037266] 524c  00040004 ee8b3480 00000000 00000000 00000000 00000000 00380038 ee13a5b0
<4>[   49.037295] 526c  ee13a5b0 00000001 00000000 00000000 00000000 00000000 ee240a90 00000001
<4>[   49.037323] 528c  00000000 ee6b9330 ee18d8c0 00000000 00000000 00000000 00000000 00000000
<4>[   49.037352] 52ac  00000000 00000000 00000000 00000001 00000000 c0bb52c0 c0bb52c0 00000000
<4>[   49.037381] 52cc  00000000 00000000 00000000 00000001 00000000 c0bb52e0 c0bb52e0 00000000
<4>[   49.037409] 52ec  00000000 00000000 00000000 00000001 00000000 c0bb5300 c0bb5300 00000000
<4>[   49.037438]
<4>[   49.037438] R9: 0xd160e4a0:
<4>[   49.037444] e4a0  d11eb39c d160e0dc d127ade8 eabd7e24 d1221200 eabd7e00 00000000 00000000
<4>[   49.037473] e4c0  00000000 00000000 00000000 00000000 00000018 00000028 00000000 00000000
<4>[   49.037502] e4e0  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.037530] e500  d160e520 00000000 00000000 edae8000 00000000 c0243048 00000000 00000000
<4>[   49.037558] e520  00000002 00000020 b6679000 00000000 00000000 00000000 00000000 00000000
<4>[   49.037587] e540  00000000 00000000 00000000 00000000 00000018 00000028 00000000 00000000
<4>[   49.037616] e560  ffffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<4>[   49.037645] e580  ed8d7600 cd6dccc0 ed8d76a0 d127afc8 d160e8d0 d160ecd0 00000000 00000000
<0>[   49.037676] Process surfaceflinger (pid: 104, stack limit = 0xedb40210)
<0>[   49.037682] Stack: (0xedb41cb0 to 0xedb42000)
<0>[   49.037690] 1ca0:                                     024000c0 c049e130 edb41cdc edb41cc8
<0>[   49.037699] 1cc0: c0160d28 c01eabe8 ee801e80 c0bb528c edb41d1c d53dbb80 ee1dec00 d53dbb80
<0>[   49.037708] 1ce0: 00000001 c0bb528c d53db300 d53db380 edb41d14 edb41d00 c045cbc0 c045c1f0
<0>[   49.037716] 1d00: c045cba0 00000001 edb41d64 edb41d18 c04832f0 c045cbac edb41d3c edb41d28
<0>[   49.037724] 1d20: c0160d28 c01eabe8 ee801f00 00076f08 edb41d7c ee1dec00 c0244aa0 d53dbb80
<0>[   49.037732] 1d40: ee1dec00 d1048800 00000001 ee1ded00 edae8000 d53dbb80 edb41d7c edb41d68
<0>[   49.037741] 1d60: c048376c c0483158 b6679000 00000000 edb41e2c edb41d80 c04848b4 c0483758
<0>[   49.037749] 1d80: 00000000 00000000 00000000 00000001 b6af68c0 b6add320 b6aca3f0 ee29c030
<0>[   49.037757] 1da0: ee1ded00 edae8000 00000018 b6aca408 00000000 00000000 00000002 00000003
<0>[   49.037765] 1dc0: ee1dec00 c0243048 c0ba6258 edb41e58 3231564e 00000000 00000000 00000000
<0>[   49.037773] 1de0: 00000001 eda95b00 00000054 00000003 00000000 ee29c028 ee1df828 00000000
<0>[   49.037781] 1e00: edb41e2c c0bb528c 00000051 ee1dec00 edb41e58 000000bc edae8000 c079a47c
<0>[   49.037790] 1e20: edb41f04 edb41e30 c0468210 c0484568 00020257 00000000 ffffffff 00000001
<0>[   49.037798] 1e40: edb41e58 c048455c befc85e0 00000000 00000038 c03864bc 00000201 00000002
<0>[   49.037806] 1e60: b6aca408 00000000 b6aca3f0 00000000 b6add320 00000000 b6af68c0 00000000
<0>[   49.037814] 1e80: 00000000 00000000 b6679000 00000000 00000000 00000000 00000000 00000000
<0>[   49.037821] 1ea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[   49.037829] 1ec0: 69746572 665f6572 00000000 00000000 00000000 000000bc 00000073 edae8e41
<0>[   49.037837] 1ee0: edae8e40 00000013 c03864bc 00000000 befc85e0 eda6f920 edb41f7c edb41f08
<0>[   49.037846] 1f00: c025e2d8 c0467ff4 edb41f20 00000053 edb41f44 ee0a9490 ed4c3f68 edae64bc
<0>[   49.037854] 1f20: edb41f0b edb41f14 c0268f78 c018ea50 edadb800 c0b4ad3c c0b4872c befc85e0
<0>[   49.037862] 1f40: c03864bc edae8e40 edb41f7c edb41f58 c036a3e4 edae8e41 edae8e40 befc85e0
<0>[   49.037869] 1f60: c03864bc 00000000 00000013 00000000 edb41fa4 edb41f80 c025e590 c025de2c
<0>[   49.037878] 1f80: b4fd5100 00000013 c03864bc 00000036 c01080e4 edb40000 00000000 edb41fa8
<0>[   49.037886] 1fa0: c0107f40 c025e520 b4fd5100 00000013 00000013 c03864bc befc85e0 befc85b0
<0>[   49.037894] 1fc0: b4fd5100 00000013 c03864bc 00000036 b6aca408 00000013 00000000 00000023
<0>[   49.037903] 1fe0: befc85e0 befc85a0 b6dde8a5 b6dd7e28 600f0010 00000013 00000000 00000000
<4>[   49.037917] [<c045c2f0>] (drm_atomic_helper_check_modeset) from [<c045cbc0>] (drm_atomic_helper_check+0x20/0x38)
<4>[   49.037930] [<c045cbc0>] (drm_atomic_helper_check) from [<c04832f0>] (drm_atomic_check_only+0x1a4/0x58c)
<4>[   49.037943] [<c04832f0>] (drm_atomic_check_only) from [<c048376c>] (drm_atomic_async_commit+0x20/0x74)
<4>[   49.037954] [<c048376c>] (drm_atomic_async_commit) from [<c04848b4>] (drm_mode_atomic_ioctl+0x358/0x7b8)
<4>[   49.037965] [<c04848b4>] (drm_mode_atomic_ioctl) from [<c0468210>] (drm_ioctl+0x228/0x4cc)
<4>[   49.037977] [<c0468210>] (drm_ioctl) from [<c025e2d8>] (do_vfs_ioctl+0x4b8/0x6f4)
<4>[   49.037988] [<c025e2d8>] (do_vfs_ioctl) from [<c025e590>] (SyS_ioctl+0x7c/0x8c)
<4>[   49.038000] [<c025e590>] (SyS_ioctl) from [<c0107f40>] (ret_fast_syscall+0x0/0x1c)
<0>[   49.038010] Code: e50b1030 e793a106 1a0001dd e59922bc (e59a3004)

------------------------------------------

Notice that the "PC is at drm_atomic_helper_check_modeset+0x10c"

We create objdump using the following command :-

arm-linux-androideabi-objdump -S out/target/product/gminfo3/obj/KERNEL_OBJ/vmlinux > vmlinux_objdump-new.txt

And its obvious that you build kernel with KALLSYMS and DEBUG_INFO configs.

Now from the OBJDUMP, we get :-

<- The function - drm_atomic_helper_check_modeset  ->

c055c304 <drm_atomic_helper_check_modeset>:

* Zero for success or -errno

*/

int

drm_atomic_helper_check_modeset(struct drm_device *dev,

                                                            struct drm_atomic_state *state)

{

c055c304:           e1a0c00d            mov       ip, sp

c055c308:           e92ddff0             push       {r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc}

c055c30c:            e24cb004            sub         fp, ip, #4

c055c310:           e24dd024           sub         sp, sp, #36           ; 0x24

c055c314:           e52de004           push       {lr}                         ; (str lr, [sp, #-4]!)

c055c318:           ebf2cb45             bl            c020f034 <__gnu_mcount_nc>

               struct drm_crtc_state *crtc_state;

               struct drm_connector *connector;

Now , we found the (c055c304 + offset) which is 0xc055c410

<- We found this from the objdump ->

               DRM_DEBUG_ATOMIC("Updating routing for [CONNECTOR:%d:%s]\n",

c055c408:           1a0001dd           bne        c055cb84 <drm_atomic_helper_check_modeset+0x880>

                                             connector->base.id,

                                             connector->name);

               if (connector->state->crtc != connector_state->crtc) {

c055c40c:            e59922bc            ldr          r2, [r9, #700]      ; 0x2bc

c055c410:           e59a3004            ldr          r3, [sl, #4]   ------------------> Point of crash.

c055c414:           e5920004           ldr          r0, [r2, #4]

c055c418:           e1500003           cmp       r0, r3

Looks like r0 and r3 is compared here (connector->state->crtc != connector_state->crtc)

From our original crash logs -

<4>[   49.034695] r10: 00000000  r9 : d160e520  r8 : c0bb528c

<4>[   49.034707] r3 : d160e300  r2 : 00000000  r1 : c0bb528c  r0 : 00000001

Possibilities :-

1. Note r2 is zero here.Hence looks like "connector->state->crtc" has crashed as the state variable is NULL.The offset 0x700 is actually offset of connector->state.However as r9 is holding a valid value, we dont know why r2 is ZERO. 

2. Crash could be on connector_state->crtc which is point of crash. Here s1 may be invalid. But r3 here is not Zero hence a NULL pointer crash cannot be explained.

Just for Note :-

the ldr instruction using the s1 below.

               DRM_DEBUG_ATOMIC("Updating routing for [CONNECTOR:%d:%s]\n",

c055c3f8:            e34c10ec            movt      r1, #49388          ; 0xc0ec

c055c3fc:            e3120010           tst          r2, #16

c055c400:           e50b1030           str          r1, [fp, #-48]       ; 0xffffffd0

               struct drm_connector_state *connector_state;

               struct drm_crtc_state *crtc_state;

               int idx, ret;

               connector = state->connectors[conn_idx];

               connector_state = state->connector_states[conn_idx];

c055c404:           e793a106            ldr          sl, [r3, r6, lsl #2]

FINAL VERDICT :-

WE HAVE MESSED UP SOMEWHERE IN THE BELOW CODE :-

if (connector->state->crtc != connector_state->crtc) {   <-  connector->state may be either NULL or a dangling pointer. This is where crash has happened.