kernel panic on boot when hdmi is plugged in on a jetson agx industrial

[elbbit01]

Hi everybody,
I am building based on kirkstone branch with l4t 35.3.1. I was actually building with 35.2.1 and everything was fine and when I updated to 35.3.1 the issue came (didnt catch it for a while since I do almost everything via ssh, but my clients found it). It is not an option at the moment to update to 35.4.1. It will come but we want to keep the library versions if possible.

I get a kernel panic when booting with hdmi connected, if I remove it, it boots normally and then I can plug it in and there is a normal behaviour.

Here is the serial outcome:

OK ] Reached target Preparation for Local File Systems.
Mounting /var/volatile...
Starting Flush Journal to Persistent Storage...
[ 10.417965] audit: type=1334 audit(1651167746.944:2): prog-id=5 op=LOAD
[ 10.419360] audit: type=1334 audit(1651167746.944:3): prog-id=6 op=LOAD
Starting Rule-based Manage…for Device Events and Files...
[ OK ] Mounted /var/volatile.
[ 10.432610] systemd-journald[303]: Received client request to flush runtime journal.
Starting Load/Save Random Seed...
[ OK ] Finished Flush Journal to Persistent Storage.
[ 10.459162] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 10.460656] Mem abort info:
[ 10.461915] ESR = 0x96000004
[ 10.464024] EC = 0x25: DABT (current EL), IL = 32 bits
[ 10.469834] SET = 0, FnV = 0
[ 10.472676] EA = 0, S1PTW = 0
[ 10.476219] Data abort info:
[ 10.478327] ISV = 0, ISS = 0x00000004
[ 10.482580] CM = 0, WnR = 0
[ 10.485218] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000104feb000
[ 10.492051] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[ 10.498460] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 10.504386] Modules linked in: tegra_udrm pwm_fan nvgpu nvmap fuse
[ 10.510511] CPU: 1 PID: 321 Comm: udevadm Not tainted 5.10.104-l4t-r35.3.ga-K 2m OKe:.2.0+ #1
[ ;39 10.62500mRu syle-based _kf_seq_showr for +0xDevice Ev0x1ents and 2c, BIO[0m.5.3.1 01/24/2023
[ 10.625041] kernfs_seq_show+0x48/0x54
[ 10.646633] seq_read_iter+0x1d4/0x4e0
[ 10.650101] kernfs_fop_read_iter+0x154/0x1d0
[ 10.654635] new_sync_read+0xf8/0x18c
[ 10.658016] vfs_read+0x134/0x1c0
[ 10.661223] ksys_read+0x78/0x100
[ 10.664886] __arm64_sys_read+0x28/0x34
[ 10.668396] el0_svc_common.constprop.0+0x80/0x1c4
[ 10.673452] do_el0_svc+0x74/0x8c
[ 10.676414] el0_svc+0x1c/0x2c
[ 10.679379] el0_sync_handler+0x9c/0x120
[ 10.683577] el0_sync+0x16c/0x180
[ 10.686558] Code: aa0303e0 d2820002 52800001 9409e5c3 (f94002a4)
[ 10.692682] ---[ end trace 813edcb2742ed1a4 ]---
[ 10.697197] Kernel panic - not syncing: Oops: Fatal exception
[ 10.702827] SMP: stopping secondary CPUs
[ 10.706852] Kernel Offset: 0x5f6285a40000 from 0xffff800010000000
[ 10.712544] PHYS_OFFSET: 0xffff857940000000
[ 10.716557] CPU features: 0x8240002,03802a30
[ 10.720897] Memory Limit: none
[ 10.724295] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---

I have found some issues in nvidia forums, but I guess you are aware of the issue and fixed it somehow. Just in case, I tried the patch described here:
https://forums.developer.nvidia.com/t/kernel-panic-has-occurred-when-doing-reboot-stress-test-with-jp5-0-2/249597
It didnt work (I mean, the patch did work on the file, I adapted it to the paths of linux-tegra, but the problem persists).

It is surprising because they describe my problem one to one but with a previous jetpack, it should have being fixed some months ago. I did not find anything on the metaoe-patches that could be related.

Ah, almost forgot to say, we have a modified bsp built over the jetson xavier agx industrial. Until now it was working but my next guess is that our device tree modifications might have something to do with this patches, if this changed on the lasts jetpacks.

Do you guys have a hint? I would really appreciate it.

Best regards,
Alvaro.

[madisongh]

Does your copy of the layer have this commit in it? If not, you might try updating to include it and see if it helps. There were some changes to the boot-time display controller initialization for the Xaviers that we missed during the R35.3.1 update.

[elbbit01]

Hi @madisongh,
thank you for your answer. This was one of my first guesses, I updated to the commit dd0d91a and built again (before I was behind that patch you are guessing). The error persists, but I start considering if my flashing process is missing some tiny part somehow.

Do you know a way how I could check if the patch is applied on the target? As I said, it is an industrial layout using QSPI.

Thanks for your help,
Alvaro.

[madisongh]

Do you know a way how I could check if the patch is applied on the target? As I said, it is an industrial layout using QSPI.

I'm not sure there's an easy way to do that at runtime. If you rebuilt everything after updating, then re-flash the QSPI flash, it should be there. You can at least check the build tree for the nvdisp-init in your workspace at that point, to see if the updated patches show up there.

[elbbit01]

Hi @madisongh.
Yes, it should be there. nvdisp-init is patched at the workspace.
I managed with divide and conquer to isolate the issue to my CORE_IMAGE_EXTRA_INSTALL += " packagegroup-base ". Does this tell you something? When I remove it from local.conf, it boots without a problem (I have other issues with the A/B tooling, then). Still I am not sure what is the broken detail, if I manage to find it out I will try to keep the information here too for helping others who might have a similar issue.

Regards,
Alvaro.

[madisongh]

Try comparing the image .manifest files between your working and non-working builds to see what packages that packagegroup is pulling in that might be causing the problem.

[elbbit01]

Hi, I just wanted to write that my issue has been postponed a couple of weeks since there are demos next week and this is not a blocker, but I will try to come back with more feedback so the ticket can get closed properly.

[elbbit01]

Hello,
I got back to the problem and could actually fix it. I will copy here what I did just in case somebody gets to the same problem.

Comparing manifest, I saw some interesting findings in the kernel modules we were including, specially nvidia-display-driver.
Somehow, when we included the packagegroup-base in our configuration, it built everything, but not for our machine config. I took the interesting modules from the comparison of the manifests between the version that was not breaking (building without packagegroup-base) and the one breaking (building with packagegroup-base) and added them directly to my image recipe one by one, instead of through the packagegroup-base. There is also something about nv-boot and nv-redundant-tools that I actually need, but explicitly the kernel recipes for nvidia-display-driver and load-drm-module were breaking when I include them like this.
It makes sense, since we build with modified bsp layers and machine config, that I had to add our config to the compatible machines of both recipes.
After that, it builds and the problem is not appearing any more, along with other problems I had noticed in between like booting always in slot 0 after power off.

You can close the ticket from my part, thank you for the hints.

Regards,
Alvaro.