Bootloader update fails on Xavier NX

[tirankas]

Hello,
Recently we are experiencing failures during the mender update process.
This happens on a device basis, few devices are failing to update while other works fine.
What we do?
Standard mender update process, either server based or directly on device: mender -install

What we see:
After the rootfs artifact has been installed, the ArtifactInstall_Leave_80_bl-update scripts says: "ERR: bootloader update failed"
Some error comes out from the nv_update_engine tool.

Anyone experienced this issue before?
We have done this same process hundreds of times already, no issue till a week ago, then started seeing 20% failures, same SW.

Full logs output:

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/mmcblk0p1
INFO[0000] Start updating from local image file: [image-prod-jetson-xavier-nx-emmc.mender]
Installing Artifact of size 1039970304...
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
INFO[0000] Opening device "/dev/mmcblk0p11" for writing
INFO[0000] Native sector size of block device /dev/mmcblk0p11 is 512 bytes. Mender will write in chunks of 1048576 bytes
.............................................................. - 100 %
INFO[0020] All bytes were successfully written to the new partition
INFO[0020] The optimized block-device writer wrote a total of 995 frames, where 995 frames did need to be rewritten (i.e., skipped)
INFO[0021] Wrote 1043202048/1043202048 bytes to the inactive partition
INFO[0021] Enabling partition with new image installed to be a boot candidate: 11
INFO[0021] Executing script: ArtifactInstall_Leave_80_bl-update
INFO[0024] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
ERR: bootloader update failed

---------- end of script output
ERRO[0024] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1
Rolling back Artifact...
INFO[0024] Rolling back to the inactive partition (11).
ERRO[0024] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1

Thanks

[ichergui]

Hey @tirankas
Could you please give us detail about your build environment (repo, branch, steps, ...) ?
I believe with the recent changes A/B system update works fine.

[tirankas]

Using the dunfell 32.5.0
And as said, it does work fine for me as well, but now new device are failing at 20% rate, same sw that works fine on other devices...
I did flash then with the tegraflash, as usual, which worked fine, but any mender install afterwards fails.

[madisongh]

When that error occurs (and before any reboot or any other action), can you run that script manually with execution tracing turned on to see exactly what is triggering it?

$ sudo bash -x /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update

That would help in diagnosing the problem.

[tirankas]

Sure, here is the output:

jetson-xavier-nx-emmc-juganu:~$ sudo bash -x /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update

• mnt=
• LABELCHARS=AB
• COPY_MACHINE_ID=yes
• echo 'Installing NVIDIA bootloader update payload'
  Installing NVIDIA bootloader update payload
  ++ nvbootctrl get-number-slots
• num_slots=2
• '[' 2 '!=' 2 ']'
  ++ nvbootctrl get-current-slot
• current_slot=0
• echo 'Current boot slot: 0'
  Current boot slot: 0
  ++ expr 1 - 0
• otherslot=1
• cfglbl='"RootfsPartB"'
  ++ grep -h '"RootfsPartB":' /etc/mender/mender.conf /var/lib/mender/mender.conf
  ++ cut -d: -f2
  ++ cut -d, -f1
  ++ tr -d '" '
• devnam=/dev/mmcblk0p11
• '[' -z /dev/mmcblk0p11 ']'
  ++ mktemp -d -t nvbup.XXXXXX
• mnt=/tmp/nvbup.H3G6V0
• '[' -z /tmp/nvbup.H3G6V0 -o '!' -d /tmp/nvbup.H3G6V0 ']'
• mount -o ro /dev/mmcblk0p11 /tmp/nvbup.H3G6V0
• '[' '!' -d /tmp/nvbup.H3G6V0/opt/ota_package ']'
• mount --bind /sys /tmp/nvbup.H3G6V0/sys
• mount --bind /proc /tmp/nvbup.H3G6V0/proc
• mount --bind /dev /tmp/nvbup.H3G6V0/dev
• mount -t tmpfs tmpfs /tmp/nvbup.H3G6V0/run
• '[' -L /tmp/nvbup.H3G6V0/etc/nv_boot_control.conf -a -x /tmp/nvbup.H3G6V0/usr/bin/setup-nv-boot-control ']'
• mkdir -p /tmp/nvbup.H3G6V0/run/tegra-nv-bootctrl
• chroot /tmp/nvbup.H3G6V0 /usr/bin/setup-nv-boot-control
• chroot /tmp/nvbup.H3G6V0 /usr/sbin/nv_update_engine --install no-reboot
  Nvidia A/B-Redundancy Update tool Version 2.0
  current slot 0
  SM: S11
  Set slot 1 as unbootable and start updating.
  Start running: /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition /opt/ota_package/bl_update_payload
  Got update payload: /opt/ota_package/bl_update_payload
  Tegra User Block Device: /dev/disk/by-partlabel
  Tegra Boot Block Device: /dev/mtdblock0
  HEADER: MAGIC NVIDIA__BLOB__V2
  HEX_VALUE 131072
  BLOB_SIZE 50609030
  HEADER_SIZE 40
  NUMBER_OF_ELEMENTS 30
  HEADER_TYPE 0
  UNCOMP_SIZE 50609030
  MB1_RATCHET_LV 115
  MTS_RATCHET_LV 112
  ROLLBACK_FUSE_LV 101
  config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
  Device TN Spec: 3668-300-0001-B.0-1-2-jetson-xavier-nx-emmc-juganu-mmcblk0p1
  Can't find matching TN Spec in OTA Blob!
  OTA Blob update failed. Status: 3
  /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed.
  Slot 1 update failed
  Error: installing bootloader updates failed: 768
• echo 'ERR: bootloader update failed'
  ERR: bootloader update failed
• cleanup
• '[' -n /tmp/nvbup.H3G6V0 ']'
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.H3G6V0/sys
• umount /tmp/nvbup.H3G6V0/sys
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.H3G6V0/proc
• umount /tmp/nvbup.H3G6V0/proc
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.H3G6V0/dev
• umount /tmp/nvbup.H3G6V0/dev
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.H3G6V0/run
• umount /tmp/nvbup.H3G6V0/run
• mountpoint -q /tmp/nvbup.H3G6V0
• umount /tmp/nvbup.H3G6V0
• rmdir /tmp/nvbup.H3G6V0
• exit 1

[tirankas]

And:
sudo nvbootctrl get-number-slots
2

[ichergui]

It looks like you have a recent Jetson Xavier NX with FAB equal to 300 which should be supported when using Jetpack 4.5.1 / L4T R32.5.1

Did you update meta-tegra because we pushed recently this commit ? if no, could you please update meta-tegra and give a try.

[madisongh]

Device TN Spec: 3668-300-0001-B.0-1-2-jetson-xavier-nx-emmc-juganu-mmcblk0p1
Can't find matching TN Spec in OTA Blob!

There's the root cause. This is a new revision of the module (FAB 300), and the BUP payload doesn't include support for it. We just added that in earlier this week, and I've just updated the demo distro to pull in that change.

[tirankas]

Thanks @madisongh and @ichergui
Appreciate your help, will add support for this new FAB