deviceQuery returns "No CUDA-capable device detected" on xavier-nx based production board, using linux-tegra 5.10

[f-gia]

Hello,

We are trying to build a working image for the xavier-nx-emmc devkit installed on a custom production board.

We built a image based on core-image-minimal and used Poky as a distro. We added the meta-tegra layer, and added cuda-samples to IMAGE-INSTALL. We also replaced linux-tegra 4.9 with linux-tegra 5.10, by editing the recipe in meta-tegra. Please see attachment for the resulting defconfig.

The built image flashes and boots successfully, but when we run deviceQuery we get the following output:

root@jetson-xavier-nx-devkit-emmc:/usr/bin/cuda-samples# ./deviceQuery          
./deviceQuery Starting...                                                       
                                                                                
 CUDA Device Query (Runtime API) version (CUDART static linking)                
                                                                                
NvRmPrivGetChipPlatform: Could not read platform information                    
Expected on kernels without fuse support, using silicon                         
cudaGetDeviceCount returned 100                                                 
-> no CUDA-capable device is detected                                           
Result = FAIL

However, if we use linux-tegra 4.9 (built by default by meta-tegra), deviceQuery works and recognizes the GPU.

What could be wrong with our configuration?
l4t5.10_defconfig.zip

[madisongh]

The 5.10 kernel is only for use with the JetPack 5-based userland binaries. If you switch meta-tegra to the wip-jetpack-5-dp branch, instead of just updating the kernel recipe, you should get a better result.

[f-gia]

We switched to the wip-jetpack-5-dp branch at the latest revision. However, with all the cuda binaries, we get a fetching error. Here is an example relative to libcudla:

ERROR: libcudla-11.4.168-1-r0 do_fetch: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.', 'https://repo.download.nvidia.com/jetson/common/pool/main/libc/libcudla/libcudla-11-4_11.4.168-1_arm64.deb;name=main;subdir=libcudla-11.4.168-1')

By looking at https://repo.download.nvidia.com/jetson/ it appears that the only package available is a new version of libcudla: libcudla-11-4_11.4.239-1_arm64.deb at https://repo.download.nvidia.com/jetson/common/pool/main/libc/libcudla/libcudla-11-4_11.4.239-1_arm64.deb.

How can we fetch the right versions of the packages?

Here is the list of all the packages for which we get a fetcher error:

  /work/layers/meta-tegra/recipes-devtools/cuda/libcublas_11.6.5.24-1.bb:do_fetch                                                                                                                                                             
  /work/layers/meta-tegra/recipes-devtools/cuda/libcurand_10.2.5.165-1.bb:do_fetch                                                                                                                                                            
  /work/layers/meta-tegra/recipes-devtools/cuda/cuda-cccl_11.4.167-1.bb:do_fetch        
  /work/layers/meta-tegra/recipes-devtools/cuda/cuda-nvcc-headers_11.4.166-1.bb:do_fetch
  /work/layers/meta-tegra/recipes-devtools/cuda/cuda-cudart_11.4.167-1.bb:do_fetch  
  /work/layers/meta-tegra/recipes-devtools/cuda/libnpp_11.4.0.155-1.bb:do_fetch
  /work/layers/meta-tegra/recipes-devtools/cuda/libcufft_10.6.0.71-1.bb:do_fetch    
  /work/layers/meta-tegra/recipes-devtools/cuda/cuda-nvrtc_11.4.166-1.bb:do_fetch                                      
  /work/layers/meta-tegra/recipes-devtools/cuda/libcudla_11.4.168-1.bb:do_fetch                                        
  /work/layers/meta-tegra/recipes-devtools/cuda/libcusparse_11.6.0.165-1.bb:do_fetch                                                                                                                                                          
  /work/layers/meta-tegra/recipes-devtools/cuda/cuda-driver_11.4.167-1.bb:do_fetch                                                                                                                                                            
  /work/layers/meta-tegra/recipes-devtools/cuda/libcusolver_11.2.0.165-1.bb:do_fetch                                                                                                                                                          

[ichergui]

Hello @f-gia
Are you using the local packages that you download via Jetpack-5.0-DP ? if no, please use the following command before running setup-env script:

$ export NVIDIA_DEVNET_MIRROR="/PATH/TO/JEPACK-5.0-DP/nvidia/sdkm_downloads/"

Please check your package if exists, in the case it doesn't exists you need to run the following command line:

$ dpkg-deb --fsys-tarfile cuda-repo-l4t-11-4-local_11.4.14-1_arm64.deb | tar -x -f- --strip-components=3 ./var/cuda-repo-l4t-11-4-local

Please try it and let us know how it goes

[madisongh]

It does look like NVIDIA dropped a new version of libcudla. I've updated the branch with a recipe to reference the newer version.

[f-gia]

Now it fails to fetch the documentation package when building libcudla:
libcudla-11.4.239-1-r0 do_fetch: Failed to fetch URL https://repo.download.nvidia.com/jetson/common/pool/main/c/cuda-documentation/cuda-documentation-11-4_11.4.167-1_arm64.deb;name=lic;subdir=libcudla-11.4.239-1, attempting MIRRORS if available

[ichergui]

😞 Yeah, It looks like Nvidia drop that package as well
We need to update to cuda-documentation-11-4_11.4.239-1_arm64.deb

[madisongh]

Then there are probably others as well. The whole set of CUDA recipes probably needs a refresh, sorry.

[ichergui]

I can do that later when I have time.
It looks like cuda-documentation package version 11.4.239-1 exists for ARM64 but not for X86_64 😞
I will take a look deeply later on the cuda recipes and will keep you updated guys.

[madisongh]

I've taken care of it. If you update to the latest on the JP5 branch again, you should see the new recipes. The old recipes are also kept for use on the x86-64 side.