[Feature] Machine Learning using Rockchip NPU on supported boards.

[yoni13]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Since immich already supports Hardware Transcoding with hardware acceleration via rkmpp, maybe we can make machine learning working on rockchip NPUs.

Links:
rknn-toolkit2
frigate's implantation

Platform

• Server
• Web
• Mobile

[yoni13]

Update: I am able to run "ViT-B-32__openai" using rknn-toolkit2 on emulator, I haven't verify the accuracy but I will say it got some progress.

[mertalev]

AnnSession uses None for the name and only provides the shape, so this can do the same (or provide None for both if the shape is also unavailable).

[yoni13]

I am out of ideas now 🫠.
I think I need to know more about insightface to solve this.

[mertalev]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

For this, just call onnx runtime to get the correct faces and unload it.

[yoni13]

(machine-learning-py3.11) root@orangepi3b:~/immich/machine-learning# ./start.sh 
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libmimalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[12/03/24 12:15:23] INFO     Starting gunicorn 23.0.0                                                                                                                                        
[12/03/24 12:15:23] INFO     Listening at: http://[::]:3003 (5620)                                                                                                                           
[12/03/24 12:15:23] INFO     Using worker: app.config.CustomUvicornWorker                                                                                                                    
[12/03/24 12:15:23] INFO     Booting worker with pid: 5621                                                                                                                                   
[12/03/24 12:15:32] I        Started server process [5621]                                                                                                                                   
[12/03/24 12:15:32] I        Waiting for application startup.                                                                                                                                
[12/03/24 12:15:32] I        Created in-memory cache with unloading after 300s of inactivity.                                                                                                
[12/03/24 12:15:32] I        Initialized request thread pool with 4 threads.                                                                                                                 
[12/03/24 12:15:32] I        Application startup complete.                                                                                                                                   
[12/03/24 12:15:34] I        Loading textual model 'ViT-B-32__openai' to memory                                                                                                              
W rknn-toolkit-lite2 version: 2.3.0
[12/03/24 12:15:34] I        Loading RKNN model from /cache/clip/ViT-B-32__openai/textual/model.rknn                                                                                         
--> Load RKNN model
I RKNN: [12:15:35.403] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [12:15:35.403] RKNN Driver Information, version: 0.9.7
I RKNN: [12:15:35.404] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
W RKNN: [12:15:35.728] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.

Can someone help me with dynamic_shapes&static_shapes? I have no idea what are them.
Or we can ignore that warning?

[mertalev]

If it is a static shape RKNN model, please ignore the above warning message

I think you can ignore it if there are no errors.

[yoni13]

Small update: I tested the model in Immich, and the ViT-B-32 shows a latency difference of 200 ms after using RKNN(from cpu bcs I am on dietpi).

[yoni13]

outputs :  list[NDArray[np.float32]] = self.rknn.inference(inputs=inputs, data_format='nchw')

and

[01/06/25 09:12:35] I        Running inference on RKNN model                                                                                                                                         
W The input[0] need NHWC data format, but NCHW set, the data format and data buffer will be changed to NHWC.

But


Am I missing something here?🤔
Wasn't the model NCHW?

[mertalev]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

[yoni13]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

So, input being NCHW should not be an issue?
🤔

[yoni13]

I used a random array to test the diff between the result run by onnx and rknn.
LOG: https://pastebin.com/26JGWZUE
DIFF: https://www.diffchecker.com/sLmjBdfZ/

The model seems to be working, might there be an issue with our inputs?

[yoni13]

No, not inputs.

from rknnlite.api import RKNNLite
import numpy as np
import onnxruntime as ort


# Model paths
rknn_model_path = "/cache/clip/ViT-B-32__openai/visual/model.rknn"
onnx_model_path = "/cache/clip/ViT-B-32__openai/visual/model.onnx"

rknn = RKNNLite()
rknn.load_rknn(rknn_model_path)
rknn.init_runtime()

session = ort.InferenceSession(onnx_model_path, providers=['CPUExecutionProvider'] )

def check_for_inf(array):
  """Checks if an array contains inf or -inf values."""
  return np.isinf(array).any()

def test_models(input_data, rknn_model_path, onnx_model_path):
  """Tests the RKNN and ONNX models with the given input data."""
  # RKNN model inference
  rknn_result = rknn.inference(inputs=[input_data], data_format='nchw')[0]

  # ONNX model inference
  onnx_result = session.run(None, {"image": input_data}, None)[0]

  return rknn_result, onnx_result


# Generate random input data and test until inf is found
iteration = 0
while True:
  iteration += 1
  input_data = np.random.rand(1, 3, 224, 224).astype(np.float32)
  rknn_result, onnx_result = test_models(input_data, rknn_model_path, onnx_model_path)

  if check_for_inf(rknn_result) or check_for_inf(onnx_result):
    print(f"Model produced inf/-inf at iteration {iteration}")
    print("RKNN result:", rknn_result)
    print("ONNX result:", onnx_result)
    break

  print(f"Iteration {iteration}: No inf/-inf found yet.")

I asked Gemini to write a script to test, but after 500 iteration, no inf,-infs found.

Hope this leads us to learn more what happened.

[sgofferj]

I don't know anything about machine learning development but I do have a FriendlyElec CM3588 NAS which is mostly idling at the moment. I was anyways planning to move my Immich to that from my 15 years old Xeon home server.
I can't offer much coding help but I can offer CPU/NPU time if that helps.

[yoni13]

Oh, wait!
After changing Smart Search concurrency to 1, it works! (clip) 🎉🎉🎉

It looks like it got itself race-conditioned or something.

[yoni13]

Search: (CPU)2s -> (NPU)750 ms
I just found out that the previous result's CPU model was not preloaded, there are 100~500ms improvements sometimes.

[yoni13]

https://github.com/leafqycc/rknn-multi-threaded
I wonder how this works.

[yoni13]

Ok, now all models (clip ViT-B-32 and buffalo_l) work with concurrency 1.
I will push updates and models to my repo later.
Also, I don't think concurrency 1 is acceptable, will investigate it further when I have time.

[yoni13]

https://github.com/yoni13/immich_to_rknn2/releases/tag/prerelease

[yoni13]

mem usage was a bit high, maybe can try to unload the Ortsession that was used to get input/output faces or just provide a dummy for it.

[yoni13]

Ok, now all models (clip ViT-B-32 and buffalo_l) work with concurrency 1. I will push updates and models to my repo later. Also, I don't think concurrency 1 is acceptable, will investigate it further when I have time.

It is fixed! By integrating rknnpool.py