How to run TFLite benchmark with QNN delegate in Android

[4570235]

I have built/installed/run TFLite benchmark following this instruction for Android, and used TensorFlow 2.15.0 according to issue#66015. I test the benchmark via the following commands and the output result seems correct.

adb push /Users/handleychen/Github/tensorflow/tensorflow/bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model /data/local/tmp
adb shell chmod +x /data/local/tmp/benchmark_model
adb shell "mkdir /data/local/tmp/models"
adb push /Users/handleychen/Github/tensorflow/models/*.tflite /data/local/tmp/models
adb shell /data/local/tmp/benchmark_model --graph=/data/local/tmp/models/mobilenet_v1_1.0_224.tflite --num_threads=4 --enable_op_profiling=true
adb shell /data/local/tmp/benchmark_model --graph=/data/local/tmp/models/mobilenet_v1_1.0_224.tflite --use_gpu=true --enable_op_profiling=true
adb shell /data/local/tmp/benchmark_model --graph=/data/local/tmp/models/mobilenet_v1_1.0_224.tflite --use_nnapi=true --enable_op_profiling=true

benchmark result.txt

Now I want to run the benchmark with QNN delegate. I setup the on device environment and run a QNN delegate using an external delegate. I tested it using the following commands, but received an error message.

adb shell "mkdir /data/local/tmp/qnn_delegate"
adb push /Users/handleychen/Github/quic/SDK/qairt/2.26.0.240828/lib/aarch64-android/* /data/local/tmp/qnn_delegate
adb shell
cd /data/local/tmp
export LD_LIBRARY_PATH=/data/local/tmp/qnn_delegate
export ADSP_LIBRARY_PATH="/data/local/tmp/qnn_delegate"
./benchmark_model --graph=/data/local/tmp/models/mobilenet_v1_1.0_224.tflite --external_delegate_path=/data/local/tmp/qnn_delegate/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/data/local/tmp/qnn_delegate/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2'

OnePlus7Pro:/data/local/tmp $ ./benchmark_model --graph=/data/local/tmp/models/mobilenet_v1_1.0_224.tflite --external_delegate_path=/data/local/tmp/qnn_delegate/libQnnTFLiteDelegate.so --external_delegate_options='backend_type:htp;library_path:/data/local/tmp/qnn_delegate/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2'
INFO: STARTING!
INFO: Log parameter values verbosely: [0]
INFO: Graph: [/data/local/tmp/models/mobilenet_v1_1.0_224.tflite]
INFO: External delegate path: [/data/local/tmp/qnn_delegate/libQnnTFLiteDelegate.so]
INFO: External delegate options: [backend_type:htp;library_path:/data/local/tmp/qnn_delegate/libQnnHtp.so;skel_library_dir:/usr/lib/rfsa/adsp;htp_precision:0;htp_performance_mode:2]
INFO: Loaded model /data/local/tmp/models/mobilenet_v1_1.0_224.tflite
INFO: Initialized TensorFlow Lite runtime.
INFO: EXTERNAL delegate created.
ERROR: [QNN Delegate] Failed to create device_handle for Backend ID 6, error=1100
ERROR: Restored original execution plan after delegate application failure.
ERROR: Failed to apply EXTERNAL delegate.
ERROR: Benchmarking failed.
1|OnePlus7Pro:/data/local/tmp $

benchmarkQNN result.txt
I have tested it on Android phones equipped with Snapdragon 855 and Gen 3 chips, and the results were the same.

Could anyone tell me how to deal with this?

[gustavla]

Hi @4570235,

For future reference, since this is not an issue with a sample app from ai-hub-apps and instead with the QNN TFLite delegate directly, I would recommend reaching out through our Slack channel instead. That way colleagues of mine that know more can chime in.

I'll take a stab at this though. Is mobilenet_v1_1.0_224.tflite a fully quantized model? If not, then setting htp_precision to 0 (meaning quantized precision) will fail. Actually, I always set this to 1 since it works well for both quantized and unquantized models. If that's not the issue here, then perhaps you can post this question in our Slack channel and then I can try to get someone from the delegate team to look at it.

I will close this issue though since this doesn't track an issue with our sample apps but a general TFLite delegate use issue.

[4570235]

@gustavla thank you. I will go to the Slack channel.
By the way, I have tried setting htp_precision to 1 and a fully quantized model 'mobilenet_v1_1.0_224_quant.tflite', neither of them solves the problem.

[4570235]

To make the problem clearer, I used the latest benchmark_model program built by the TFLite official and the model 'quicksrnetsmall_quantized.tflite' in Qualcomm AI-Hub, and the results were the same.
I carefully read the test commands in the tutorial "run-qnn-delegate-using-the-external-delegate-interface". There is a detailed difference that there is a "skel_library_dir" parameter in the "--external_delegate_options" parameter. I don't know how to use this parameter and whether it is related to this problem.