How to get this working with Yolo v3-tiny?

[ROBYER1]

So I used AlexeyAB's Darknet fork to train a custom model using Yolo-v3-tiny making my own weights from scratch to make a detector for apples/oranges as a quick test.

I used this guide to convert my model to onnx
https://github.com/jkjung-avt/tensorrt_demos#demo-5-yolov4

Apparently there is an issue with yolov3 tiny import in Unity Barracuda 1.2.0 preview and above
Unity-Technologies/barracuda-release#122

I can't seem to get the model to work in your project, using Barracuda preview 1.2.0 and using my names file + modifying Detector.cs to read the input and one of the outputs.

Here is the link to my onnx file, some test images and the names file https://drive.google.com/file/d/1uY81Gh2Edh63R7G_6i5KI_oVLdMCuiem/view?usp=sharing

Any help or direction on why the Yolo v3 onnx model doesn't work would be great, I'm still investigating.

[ROBYER1]

The model itself in action in Darknet before conversion:

[ROBYER1]

After some investigation, I have found that the Yolo v3 tiny inputs/outputs are different to v2-tiny quite a bit which is why the v3 tiny model isn't working with Detector.cs

I have seen that Detector.cs follows this Microsoft C# example very closely which was made for Yolo-v2-tiny, however they have not updated or modified the sample to show how to parse a Yolo-v3-tiny model.

I am very interested in using the Yolo-v3-tiny model in this project if possible and can help but I am not very experienced in writing onnx model parsers in C#, I am however very proficient in Unity programming and AR development.

Please let me know if I can help update this repo for v3-tiny!

[derenlei]

Hi @ROBYER1, it would be great if you could help to support Yolo-v3-tiny for the detector!

Barracuda didn't support Yolo-v3 or v4 the last time I checked (September?).

Here are the Barracuda supported ONNX operators. I think it's related. https://docs.unity3d.com/Packages/[email protected]/manual/SupportedOperators.html

[ROBYER1]

Hi @ROBYER1, it would be great if you could help to support Yolo-v3-tiny for the detector!

Barracuda didn't support Yolo-v3 or v4 the last time I checked (September?).

Here are the Barracuda supported ONNX operators. I think it's related. https://docs.unity3d.com/Packages/[email protected]/manual/SupportedOperators.html

Hey, this was added in Barracuda 1.2.0 preview package. You can find the preview packages using the instructions here to un-hide them in package manager https://docs.unity3d.com/2020.1/Documentation/Manual/class-PackageManager.html

I am currently looking into this but I'm more experienced with Unity AR development and C# programming than I am with reading python or understanding how to get values in/out of Darknet Yolo models!

I got some interesting discussion going with Unity Barracuda team here about how they could use some examples of how to use these models in Unity and get inputs/outputs from them and hope to push forward with them on getting examples like this online and into people's hands to use.

Are you available if I need any pointers or assistance with upgrading this to Yolo v3 tiny? I'd really appreciate any help or tips as I'm still very new to ML in general.

[ROBYER1]

@derenlei I made some progress with v3, will push to my fork shortly.

You may need one of these to test unless you have your own v3-tiny ONNX already
Apple/Orange ONNX v3-tiny
COCO ONNX v3-tiny

I am using this asset made by some friends of mine to quickly test with simulated AR in the Editor, you may already have your own way of simulating the camera view in editor for your own testing as I saw you had a hidden canvas in the scene, but I couldn't figure out how to use it for testing so I used AR Sim. I have verified the values below I get when using my own test v3-tiny model for detecting Apples/Oranges against the behaviour I get with your v2 version using a COCO v2 tiny ONNX and your own food dataset v2-tiny ONNX.

I am getting the correct classes + predictions, however I barely ever see a bounding box, if I do it is a small line across the screen + the GetConfidence calculation in Detector.cs seems to be causing issues and I'm not sure what is going wrong with the 'Sigmoid' calculation I have left unchanged. I had to set the threshold very low to see any anchors placed and most of the time they were low like 5%.

You can see in #1 That I get the class Apple at 0.998, which sounds spot on

#2 Okay so as soon as we go to get the top score, the confidence value is always really low

#3 The bounding box dimensions are all minus values

[ROBYER1]

Merry Christmas! I got the top results coming in which means things are.. sort of working (woo!) but the values for confidence are wrong and the bounding boxes have minus values so I never see them, they are inverted or they are wildly off screen.

I had to set the threshold/minimum confidence to 0.001 to even see those results otherwise it never detects anything, and all top results are 0% confidence.

I'm really stuck at this point, but the results are amusing!

[ROBYER1]

I am using this converter for converting Darknet Yolo v3 tiny to ONNX https://github.com/jkjung-avt/tensorrt_demos

But there are some issues with the converted mAP accuracy of the onnx files vs the original darknet results
Issues to track:
jkjung-avt/tensorrt_demos#315
isarsoft/yolov4-triton-tensorrt#7

This converter may have better mAP results: https://github.com/zldrobit/onnx_tflite_yolov3

Or this one, I may try soon jkjung-avt/tensorrt_demos#237

Found it in this very helpful article: https://prashantdandriyal.medium.com/darknet-to-tensorflow-to-tensorflowlite-b783d55b106a

[derenlei]

Update tiny Yolo3 support in #12