trt_pose for Basler GigE Cameras

trt_pose is aimed at enabling real-time pose estimation on NVIDIA Jetson. This repository is focused on using trt_pose with a data stream from a Basler GigE camera. You may find it useful for other NVIDIA platforms as well. Currently the project includes

• Pre-trained models for human pose estimation capable of running in real time on Jetson Nano. This makes it easy to detect features like left_eye, left_elbow, right_ankle, etc.

• Training scripts to train on any keypoint task data in MSCOCO format. This means you can experiment with training trt_pose for keypoint detection tasks other than human pose.

To get started, follow the instructions below.

Getting Started

To get started with trt_pose, follow these steps.

Step 1 - Install Dependencies

1. Install PyTorch and Torchvision. To do this on NVIDIA Jetson, we recommend following this guide

2. Install torch2trt

   git clone https://github.com/NVIDIA-AI-IOT/torch2trt
   cd torch2trt
   pip3 install packages
   python3 setup.py install --plugins --user

3. Install other miscellaneous packages

   pip3 install tqdm cython pycocotools
   sudo apt-get install python3-matplotlib

Step 2 - Install trt_pose for GigE Basler Cameras

git clone https://github.com/EdoWhite/BaslerGigE-trt_pose.git
cd BaslerGigE-trt_pose
python3 setup.py install --user

Step 3 - Run the example notebook

We provide a couple of human pose estimation models pre-trained on the MSCOCO dataset. The throughput in FPS is shown for each platform

Model	Jetson Nano	Jetson Xavier	Weights
resnet18_baseline_att_224x224_A	22	251	download (81MB)
densenet121_baseline_att_256x256_B	12	101	download (84MB)

To run the live Jupyter Notebook demo on real-time camera input, follow these steps

1. Download the model weights using the link in the above table.

2. Place the downloaded weights in the tasks/human_pose directory

3. Open and follow the live_demo.ipynb notebook. The live demo works on images, videos, and live stream from a Basler GigE camera.

   You may need to modify the notebook, depending on which model you use

4. In the LiveCamera folder there is a python script that performs pose estimation on the stream coming from a Basler GigE Camera. To use the GigE camera, additional software should be installed. Refer to the official Basler documentation. Run the demo with:

   cd BaslerGigE-trt_pose/tasks/human_pose/LiveCamera/
   python3 live.py

See also

• trt_pose_hand - Real-time hand pose estimation based on trt_pose

• torch2trt - An easy to use PyTorch to TensorRT converter

• JetBot - An educational AI robot based on NVIDIA Jetson Nano

• JetRacer - An educational AI racecar using NVIDIA Jetson Nano

• JetCam - An easy to use Python camera interface for NVIDIA Jetson

References

The trt_pose model architectures listed above are inspired by the following works, but are not a direct replica. Please review the open-source code and configuration files in this repository for architecture details. If you have any questions feel free to reach out.

• Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

• Xiao, Bin, Haiping Wu, and Yichen Wei. "Simple baselines for human pose estimation and tracking." Proceedings of the European Conference on Computer Vision (ECCV). 2018.