First Robotics Competition (FRC) Team 2551
This repo descibes our team's setup for getting video images from the robot back to the driver station. Similar systems were used for the 2018, 2019 and 2021 competitions.
The goal is to allow the human driver to see from the robot's perspective and provide views that help with pathfinding, obstacle avoidance and help them see the robot's proximity to field elements and playing pieces (balls, gears, etc). The same video streams could potentially be used in other ways as an input to another computer vision system but that is beyond the scope of this document.
The rules for each FRC game determine many constraints. Carefully read and follow the rules. Common constraints include robot size and weight, robot control system (National Instruments RoboRio), power (12V battery), communications with driver station (WiFi).
The following hardware was used for Video Encoding:
- NVIDIA Jetson TX1 module
- Connect Tech Orbitty Carrier for NVIDIA Jetson TX1/TX2
- CTR Voltage Regulator Module (to supply constant voltage to the TX1 and carrier)
- 2 Microsoft LifeCam HD3000 USB Cameras (1280x720 @ 30fps)
- Dell PC running Windows 10 for the Driver's Station
While much of this code could potentially be used in other environments, the focus of this repo is on supporting the hardware listed above for this specific FRC application.
- GStreamer
- GStreamer plugin packages
- omxh264enc Video Encoder
- Linux for Tegra (L4T)
- Video4Linux2 (V4L2), specifically
- Attach the TX1 module to the Orbitty Carrier if it isn't already.
- Connect at least one LifeCam USB Camera to a USB port on the Orbitty.
- Optional - connect HDMI Display to the Orbitty for the linux desktop.
- Optional - connect USB Keyboard/Mouse with a USB hub to the Orbitty if needed.
- Connect the green power block on the Orbitty to 12V/2A on the Voltage Regulator Module. (Be careful with polarity)
Refer to 2022-02-13_history.txt in the root of this repo to see the full list of commands that have been executed on the TX1. Some of the important commands of note include:
Download and update the package information from all of the configured sources:
sudo apt update
Download any available updates and apply them to the out-dated packages installed:
sudo apt upgrade
Update the package index files to get the latest list of available packages in the repositories:
sudo apt-get update
Install the nano text editor:
sudo apt install nano
Install GStreamer tools:
sudo apt install gstreamer1.0-tools
Install Gstreamer Plugins:
sudo apt install gstreamer1.0-plugins-base gstreamer1.0-plugins-good
The "server" folder in this repo includes three shell scripts that can be copied to the TX1.
Also copy the twostreams.service to systemd as a service, which will automatically start on boot and restart if errors arise:
sudo cp twostreams.service /etc/systemd/system/
Enable the twostreams.service:
sudo systemctl enable twostreams.service
Check the status of the twostreams.service:
sudo systemctl status twostreams.service
Make the shell scripts executable:
chmod +x twostream.sh
chmod +x stream1.sh
chmod +x stream2.sh
Use tegrastats to check memory usage and processor usage for the Tegra-based TX1 to confirm the GPU is being used:
sudo ./tegrastats
Windows 10 Driver's Station IP: 10.25.51.20
Linux4Tegra (Ubuntu) TX1's IP: 10.25.51.15
Ports 5803 and 5804
Typical Ubuntu/L4T networking commands for reference:
Show the current configuration including the assigned IP address for the eth0 interface:
ifconfig | grep eth0
Configure the network interface by editing the interfaces file with the nano text editor, then reboot to apply the changes:
sudo nano /etc/network/interfaces
sudo reboot
Modify the kernel parameters for networking, then restart networking to apply the changes:
sudo nano /etc/sysctl.conf
sudo service networking restart
v4l2-ctl Brightness settings:
v4l2-ctl [-h] [-d <dev>] [many other options]
-d, --device <dev>
Use device <dev> as the V4L2 device. if <dev> starts with a digit, then
/dev/video<dev> is used Otherwise if -z was specified earlier, then <dev> is the
entity name or interface ID (if prefixed with 0x) as found in the topology of the
media device with the bus info string as specified by the -z option.
-c, --set-ctrl <ctrl>=<val>[,<ctrl>=<val>...]
Set the value of the controls [VIDIOC_S_EXT_CTRLS].
For reference, here's the output of v4l2-ctl -d /dev/video1 -l (helpful for min/max/defaults)
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=128 value=128
saturation (int) : min=0 max=255 step=1 default=128 value=128
white_balance_temperature_auto (bool) : default=1 value=1
gain (int) : min=0 max=255 step=1 default=0 value=0
power_line_frequency (menu) : min=0 max=2 default=2 value=2
white_balance_temperature (int) : min=2000 max=6500 step=1 default=4000 value=2594 flags=inactive
sharpness (int) : min=0 max=255 step=1 default=128 value=128
backlight_compensation (int) : min=0 max=1 step=1 default=0 value=0
exposure_auto (menu) : min=0 max=3 default=3 value=1
exposure_absolute (int) : min=3 max=2047 step=1 default=250 value=333
exposure_auto_priority (bool) : default=0 value=1
pan_absolute (int) : min=-36000 max=36000 step=3600 default=0 value=0
tilt_absolute (int) : min=-36000 max=36000 step=3600 default=0 value=0
focus_absolute (int) : min=0 max=250 step=5 default=0 value=125
focus_auto (bool) : default=1 value=0
zoom_absolute (int) : min=100 max=500 step=1 default=100 value=100
The Accelerated GStreamer User Guide (pdf) may be helpful. It includes other examples for the use of the omxh264enc Video Encoder with the gst-launch-1.0 command.