Skip to content

Collection of tools to manage ROSbag recordings data from AV vehicle

Notifications You must be signed in to change notification settings

ipab-rad/tartan_forklift

Repository files navigation

tartan_dataset_management

Collection of tools to manage ROSbag recordings data from AV vehicle

Metadata Generator Usage

This script generates metadata for ROSbag MCAP files. The metadata is compiled into a resources.json file that complies with the EIDF requirements

Features

  • Reads MCAP files and extracts metadata such as:
    • Duration of the log
    • Topics and message counts
    • File size
    • File hash (MD5)
  • Generates a JSON file (resources.json) with metadata for all MCAP files in a given directory.
  • Metadata includes:
    • File name
    • Identifier
    • Description
    • Format
    • License
    • Size
    • Hash
    • Issued date
    • Modified date
    • Duration
    • Topics and message counts

Usage

1. Setup

Ensure all dependencies are installed. You can use the following command to install required packages:

pip install mcap

2. Running the Script

To generate the metadata JSON file, follow these steps:

  • Place all your MCAP files in a directory.

  • The default directory is /recorded_datasets/edinburgh

  • Run the script:

    python metadata_generator.py

If you want to generate metadata for files in a specified path, run the script:

python metadata_generator.py -p path/to/file

3. Output

The script will generate a resources.json file in the specified directory. This JSON file will contain metadata for each MCAP file in the directory.

Upload Vehicle Data

This script automates the process of uploading rosbags from the IPAB-RAD autonomous vehicle server to a cloud instance within the EIDF (Edinburgh International Data Facility) infrastructure. It streamlines data collection and transfer by first compressing the rosbags using the MCAP CLI, and then uploading the compressed files. This ensures efficient handling and storage of large datasets generated by vehicle sensors.

1. Dependencies

Host Machine (Cloud instance)

  • Install iperf3:
    sudo apt-get install iperf3
  • Set up SSH key authentication:
    • Generate an SSH key:
      ssh-keygen -t rsa -b 2048
    • Copy the SSH key to the remote machine:
      ssh-copy-id username@hostname_ip

Remote Machine (Vehicle Server)

  • Install iperf3:
    sudo apt-get install iperf3
  • Install MCAP CLI 0.0.47 for compressing rosbags:
    wget -O $HOME/mcap https://github.com/foxglove/mcap/releases/download/releases%2Fmcap-cli%2Fv0.0.47/mcap-linux-amd64 && chmod +x $HOME/mcap

2. Usage

To execute the script, use the following command from the host machine:

python3 upload_vehicle_data.py -config <path_to_yaml_config>

or

python3 upload_vehicle_data.py -c <path_to_yaml_config>

Replace <path_to_yaml_config> with the path to your YAML configuration file. Use the -d or --debug option to enable detailed information:

python3 upload_vehicle_data.py -c <path_to_yaml_config> -d

3. YAML Parameters

  • remote_user (str): Username for the remote machine.
  • remote_ip (str): IP address of the remote machine (default: 129.215.117.104).
  • remote_directory (str): Directory on the remote machine containing rosbags (default: /mnt/mydrive/rosbags).
  • cloud_upload_directory (str): Remote directory for uploading compressed files (default: /mnt/vdb/data).
  • clean_up (bool): Whether to delete all rosbags from the vehicle machine after uploading.
  • upload_attempts (int): Number of attempts to upload each rosbag file to the cloud host (default: 3).
  • mcap_path (str): Path to the mcap CLI binary. Found using which mcap if installed.
  • parallel_processes (int): Number of parallel processes for compression and upload.

Example YAML configuration:

remote_user: "username"
remote_ip: "129.215.117.104"
remote_directory: "/mnt/mydrive/rosbags"
cloud_upload_directory: "/mnt/vdb/data"
clean_up: false
upload_attempts: 3
mcap_path: "$HOME/mcap"
parallel_processes: 1

4. Script Workflow

  • Measures available bandwidth using iperf3.
  • Creates a remote temporary directory for storing compressed .mcap files.
  • Lists all .mcap files in the specified directory.
  • Displays the total number of files, their combined size, and the estimated upload time based on the measured bandwidth.
  • Prompts for user confirmation before proceeding with the upload.
  • Compresses each .mcap file using mcap CLI with zstd level 2.
  • Uploads each compressed file to the remote server using rsync.
  • Verifies the integrity of each uploaded file.
  • Deletes the original and compressed files from the vehicle after successful upload and verification, if clean_up is set to true.

5. Logging

The script logs its activities to upload_vehicle_data_<timestamp>.log. This log file includes:

  • Information about the processed files.
  • Bandwidth measurements.
  • Any errors encountered during the process.

ROS2 Bag Merging Script

This script automates the merging of ROS2 bag files using the ros2 bag convert command.

The merging is based on a .yaml file that complies with rosbag2 guidelines.

Prerequisites

Before running the Python script, ensure that the Docker container is set up by running the dev.sh script:

./dev.sh

Note: dev.sh will automatically mount the /mnt/vdb/data directory as the rosbags directory in the container. This is the current default SSD location on the EIDF cloud machine. Use the -p flag to specify a different directory.

Usage

Once the Docker container is running, you can execute the Python script with the following command:

merge_rosbags.py --input <input_directory> --config <config_file.yaml> --range <start:end>

Arguments:

  • --input <input_directory>: The directory containing .mcap files to be merged.
  • --config <config_file.yaml>: Path to the YAML configuration file used for the merge.
    • Please refer to the rosbag_util directory and rosbag2 Converting bags README for further reference.
  • --range <start:end> (optional): Range of rosbag split indices to process, in the format start:end.

Example:

If we have the following directory structure:

└── rosbags
      ├── 2024_10_24-14_15_33_haymarket_1
      │   ├── 2024_10_24-14_15_33_haymarket_1_0.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_1.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_2.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_3.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_4.mcap
      ...

We can run the script as:

merge_rosbags.py --input ./rosbags/2024_10_24-14_15_33_haymarket_1 --range 0:2 --config ./rosbag_util/mapping_merge_params.yaml

# Or omit the range to merge all available rosbags
merge_rosbags.py --input ./rosbags/2024_10_24-14_15_33_haymarket_1 --config ./rosbag_util/mapping_merge_params.yaml

The resulting merged .mcap file will be saved within the specified directory under a sub-directory named the same as the value of the uri parameter defined in the YAML file. The file name will be in the format <current_rosbag_name>_<uri>_<from-to>. If no range is specified, the name will be <current_rosbag_name>_<suffix>.

└── rosbags
      ├── 2024_10_24-14_15_33_haymarket_1
      │   ├── 2024_10_24-14_15_33_haymarket_1_0.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_1.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_2.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_3.mcap
      │   ├── 2024_10_24-14_15_33_haymarket_1_4.mcap
      │   └── mapping
              ├── 2024_10_24-14_15_33_haymarket_mapping_0-2.mcap
              └── 2024_10_24-14_15_33_haymarket_mapping.mcap # Larger file
      ...

About

Collection of tools to manage ROSbag recordings data from AV vehicle

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •