Skip to content

Latest commit

 

History

History
49 lines (36 loc) · 3.11 KB

testing.md

File metadata and controls

49 lines (36 loc) · 3.11 KB

Testing

Unit tests

Unit tests are performed with catch2. The framework is automatically downloaded by cmake.

For each .cpp file, a file .test.cpp should be created. Each contributor is reponsible for the unit tests associated to the implemented contribution. Pushed code that does not provide enough unit tests, and does not pass these tests will not be merged.

Once the project is compiled, unit tests can be started with

make test

for Linux and directly from the menu Test/Run CTests for RM0 in MS Visual Studio.

Tests can also be run independently, executables are built during the compilation. They provide more detailed information.

These tests will be automatically performed when a merge is requested. The reference machine is a Linux AWS instance handled by InterDigital. A merge will not be accepted until all tests are passing. Please contact the software coordinator if you encounter any issue.

Metrics

PSNRs and bitrates are automatically computed at the test stage of the CI pipeline. Data are fetch from the mpeg content website, verified, and processed by python scripts. The file to be tested are listed in test/psnr_2k.json.

If you want to compute these metrics locally, you need to first download the data from mpeg content : MPEG-I/Part40-HapticSupport (right click on the left menu, Download). Password is provided by the AHG group chairman.

Then data integrity must be checked (md5 hash are compared to a reference list_files.md5). Run this command from the root of the repo :

python ./scripts/tools/check_data.py --data_dir ~/data --md5 ./test/list_files.md5

Finally metrics are computed with :

pytest ./scripts/test/integration_tests.py --autopad 
      --install_dir ~/install/ 
      --data_dir ~/data/Test/
      --junitxml=reports/report.xml -o junit_family="xunit1" 
      -n auto --csv reports/report.csv 
      --psnr_ref ./test/psnr_2k.json
      --csv-columns host,system,python_version,function,status,success,duration,properties_as_columns
      --save_encoder_dir ~/encoded_files

The pytest framework is used to run all the tests. The plugins xdist and csv are required. The dockerfile may be useful to setup a test environment. It is assumed that RM0 is compiled and installed (in ~/install/ for instance here).

The script produces a report.xml file directly ingested by the gitlab CI system, and a report.csv file that can be used to compute statistics. The argument psnr_ref may be removed to compute the metrics of all wav files in the data folder. The argument save_encoder_dir can be omitted if you don't need to save the encoded files.

If the PSNR of a single file have to be computed, the following script may be called :

python --autopad ./scripts/test/psnr.py <original_file.wav> <decompressed_file.wav>

The autopad option is optional and allows comparaison of two files with a different samples number. The bitrate can also be computed with the bitrate.py file.