This document provides best practices and guidelines for testing your project. It will help you to ensure that your code is reliable, robust, and maintainable, it also provides a comprehensive understanding of how to structure, write, and maintain effective tests to ensure your code remains reliable and robust.
Testing is a crucial part of software development. Well-written tests ensure that your code fulfills your requirements and behaves as expected. When a project grows, it becomes harder to add new features or fix bugs without breaking existing functionalities. As manual testing is time-consuming and error-prone, adding tests will ensure that you can make changes to your codebase without and rapidly check everything will working as expected.
Most of machine learning projects are written in Python, therefore, we
it is natural to use a high featured testing framework like pytest
.
It is easy to use, supports fixtures for managing test setup and teardown,
and has a rich ecosystem of plugins for extending functionality.
Organize your project to separate the test code from the main codebase. A typical structure for an API ML code might look like this:
ai4life/
│
├── .sqa
│ ├── config.yml <-- Software Quality Assurance configuration
│ ├── docker-compose.yml <-- Compose file for integration configuration
│ └── dockerfile.py <-- Instructions to build testing container
|
├── api
│ ├── __init__.py <-- API importable objects
│ └── ... <-- See base README.md for more info
│
├── (<your_model_source>)* # Optional
│ ├── __init__.py <-- Model importable objects
│ ├── data_processing.py <-- Data preprocessing functions
│ ├── model_training.py <-- Model training functions
│ └── model_prediction.py <-- Model prediction functions
│
├── tests
│ ├── configurations <-- Configuration files to test
| ├── data <-- Data to test and use for testing
│ ├── test_metadata <-- Tests for metadata endpoints
│ ├── test_predictions <-- Tests for prediction endpoints
│ ├── test_training <-- Tests for training endpoints
│ └── conftest.py <-- Pytest fixtures
│
├── Jenkinsfile <-- CICD pipeline configuration for Jenkins
├── pyproject.toml <-- Project configuration file
├── requirements-test.txt <-- Testing requirements
├── tox.ini <-- Standardization testing config
└── ... <-- See base README.md for more info
This structure contains a lot of components, but we will introduce them gradually in the following sections. Model source* is optional for this template. As its main purpose is to provide a model API, in some cases the model source is not included in the project but installed from a package or as a submodule.
- Use Small, Representative Datasets: Create small datasets that represent the variety of cases your model will encounter. For example a few Mb model or a few images to use as input to your load, predict and train functions.
- Avoid Using Real Data and Used Secrets: Use synthetic or anonymized data for testing to ensure privacy and reproducibility. When you require some credential or passwords, do not use the same you will use in production, use different ones for testing.
- Cover All Critical Paths: Test writing is time-consuming, so focus on testing the critical paths, reach the highest coverage possible, but keep in mind that 100% coverage will not guarantee that your code is bug-free.
- Edge Cases and Invalid Inputs: Test some invalid inputs to ensure your code handles them gracefully.
- Leverage Fixtures: Use fixtures to manage setup and teardown of test resources (e.g., datasets, model instances). Set the highest scope as possible to reduce testing time.
- Modular Fixtures:
Create reusable fixtures in your
conftest.py
files to avoid code duplication. Divide them between folders to simplify the maintenance. - Error vs Failures: Use your fixtures to execute code and your test files to assert the results. This will improve the readability of your tests and make it easier to understand what is failing. An error means your code failed, a failure means your code returned an unexpected result.
- Isolate Tests:
Only integration tests should test the interaction between different
components. When doing unit tests, test only the code you own, not the
libraries you are using. In most cases you do not need to test a real model
but is enough to mock one with its
predict
andfit
methods. - Mock using Specs:
Utilize libraries such as
unittest.mock
to create mock objects. When you mock an object or function, useautospec=True
to ensure that the test fails if you call the mock with unexpected arguments. I recommend to read Python Unittest Mock Autospeccing.
- Include tox for validation: You do not need to run always tests using tox, but before pushing your code to the repository, you should run tox to ensure that your code is working in different environments and machines or Python versions.
- Use src folder when developing an installable:
If your code is planned to be installed, use a
src
folder to store your code. Python adds your working directory to the path, andimport
command might succeed on your development machine but fail when installed. - Split test and software requirements: Keep your test requirements separate from your software requirements. This will make it easier to manage dependencies and ensure that your production code does not include testing libraries.
- Configure pytest at pyproject.toml:
Use
pyproject.toml
to configurepytest
and plugins. This will make your project more maintainable and easier to share with others.
- Check code quality and style:
Use
flake8
or other tools to ensure your code follows the programming recommendations and style guides. This will make your code more readable and maintainable. - Check for security vulnerabilities:
Use
bandit
or other tools to check for security vulnerabilities in your code. This will help you to identify and fix potential security issues.
All your tests should be placed in the tests
folder and start with test_
meanwhile you should try to locate your fixtures in the conftest.py
file.
The command pytest
automatically discovers and run all files that match
this pattern, so it should be easy to start and edit the project tests.
Specific to this template, you have a folder api
with the code expected to
be tested. Some projects include a "model source code", which contains the
code to train and predict the model. This template includes basic tests for
testing the api
methods, which expect to make use of your model code and
therefore test it indirectly.
As explained in the best practices, you should restrict the tests in your
repository to the code you own. This means that you should not test an
external library of model, specially if that one already includes tests
of its own. For example, you might be building an API for yolov
model,
in that case, you are not expected to have yolov
source code in the
repository as it is already tested in its own repository.
To run your tests in your current environment, simply execute:
python -m pytest tests
Some IDEs like Visual Studio Code have built-in support for pytest.
For more advanced testing, you can use tox
to run tests in multiple
environments. To run tests with tox
, execute:
python -m tox
Tox will run tests in multiple environments, such as different Python versions
in this template, tox include custom environments to test your code quality
and security with flake8
and bandit
.
CICD is the last step on the automation of the testing process. This template
uses Jenkins and SQAaaS to automate the
testing process. You can use the provided .sqa
folder to configure your own
testing pipeline for project.
Jenkinsfile in this project is configured to use the Jenkins server and tools at
jenkins.services.ai4os.eu.
We do not recommend to edit the Jenkinsfile, but you add your configuration
files to the .sqa
folder to customize your testing pipeline.