Add yourself to the "authors" section in the pyproject.toml
file to ensure proper documentation.
.
├── ixmp4
│ ├── cli # cli
│ ├── conf # configuration module, loads settings etc.
│ ├── core # contains the facade layer for the python API
│ ├── data
│ │ ├── abstract # ABCs for data source models and repositories
│ │ ├── api # data source implementation for the web api
│ │ ├── backend # data source backends
│ │ └── db # data source implementation for databases (sqlalchemy)
│ ├── db # database management
│ ├── server # web application server
│ └── rest # REST endpoints
├── run # runtime artifacts
└── tests # tests
When using ixmp4 via the python API we would traverse this diagram from left to right.
Platform Backend Server Backend
│ ┌────────────┐ ┌───────────┐ ┌─ │ ┌──────────┐ ┌───────────┐ ─┐ │ ┌─┐
P │ │ │ │ │ │ │ │ │ │ │ │ S │ │ │
y │ │ ┌────────┐ │ │ ┌───────┐ │ │ R │ │ ┌──────┐ │ │ ┌───────┐ │ │ Q │ │D│
t │ │ │ │ │ │ │ │ │ ┌─┘ E │ │ │Endp. │ │ │ │ │ │ └─┐ L │ │a│
h │ │ │Facade │ │ │ │Model │ │ │ S │ │ └──────┘ │ │ │Model │ │ │ A │ │t│
o │ │ └────────┘ │ │ ├───────┤ │ │ T │ │ │ │ ├───────┤ │ │ l │ │a│
n │ │ │ │ ├───────┤ │ │ │ │ ┌──────┐ │ │ ├───────┤ │ │ c │ │b│
│ │ ... │ │ │ │ │ │ A │ │ │Endp. │ │ │ │ │ │ │ h │ │a│
A │ │ │ │ │Repo. │ │ └─┐ P │ │ └──────┘ │ │ │Repo. │ │ ┌─┘ e │ │s│
P │ │ │ │ └───────┘ │ │ I │ │ │ │ └───────┘ │ │ m │ │e│
I │ │ │ │ ... │ │ │ │ ... │ │ ... │ │ y │ │ │
│ └────────────┘ └───────────┘ └─ │ └──────────┘ └───────────┘ ─┘ │ └─┘
ixmp4.core ixmp4.data ixmp4.server ixmp4.data
Note that the bracketed part of the diagram is only in use when using a web-based platform (using ixmp4 over the REST API). Note also that a REST SDK in another programming language would have to implement only the components before the bracketed part of the diagram (ixmp4.data.api
+ optionally a facade layer).
Overall both the "facade" layer and the "data source" layer are split into "models" (representing a row in a database or a json object) and "repositories" (representing a database table or a collection of REST endpoints) which manage these models.
Contains the user-facing python API ("facade layer") and user documentation.
The module ixmp4.data.abstract
contains an abstract definition of all data and functionality used by the facade layer of ixmp4. This module also contains internal documentation.
The modules ixmp4.data.db
and ixmp4.data.api
contain data source implementations for the abstract "interface". The database implementation uses sqlalchemy
for database interaction and the API implementation talks to ixmp4.server
via a web API.
Uses fastapi
as a micro-framework for the REST API.
# Install the following package
sudo apt-get install build-essential
sudo apt install python3-dev
# Install Poetry, minimum version >=1.2 required
curl -sSL https://install.python-poetry.org | python -
# You may have to reinitialize your shell at this point.
source ~/.bashrc
# Activate in-project virtualenvs
poetry config virtualenvs.in-project true
# Add dynamic versioning plugin
poetry self add "poetry-dynamic-versioning[plugin]"
# Install dependencies
# (using "--with docs" if docs dependencies should be installed as well)
poetry install --with docs,server,dev
# Activate virtual environment
poetry shell
# Copy the template environment configuration
cp template.env .env
Developing ixmp4 requires poetry >= 1.2
.
If you already have a previous version of poetry installed you will need to update. The first step is removing the old poetry version:
curl -sSL https://install.python-poetry.org | python3 - --uninstall
after that, the latest poetry version can be installed using:
curl -sSL https://install.python-poetry.org | python3 -
details can be found here in the poetry docs: https://python-poetry.org/docs/#installation.
When updating dependencies it can happen that a conflict between the current and the target poetry.lock file occurs. In this case the following steps should be taken to resolve the conflict.
- Do not attempt to manually resolve in the GitHub web interface.
- Instead checkout the target branch locally and merge into your branch:
git checkout main
git pull origin main
git checkout my-branch
git merge main
- After the last step you'll have a merge conflict in poetry.lock.
- Instead of resolving the conflict, directly checkout the one from main and rewrite it:
# Get poetry.lock to look like it does in master
git checkout main poetry.lock
# Rewrite the lock file
poetry lock --no-update
- After that simply add poetry.lock to mark the conflict as resolved and commit to finalize the merge:
git add poetry.lock
git commit
# and most likely needed
poetry install
(Taken from https://www.peterbe.com/plog/how-to-resolve-a-git-conflict-in-poetry.lock)
This project uses sqlalchemy and alembic for database management. As of now only SQLite databases are supported.
ixmp4 itself provides a CLI for interacting with the platforms (i.e. databases) it uses:
# list all existing databases
ixmp4 platforms list
# run all migrations on all databases in your platforms.toml file
ixmp4 platforms upgrade
# create a new sqlite database
ixmp4 platforms add <platform-name>
# delete a platform
ixmp4 platforms delete <database-name>
In development mode additional commands are available:
# set the revision hash of a database without running migrations
ixmp4 platforms stamp <revision-hash>
There is a development database at run/db.sqlite
which is used for generating migrations, nothing else.
It can be manipulated with alembic directly using these commands:
# run all migrations until the current state is reached
alembic upgrade head
# run one migration forward
alembic upgrade +1
# run one migration backward
alembic downgrade -1
# autogenerate new migration (please choose a descriptive change message)
alembic revision -m "<message>" --autogenerate
You will have to run all migrations before being able to create new ones in the development database.
Be sure to run ruff
on newly created migrations!
Run tests with the CLI for a default pytest
configuration (coverage report, etc.):
ixmp4 test
Or use pytest
directly:
py.test
In order to run the local tests with PostgreSQL you'll need to have a local instance of this database running. The easiest way to do this using a docker container.
Using the official postgres
image is recommended.
Get the latest version on you local machine using (having docker installed):
docker pull postgres
and run the container with:
docker run -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=test -p 5432:5432 -d postgres
Please note that you'll have to wait for a few seconds for the databases to be up and running.
The tests/docker-compose.yml
file might help you accomplish all of this with a single
command for your convenience.
docker-compose -f tests/docker-compose.yml up
The tests contain two types of benchmarks:
-
test_benchmarks.py
Simple throughput benchmarks for a few operations and pre-conditions. Check the test docstrings for more information. These tests require a long-format iamc file at./tests/test-data/iamc-test-data_annual_big.csv
-
test_benchmark_filters.py
Benchmarks the retrieval of iamc data with different filters. Primarily used for testing query speed.
Some tests will output profiler information to the .profiles/
directory (using the
profiled
fixture). You can analyze these using snakeviz
. For example:
snakeviz .profiles/test_add_datapoints_full_benchmark\[test_api_pgsql_mp\].prof
Run the web api with:
ixmp4 server start
This will start ixmp4's asgi server. Check http://127.0.0.1:9000/v1/<platform>/docs
.
To build:
docker build -t ixmp4:latest .
Optionally, supply POETRY_OPTS:
docker build --build-arg POETRY_OPTS="--with docs,dev" -t ixmp4-docs:latest .
On release, new images will be built akin to this:
docker build --build-arg POETRY_OPTS="--with server" -t ixmp4-server:latest .
Use the image like this in a docker-compose file:
version: "3"
services:
ixmp4_server:
image: registry.iiasa.ac.at/ixmp4/ixmp4-server:latest
# To change the amount of workers in a single container
# override the ixmp4 cli command:
command:
- ixmp4
- server
- start
- --host=0.0.0.0
- --port=9000
- --workers=2
volumes:
- ./run:/opt/ixmp4/run
env_file:
- ./.env
deploy:
mode: replicated
replicas: 2
ports:
- 9000-9001:9000
This configurations spawns two containers at ports 9000
and 9001
with 2 workers each.
This package uses the poetry-dynamic-versioning plugin to generate a version number either out of a tag or a current revision.
For this reason the version number is intentionally set to 0.0.0 in pyproject.toml
.
It is overwritten on the fly by the poetry-dynamic-versioning plugin.
-
Before releasing, check that the "pytest" GitHub action on the current "main" branch passes. Address any failures before releasing.
-
Test on your local machine if the build runs by running
python -m build --sdist --wheel --outdir dist/
. Fix any packaging issues or errors by creating a PR. -
Tag the release candidate (RC) version on the main branch as vrc and push to upstream:
git tag v<release version>rc<N>>
git push upstream v<release version>rc<N>
- Check that the GitHub action "Publish ixmp4" was executed correctly and that the release candidate was successfully uploaded to TestPyPI. The address will be https://test.pypi.org/project/ixmp4/rc. E.g.: https://test.pypi.org/project/ixmp4/0.2.0rc1/
- Visit https://github.com/iiasa/ixmp4/releases and mark the new release by creating the tag and release simultaneously. The name of the tag is v (without the rc).
- Check that the "Publish to PyPI and TestPyPI" GitHub action passed and that the distributions are published on https://pypi.org/project/ixmp4/ .
Contributions to the code are always welcome! Please make sure your code follows our code style so that the style is consistent. Each PR will be checked by a Code Quality test that examines compliance with ruff and mypy.
We use pre-commit to check the code style. You can install
pre-commit locally by installing ixmp4 with the optional dev
group. Running
pre-commit install
will set pre-commit up to run on every git commit
. Per default, pre-commit will run
on changed files, but if you want to run it on all files, you can run
pre-commit run --all-files
If you only want certain hooks to run, choose from ruff
and mypy
as
hook-ids
and run
pre-commit run <hook-ids> --all-files
Whether you run pre-commit locally or see it on your PR for the first time, the checks are the same. You can, of course, run the code style tools manually. From within the ixmp4 directory, this would look similar to this:
mypy .
ruff check .
ruff format .
# Or to enable ruff's automic fixes
ruff check --fix .
However, it is easy to forget running these commands manually. Therefore, we recommend
setting your editor up to run at least ruff
automatically whenever you hit save
. A few minutes of configuration will save you time
and nerves later on.