Generate heap benchmarks automatically (#376)

matth-x · Oct 13, 2024 · 942fb88 · 942fb88
1 parent 6cb5066
commit 942fb88
Show file tree

Hide file tree

Showing 4 changed files with 500 additions and 7 deletions.
diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml
@@ -2,7 +2,7 @@
 # Copyright Matthias Akstaller 2019 - 2024
 # MIT License
 
-name: documentation 
+name: Documentation
 on:
   push:
     branches:
@@ -11,8 +11,92 @@ on:
 
 permissions:
   contents: write
+
 jobs:
+  build_simulator:
+    name: Build Simulator
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v4
+      with:
+        python-version: 3.x
+    - uses: actions/cache@v2
+      with:
+        key: ${{ github.ref }}
+        path: .cache
+    - name: Get build tools
+      run: |
+        sudo apt update
+        sudo apt install cmake libssl-dev build-essential
+    - name: Checkout Simulator
+      uses: actions/checkout@v3
+      with:
+        repository: matth-x/MicroOcppSimulator
+        path: MicroOcppSimulator
+        ref: feature/remote-api
+        submodules: 'recursive'
+    - name: Clean MicroOcpp submodule
+      run: |
+        rm -rf MicroOcppSimulator/lib/MicroOcpp
+    - name: Checkout MicroOcpp submodule
+      uses: actions/checkout@v3
+      with:
+        path: MicroOcppSimulator/lib/MicroOcpp
+    - name: Generate CMake files
+      run: cmake -S ./MicroOcppSimulator -B ./MicroOcppSimulator/build -DCMAKE_CXX_FLAGS="-DMO_OVERRIDE_ALLOCATION=1 -DMO_ENABLE_HEAP_PROFILER=1"
+    - name: Compile
+      run: cmake --build ./MicroOcppSimulator/build -j 32 --target mo_simulator
+    - name: Upload Simulator executable
+      uses: actions/upload-artifact@v4
+      with:
+        name: Simulator executable
+        path: |
+          MicroOcppSimulator/build/mo_simulator
+          MicroOcppSimulator/public/bundle.html.gz
+        if-no-files-found: error
+        retention-days: 1
+
+  measure_heap:
+    needs: build_simulator
+    name: Heap measurements
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v4
+      with:
+        python-version: 3.x
+    - name: Install Python dependencies
+      run: pip install requests paramiko pandas
+    - name: Get Simulator
+      uses: actions/download-artifact@v4
+      with:
+        name: Simulator executable
+        path: MicroOcppSimulator
+    - name: Measure heap and create reports
+      run: |
+        mkdir -p docs/assets/tables
+        python tests/benchmarks/scripts/measure_heap.py
+      env:
+        TEST_DRIVER_URL: ${{ secrets.TEST_DRIVER_URL }}
+        TEST_DRIVER_CONFIG: ${{ secrets.TEST_DRIVER_CONFIG }}
+        TEST_DRIVER_KEY: ${{ secrets.TEST_DRIVER_KEY }}
+        MO_SIM_CONFIG: ${{ secrets.MO_SIM_CONFIG }}
+        MO_SIM_OCPP_SERVER: ${{ secrets.MO_SIM_OCPP_SERVER }}
+        MO_SIM_API_CERT: ${{ secrets.MO_SIM_API_CERT }}
+        MO_SIM_API_KEY: ${{ secrets.MO_SIM_API_KEY }}
+        MO_SIM_API_CONFIG: ${{ secrets.MO_SIM_API_CONFIG }}
+        SSH_LOCAL_PRIV: ${{ secrets.SSH_LOCAL_PRIV }}
+        SSH_HOST_PUB: ${{ secrets.SSH_HOST_PUB }}
+    - name: Upload reports
+      uses: actions/upload-artifact@v4
+      with:
+        name: Memory usage reports CSV
+        path: docs/assets/tables
+        if-no-files-found: error
+
   build_firmware_size:
+    name: Build firmware
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
@@ -51,6 +135,7 @@ jobs:
 
   evaluate_firmware:
     needs: build_firmware_size
+    name: Static firmware analysis
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
@@ -87,8 +172,7 @@ jobs:
         path: firmware
     - name: Run bloaty
       run: |
-        mkdir docs/assets
-        mkdir docs/assets/tables
+        mkdir -p docs/assets/tables
         tools/bloaty/build/bloaty firmware/firmware_v16.elf  -d compileunits --csv -n 0 > docs/assets/tables/bloaty_v16.csv
         tools/bloaty/build/bloaty firmware/firmware_v201.elf -d compileunits --csv -n 0 > docs/assets/tables/bloaty_v201.csv
     - name: Evaluate and create reports
@@ -101,7 +185,8 @@ jobs:
         if-no-files-found: error
 
   deploy:
-    needs: evaluate_firmware
+    needs: [evaluate_firmware, measure_heap]
+    name: Deploy docs
     runs-on: ubuntu-latest
     if: github.ref == 'refs/heads/main'
     steps:
@@ -120,5 +205,10 @@ jobs:
       with:
         name: Firmware size reports CSV
         path: docs/assets/tables
+    - name: Get memory occupation reports
+      uses: actions/download-artifact@v4
+      with:
+        name: Memory usage reports CSV
+        path: docs/assets/tables
     - name: Run mkdocs
       run: mkdocs gh-deploy --force
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -25,7 +25,7 @@
 - Support for TransactionMessageAttempts/-RetryInterval ([#345](https://github.com/matth-x/MicroOcpp/pull/345))
 - Heap profiler and custom allocator support ([#350](https://github.com/matth-x/MicroOcpp/pull/350))
 - Migration of persistent storage ([#355](https://github.com/matth-x/MicroOcpp/pull/355))
-- Benchmarks pipeline ([#369](https://github.com/matth-x/MicroOcpp/pull/369))
+- Benchmarks pipeline ([#369](https://github.com/matth-x/MicroOcpp/pull/369), [#376](https://github.com/matth-x/MicroOcpp/pull/376))
 - MeterValues port for OCPP 2.0.1 ([#371](https://github.com/matth-x/MicroOcpp/pull/371))
 - UnlockConnector port for OCPP 2.0.1 ([#371](https://github.com/matth-x/MicroOcpp/pull/371))
 - More APIs ported to OCPP 2.0.1 ([#371](https://github.com/matth-x/MicroOcpp/pull/371))

diff --git a/docs/benchmarks.md b/docs/benchmarks.md
@@ -34,14 +34,26 @@ The following table shows the cumulated size of the objects files per module. Th
 
 {{ read_csv('modules_v201.csv') }}
 
+## Memory usage
+
+MicroOCPP uses the heap memory to process incoming messages, maintain the device model and create outgoing OCPP messages. The total heap usage should remain low enough to not risk a heap depletion which would not only affect the OCPP module, but the whole controller, because heap memory is typically shared on microcontrollers. To assess the heap usage of MicroOCPP, a test suite runs a variety of simulated charger use cases and measures the maximum occupied memory. Then, the maximum observed value is considered as the memory requirement of MicroOCPP.
+
+Another important figure is the base level which is much closer to the average heap usage. The total heap usage consists of a base level and a dynamic part. Some memory objects are only initialized once during startup or as the device model is populated (e.g. Charging Schedules) and therefore belong to the base which changes only slowly over time. In contrast, objects for the JSON parsing and serialization and the internal execution of the operations are highly dynamic as they are instantiated for one operation and freed again after completion of the action. If the firmware contains multiple components besides MicroOCPP with this usage pattern, then the average total memory occupation of the device RAM is even closer to the base levels of the individual components.
+
+The following table shows the dynamic heap usage for a variety of test cases, followed by the base level and resulting maximum memory occupation of MicroOCPP. At the time being, the measurements are limited to only OCPP 2.0.1 and a narrow set of test cases. They will be gradually extended over time.
+
+**Table 3: Memory usage per use case and total**
+
+{{ read_csv('heap_v201.csv') }}
+
 ## Full data sets
 
 This section contains the raw data which is the basis for the evaluations above.
 
-**Table 3: All compilation units for OCPP 1.6 firmware**
+**Table 4: All compilation units for OCPP 1.6 firmware**
 
 {{ read_csv('compile_units_v16.csv') }}
 
-**Table 4: All compilation units for OCPP 2.0.1 firmware**
+**Table 5: All compilation units for OCPP 2.0.1 firmware**
 
 {{ read_csv('compile_units_v201.csv') }}