Failure to run models on ovms binary due to Memory Fault

[adarshan-intel]

Description

Please read gramineproject/gsc#229 for context

Downloaded OpenVINO Model Server on bare metal using the instructions from OpenVINO documentation.

Steps to Reproduce:

1. Download precompiled package:

   wget https://github.com/openvinotoolkit/model_server/releases/download/v2024.5/ovms_ubuntu22.tar.gz
   tar -xzvf ovms_ubuntu22.tar.gz

2. Or build it yourself:

   # Clone the model server repository
   git clone https://github.com/openvinotoolkit/model_server
   cd model_server
   # Build docker images (the binary is one of the artifacts)
   make docker_build PYTHON_DISABLE=1 RUN_TESTS=0
   # Unpack the package
   tar -xzvf dist/ubuntu22/ovms.tar.gz

3. Install required libraries:

   sudo apt update -y && apt install -y libxml2 curl

4. Set path to the libraries and add binary to the PATH:

   export LD_LIBRARY_PATH=${PWD}/ovms/lib
   export PATH=$PATH:${PWD}/ovms/bin

5. Create a Makefile and manifest.template file. (attached in logs)

6. Run the script runs.sh to execute the file. (attached in logs)

Issue:

Encountered a memory issue. Logs indicate the following error:

(libos_signal.c:351:memfault_upcall) [P1:T1:ovms] debug: memory fault at 0x73e13ed194f8 (IP = 0x73e1266fed2d)

Notes:

• Initially thought this was an issue with GSC, but it is also reproducible on core Gramine. So raising an issue here as suggested by Failure to run encrypted models on openvino/model_server:latest due to Memory Fault gsc#229 (comment)
• Both gramine-direct and gramine-sgx exhibit this issue.
• Logs are attached: log.zip

[anjalirai-intel]

I am able to repro the issue with Gramine v1.7