Apple M3 Rosetta Linux UBI9 illegal instruction on import

[PhillipDowney]

Describe the bug
Running a ubi9 image on Apple M3 rosetta Podman linux image, on import of library a segmentation fault occurs.

To Reproduce
import tiledbsoma while running linux container on apple using rosetta
Versions (please complete the following information):

• TileDB-SOMA version: tiledb 0.32.5
  tiledbsoma 1.14.5
  -Python 3.10.14 (main, Nov 18 2024, 20:00:51) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] on linux

Additional context
Receiving following core dump on import

import tiledbsoma
Fatal Python error: Illegal instruction

Current thread 0x00007fffff4a2740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1176 in create_module
  File "<frozen importlib._bootstrap>", line 571 in module_from_spec
  File "<frozen importlib._bootstrap>", line 674 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
  File "/usr/local/lib/python3.10/site-packages/tiledbsoma/__init__.py", line 104 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883 in exec_module
  File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
  File "<stdin>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c (total: 14)
Illegal instruction (core dumped)

[PhillipDowney]

Hi I was wondering if there is an update to see whether this is addressable easily or something for a future delivery.

[johnkerl]

@PhillipDowney thanks for writing! I'm unsure re the ease of addressability.

I am unsurprised that we're seeing illegal instruction on x86 Linux VM atop Rosetta atop MacOS software atop arm64 hardware -- that sounds like there are multiple opportunities for misidentification of the correct processor family. I don't have a system at hand to debug with, though.

I don't know if this helps, but, we do have both x86_64 and arm64 binaries for TileDB-SOMA, as well as Linux binaries, at PyPI ...

[PhillipDowney]

@johnkerl Thank you for the reply, I did a lot of diagnosis trying to identify it. Also had to pick my libraries for torch and tensor flow to ensure the right process or absence of avx is detected.

What I am trying to do is build an Open Container Images that run our opensource models on arm or x86 so the user does to simplify upstream decision making... so have been knocking off the obstacles one by one. I will look if I can get the container to apply an arm wheel if it is detected on aarm. see if that fixes the problem.

[PhillipDowney]

would be good if Intel and aarm agreed on a API standard that is open so Users and developers did not have to make this call.