Incompatibility with numpy 2.x?

[fedarko]

From biocore/empress#567. Replicated this issue on Ubuntu.

$ # Pinning python = 3.10 to match the linked Empress issue.
$ mamba create -n iowtest20241210 python=3.10 "numpy >= 2" cython  # installs numpy 2.1.3
$ conda activate iowtest
$ pip install iow  # installs iow 1.0.7
$ python3
Python 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marcus/anaconda3/envs/iowtest20241210/lib/python3.10/site-packages/bp/__init__.py", line 8, in <module>
    from ._bp import BP
  File "bp/_bp.pyx", line 1, in init bp._bp
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

I verified that the problem persists with Python 3.9 and numpy 2.0.x:

$ conda deactivate
$ mamba create -n iowtest20241210-py39 python=3.9 "numpy >= 2" cython  # installs numpy 2.0.1
$ conda activate iowtest20241210-py39
$ pip install iow  # installs iow 1.0.7
$ python3
Python 3.9.20 (main, Oct  3 2024, 07:27:41) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marcus/anaconda3/envs/iowtest20241210-py39/lib/python3.9/site-packages/bp/__init__.py", line 8, in <module>
    from ._bp import BP
  File "bp/_bp.pyx", line 1, in init bp._bp
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The issue can be resolved by downgrading numpy to a version earlier than 2.x:

$ # Still in the iowtest20241210-py39 conda environment.
$ mamba install "numpy < 2"  # installs numpy 1.26.4
$ python3
Python 3.9.20 (main, Oct  3 2024, 07:27:41) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
>>> # no error!

... but from what I can tell, iow is intended to support numpy 2.x. Not sure if something has changed upstream recently that breaks things -- figured I should pass this on if you haven't already seen this problem. Thanks!

[wasade]

Mixing and matching where things are installed from could be a problem. Just using pip works fine:

$ conda create --name iowtest
Retrieving notices: ...working... done
Channels:
 - defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/dtmcdonald/miniconda3/envs/iowtest



Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate iowtest
#
# To deactivate an active environment, use
#
#     $ conda deactivate

$ conda activate iowtest
$ pip install iow
Collecting iow
  Using cached iow-1.0.7-cp312-cp312-macosx_11_0_arm64.whl
Collecting numpy>=1.9.2 (from iow)
  Downloading numpy-2.2.0-cp312-cp312-macosx_14_0_arm64.whl.metadata (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.0/62.0 kB 3.2 MB/s eta 0:00:00
Collecting nose>=1.3.7 (from iow)
  Using cached nose-1.3.7-py3-none-any.whl.metadata (1.7 kB)
Collecting cython<1.0.0,>=0.24.1 (from iow)
  Using cached Cython-0.29.37-py2.py3-none-any.whl.metadata (3.1 kB)
Collecting pandas (from iow)
  Using cached pandas-2.2.3-cp312-cp312-macosx_11_0_arm64.whl.metadata (89 kB)
Requirement already satisfied: click in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from iow) (8.1.7)
Collecting scikit-bio>=0.5.0 (from iow)
  Using cached scikit_bio-0.6.2-cp312-cp312-macosx_11_0_arm64.whl
Requirement already satisfied: requests>=2.20.0 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from scikit-bio>=0.5.0->iow) (2.32.2)
Collecting decorator>=3.4.2 (from scikit-bio>=0.5.0->iow)
  Using cached decorator-5.1.1-py3-none-any.whl.metadata (4.0 kB)
Collecting natsort>=4.0.3 (from scikit-bio>=0.5.0->iow)
  Using cached natsort-8.4.0-py3-none-any.whl.metadata (21 kB)
Collecting scipy>=1.9.0 (from scikit-bio>=0.5.0->iow)
  Downloading scipy-1.14.1-cp312-cp312-macosx_14_0_arm64.whl.metadata (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 6.1 MB/s eta 0:00:00
Collecting h5py>=3.6.0 (from scikit-bio>=0.5.0->iow)
  Downloading h5py-3.12.1-cp312-cp312-macosx_11_0_arm64.whl.metadata (2.5 kB)
Collecting biom-format>=2.1.16 (from scikit-bio>=0.5.0->iow)
  Using cached biom_format-2.1.16-cp312-cp312-macosx_11_0_arm64.whl
Collecting statsmodels>=0.14.0 (from scikit-bio>=0.5.0->iow)
  Downloading statsmodels-0.14.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (9.2 kB)
Collecting python-dateutil>=2.8.2 (from pandas->iow)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas->iow)
  Using cached pytz-2024.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas->iow)
  Using cached tzdata-2024.2-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->iow)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from requests>=2.20.0->scikit-bio>=0.5.0->iow) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from requests>=2.20.0->scikit-bio>=0.5.0->iow) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from requests>=2.20.0->scikit-bio>=0.5.0->iow) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from requests>=2.20.0->scikit-bio>=0.5.0->iow) (2024.8.30)
Collecting patsy>=0.5.6 (from statsmodels>=0.14.0->scikit-bio>=0.5.0->iow)
  Downloading patsy-1.0.1-py2.py3-none-any.whl.metadata (3.3 kB)
Requirement already satisfied: packaging>=21.3 in /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages (from statsmodels>=0.14.0->scikit-bio>=0.5.0->iow) (23.2)
Using cached Cython-0.29.37-py2.py3-none-any.whl (989 kB)
Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Downloading numpy-2.2.0-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 56.9 MB/s eta 0:00:00
Using cached pandas-2.2.3-cp312-cp312-macosx_11_0_arm64.whl (11.4 MB)
Using cached decorator-5.1.1-py3-none-any.whl (9.1 kB)
Downloading h5py-3.12.1-cp312-cp312-macosx_11_0_arm64.whl (2.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 30.9 MB/s eta 0:00:00
Using cached natsort-8.4.0-py3-none-any.whl (38 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2024.2-py2.py3-none-any.whl (508 kB)
Downloading scipy-1.14.1-cp312-cp312-macosx_14_0_arm64.whl (23.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.1/23.1 MB 79.1 MB/s eta 0:00:00
Downloading statsmodels-0.14.4-cp312-cp312-macosx_11_0_arm64.whl (9.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.9/9.9 MB 82.4 MB/s eta 0:00:00
Using cached tzdata-2024.2-py2.py3-none-any.whl (346 kB)
Downloading patsy-1.0.1-py2.py3-none-any.whl (232 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.9/232.9 kB 22.5 MB/s eta 0:00:00
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, nose, tzdata, six, numpy, natsort, decorator, cython, scipy, python-dateutil, patsy, h5py, pandas, statsmodels, biom-format, scikit-bio, iow
Successfully installed biom-format-2.1.16 cython-0.29.37 decorator-5.1.1 h5py-3.12.1 iow-1.0.7 natsort-8.4.0 nose-1.3.7 numpy-2.2.0 pandas-2.2.3 patsy-1.0.1 python-dateutil-2.9.0.post0 pytz-2024.2 scikit-bio-0.6.2 scipy-1.14.1 six-1.17.0 statsmodels-0.14.4 tzdata-2024.2
(iowtest) 14:02:16 (dtmcdonald@when):go2bank-images$ python
Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:54:21) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
>>>

[fedarko]

Interesting! I am not sure about the conda environment shown in that log -- it looks like it does not contain an explicitly installed version of Python for this environment? (Since the Requirement already satisfied messages point to /Users/dtmcdonald/miniconda3/lib/python3.12/site-packages rather than ...miniconda3/envs/iowtest/lib/python3.12/site-packages, I think.)

I tried to reproduce a more "consistent" install than I had before (only installing Python using conda, and installing everything else using pip) and still ran into the problem:

$ mamba create -n iowtest-again
$ conda activate iowtest-again
$ mamba install python=3.10
$ which pip
/home/marcus/anaconda3/envs/iowtest-again/bin/pip
$ pip install iow  # installs iow 1.0.7, numpy 2.2.0, Cython 0.29.37, ...
$ python3
Python 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marcus/anaconda3/envs/iowtest-again/lib/python3.10/site-packages/bp/__init__.py", line 8, in <module>
    from ._bp import BP
  File "bp/_bp.pyx", line 1, in init bp._bp
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

(One terrifying idea I had was that this could be a mamba vs. conda issue, but I repeated the above steps using conda instead of mamba and again reproduced the error. So thankfully that's not it.)

[wasade]

Can you use CPython from conda-forge? Note:

$ python          
Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:54:21) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

[fedarko]

Got it. Tried three things -- strangely, none of them seemed to resolve the issue:

1. Installing python=3.10 from conda-forge (problem remains)

(I am not sure if the version of Python this installs is explicitly CPython. I just wanted to check to see if using conda-forge by itself would solve the problem. It didn't.)

$ mamba create -n iowtest-cf
$ conda activate iowtest-cf
$ mamba install -c conda-forge python=3.10
$ which pip
/home/marcus/anaconda3/envs/iowtest-cf/bin/pip
$ pip install iow  # installs iow 1.0.7, numpy 2.2.0, Cython 0.29.37, ...
$ python3
Python 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marcus/anaconda3/envs/iowtest-again/lib/python3.10/site-packages/bp/__init__.py", line 8, in <module>
    from ._bp import BP
  File "bp/_bp.pyx", line 1, in init bp._bp
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

2. Installing cpython=3.10 from conda-forge (problem remains)

As recommended by https://anaconda.org/conda-forge/python, I installed cpython specifically:

$ mamba create -n iowtest-cf-cpy
$ conda activate iowtest-cf-cpy
$ mamba install -c conda-forge cpython=3.10
$ which pip
/home/marcus/anaconda3/envs/iowtest-cf-cpy/bin/pip
$ pip install iow  # installs iow 1.0.7, numpy 2.2.0, Cython 0.29.37, ...
$ python3
Python 3.10.16 | packaged by conda-forge | (main, Dec  5 2024, 14:16:10) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marcus/anaconda3/envs/iowtest-cf-cpy/lib/python3.10/site-packages/bp/__init__.py", line 8, in <module>
    from ._bp import BP
  File "bp/_bp.pyx", line 1, in init bp._bp
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

3. Installing the latest version of cpython from conda-forge (a different problem occurs?)

$ mamba create -n iowtest-cf-cpy-latest
$ conda activate iowtest-cf-cpy-latest
$ mamba install -c conda-forge cpython  # installs cpython 3.13.1
$ which pip
/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/bin/pip
$ pip install iow  # gives a long error, i assume due to no numpy being installed
Collecting iow
  Using cached iow-1.0.7.tar.gz (1.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-iop_i01p/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-iop_i01p/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-iop_i01p/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 522, in run_setup
          super().run_setup(setup_script=setup_script)
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-iop_i01p/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 18, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
$ pip install numpy  # installs numpy 2.2.0
$ pip install cython  # installs Cython 3.0.11
$ pip install iow  # again gives an error:
Collecting iow
  Using cached iow-1.0.7.tar.gz (1.2 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Traceback (most recent call last):
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/marcus/anaconda3/envs/iowtest-cf-cpy-latest/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-doi3pv6s/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-doi3pv6s/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-doi3pv6s/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 522, in run_setup
          super().run_setup(setup_script=setup_script)
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-doi3pv6s/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 320, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 18, in <module>
      ModuleNotFoundError: No module named 'numpy'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I noticed that running pip install cython installed Cython 3.0.11, and I remembered you mentioned here that iow and Cython 3 were not compatible. So I ran pip install cython==0.29.37 (which succeeded), then reran pip install iow, and again got a long error message about No module named 'numpy'.

[wasade]

Thanks