Pyinstaller fail to package echopype as it cannot import name 'Blosc' from 'numcodecs'

[cywhale]

My .py that use echopype can normally run without error in dev. But when using Pyinstaller to make the .py executable, I got the error when running it (as following) after Pyinstaller successfully make it installed. I found the same error was also issued in
zarr-developers/numcodecs#291

I still have trouble to make it work after trying some methods and wonder if anyone use echopype had ever handled it before?
platform: Ubuntu 20.04.3 LTS virtualenv of python 3.8
echopype==0.5.5
pyinstaller==4.8
zarr==2.10.3
numcodecs==0.9.1

(py3.8) $ ./dist/myFunc 
Traceback (most recent call last):
  File "myFunc.py", line 25, in <module>
    import echopype as ep
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "echopype/__init__.py", line 5, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "echopype/calibrate/__init__.py", line 1, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "echopype/calibrate/api.py", line 3, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "xarray/__init__.py", line 1, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "xarray/tutorial.py", line 13, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "xarray/backends/__init__.py", line 17, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "xarray/backends/zarr.py", line 22, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "zarr/__init__.py", line 2, in <module>
  File "PyInstaller/loader/pyimod03_importers.py", line 495, in exec_module
  File "zarr/codecs.py", line 3, in <module>
ImportError: cannot import name 'Blosc' from 'numcodecs' (/tmp/_MEI1DhIqz/numcodecs/__init__.pyc)

[emiliom]

I've never used pyinstaller. I don't recall us running into blosc errors when building or using echopype, and I'm reasonably certain that we've used blosc compression with zarr exports fairly often; I think it's the default.

The error described in zarr-developers/numcodecs#291 definitely seems to be the same as yours. It helps that it's a very recent issue, though an important difference is that it's on Windows rather than Ubuntu. Unfortunately it doesn't look like they've arrived at helpful diagnostics.

While I haven't experienced problems with numcodecs/blosc myself, it looks like such issues are not rare, though more typically associated with Windows and Mac OS than Linux. This comment from 6 months ago is a good summary:

More generally there are other compiled bits that would be nice to replace with 3rd party libraries ( zarr-developers/numcodecs#264 ). This would cutdown (maybe even eliminate) compilation on the Numcodecs side, which seems to be an endless source of pain for some users (as you have experienced first hand). There's PR ( zarr-developers/numcodecs#274 ) that starts down this path. Would be good if someone picked that up and pushed it across the finish line

However, that still won't help you. If you're still having problems we may be in a better position to explore this issue next week.

[leewujung]

I haven't used pyinstaller before, but I've run into a very similar error when packaging echopype sometime before, also coming from Blosc via zarr. At that time I pinned a lower zarr version since it is coming from a new release, and waited until the issue was resolved.

@cywhale : do you have access to another OS that you could try it on? Also wonder what happens if you pinned the suggested zarr >= version in zarr-developers/numcodecs#291 before you bundle things together in pyinstaller?

[cywhale]

@leewujung I set up a Win10 python(3.9) virtualenv by conda, with echopype==0.5.5, zarr is newest 2.10.3, and then got the same error exactly as numcodecs#291. FYI

[IanTBlack]

I also received a numcodecs error when trying to install echopype via pip.
Here is the error.

Running setup.py install for numcodecs ... error
  error: subprocess-exited-with-error

  × Running setup.py install for numcodecs did not run successfully.
  │ exit code: 1
  ╰─> [126 lines of output]
   
.......

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> numcodecs

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I tried to use the numcodecs pre-release.
I was able to install echopype just fine after installing the pre-release, but now I am getting a

DeprecationWarning: notifyAll() is deprecated, use notify_all() instead self.bytes_consumed.notifyAll()

If I import numcodecs, I don't see the Warning. The warning only appears after importing echopype or running an echopype function. Given that it is a warning, I will continue to see if I can use echopype.

Update: I typically use PyCharm, so the deprecation warning may be associated with the IDE. I just ran through the OOI example in Windows Powershell and had no problems.

[leewujung]

@IanTBlack : Thanks so much for the very detailed description on what you have seen and a temp workaround.

It sounded like that we could wait to see if numcodecs's next release would resolve this problem. Not sure where the bytes_condumed.notifyAll() error came from -- were you generating or performing computation using .zarr files?

[IanTBlack]

@leewujung I definitely think the notifyAll() error is associated with my IDE.

I only ran through the first part of the OOI example, which I believe generates a zarr from the OOI raw data archive.

[leewujung]

@IanTBlack : ok, this makes sense then, since this is related to the zarr component, so is consistent with the previous numcodecs issue. Thanks for reporting everything! Please keep us posted if this goes away in your later use.

[psavery]

Hi all. I believe this should be fixed via pyinstaller/pyinstaller-hooks-contrib#420.

Until a new release comes out, the master branch may be installed via a command like pip install git+https://github.com/pyinstaller/pyinstaller-hooks-contrib.

[cywhale]

Hi all. I believe this should be fixed via pyinstaller/pyinstaller-hooks-contrib#420.

Until a new release comes out, the master branch may be installed via a command like pip install git+https://github.com/pyinstaller/pyinstaller-hooks-contrib.

It works. Now I can correctly package echopype to exe. Thanks for sharing this.

[leewujung]

Thanks @psavery for letting us know! @cywhale glad that you are able to make this work now!