Can't install numcodecs on OSX Big Sur

[d70-t]

Minimal, reproducible code sample, a copy-pastable example if possible

pip install numcodecs

Problem description

Installation fails due to Blosc/python-blosc#229. Updating the c-blosc submodule to v1.21.0 fixes the issue.

Version and installation information

Please provide the following:

• Value of numcodecs.__version__: current master branch
• Version of Python interpreter: 3.9
• Operating system (Linux/Windows/Mac): Mac

[jakirkham]

Thanks for the data point! 😄

[adamjstewart]

I'm also seeing this problem on macOS Catalina. The issue is with Apple Clang 12.0.0, not with the specific macOS version. See Blosc/c-blosc#316 for my error msg. It would be great if c-blosc could be installed externally at any version we need instead of using the git submodule.

[olly-writes-code]

I'm getting the same issue on a M1 MacBook Pro MacOS Big Sur 11.5 when trying to install numcodec v0.8.0 in a virtualenv using poetry and pyenv. Trying to install for python 3.8.11.

The error message is

c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: error: implicit declaration of function 'read' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = read(state->fd, buf + *have, len - *have);
              ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:30:15: note: did you mean 'fread'?
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:158:9: note: 'fread' declared here
size_t   fread(void * __restrict __ptr, size_t __size, size_t __nitems, FILE * __restrict __stream);
         ^
c-blosc/internal-complibs/zlib-1.2.8/gzread.c:591:11: error: implicit declaration of function 'close' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    ret = close(state->fd);
          ^
2 errors generated.
[numcodecs] command 'clang' failed with exit status 1

@d70-t - how did you force the numcodecs install to use a newer version of c-blosc? Did you manage to figure out a workaround?

[olly-writes-code]

The issue is slightly different when installing in a Python 3.9.6 virtualenv. Instead I get

 c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              state->start = LSEEK(state->fd, 0, SEEK_CUR);
                             ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:256:24: note: did you mean 'fseek'?
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:162:6: note: 'fseek' declared here
      int      fseek(FILE *, long, int);
               ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:355:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
              ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:396:15: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
                    ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:492:14: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
          offset = LSEEK(state->fd, 0, SEEK_CUR);
                   ^
      c-blosc/internal-complibs/zlib-1.2.8/gzlib.c:14:17: note: expanded from macro 'LSEEK'
      #  define LSEEK lseek
                      ^
      4 errors generated.
      [numcodecs] command '/usr/bin/clang' failed with exit code 1

[jakirkham]

Probably we just need to update the vendored Blosc. In particular commit ( Blosc/c-blosc@406f596 ) is referenced as having fixed that issue. Could someone please open a PR updating the Blosc submodule in Numcodecs to the latest version?

[olly-writes-code]

It seems like it should be installing the latest version of blosc https://github.com/zarr-developers/numcodecs/blob/master/.gitmodules

[olly-writes-code]

more details - the blosc community seem to believe this is a numcodecs issue Blosc/c-blosc#316

[jakirkham]

It seems like it should be installing the latest version of blosc https://github.com/zarr-developers/numcodecs/blob/master/.gitmodules

Exactly can you please send a PR to update it?

[olly-writes-code]

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

[olly-writes-code]

I'll have a look into alternative methods for building blosc. Maybe this is the answer long term? https://github.com/Blosc/python-blosc

[jakirkham]

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

I don't think that is correct. We are using v1.18.1. The latest tag is v1.21.0. The fix alluded to earlier is v1.20.1.

Anyways if you have a moment please send a PR, would be happy to review 🙂

[jakirkham]

I'll have a look into alternative methods for building blosc. Maybe this is the answer long term? https://github.com/Blosc/python-blosc

Yeah this is something we have discussed. Agree it would be beneficial. Previously this was not possible as there were not Blosc wheels. That has since changed ( #262 ). More generally there are other compiled bits that would be nice to replace with 3rd party libraries ( #264 ). This would cutdown (maybe even eliminate) compilation on the Numcodecs side, which seems to be an endless source of pain for some users (as you have experienced first hand). There's PR ( #274 ) that starts down this path. Would be good if someone picked that up and pushed it across the finish line

[olly-writes-code]

I haven't worked with the numcodecs module so I could be wrong. My reading of the code is that the .gitmodules import uses the latest blosc package so I don't think the issue is the blosc package version. I believe the issue is to do with building blosc as a vendor submodule Blosc/c-blosc#316 (comment)

I don't think that is correct. We are using v1.18.1. The latest tag is v1.21.0. The fix alluded to earlier is v1.20.1.

Anyways if you have a moment please send a PR, would be happy to review 🙂

Ahh yes I understand now. Here's the PR #283

[olly-writes-code]

I now have numcodecs and zarr working with this PR #283