How to handle bindgen ${name}-sys style crates?

[Avi-D-coder]

I'm trying to use whisper-rs/whisper-rs-sys as a dependency, it builds in my projects shell, it builds with vendored deps via cargo vendor in the shell, but nix build fails.

The error is reproduced directly on whisper-rs with the minimal flake in this fork whisper-rs/tree/flake.

build.rs is running make in the whisper.cpp directory.

last 10 log lines:
       >   cargo:rerun-if-changed=/nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/bits/types.h
       >   cargo:rerun-if-changed=/nix/store/78ywcf25bhbbfpv91wqirg4bd9dh3495-clang-wrapper-11.1.0/resource-root/include/stdbool.h
       >   gcc  -I.              -O3 -std=c11   -fPIC -pthread -mavx -mavx2 -mfma -mf16c   -c ggml.c -o ggml.o
       >
       >   --- stderr
       >   Assembler messages:
       >   Fatal error: can't create ggml.o: Permission denied
       >   make: *** [Makefile:145: ggml.o] Error 1
       >   thread 'main' panicked at 'Failed to build libwhisper.a', /nix/store/b21i5fqrmyrvmf7jnpbvnl4b4db3vdl3-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/whisper-rs-sys-0.2.1/build.rs:55:9
       >   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[ipetkov]

Hi @Avi-D-coder! The problem is with symlinking whisperCpp.src because the build script tries to cd into that directory before invoking make. But with the symlink it ends up in the Nix store which is read only.

You have a couple of options:

• Don't symlink the source but do a regular cp -r. You could probably get away with symlinking the regular files but the root directory will have to be writable (without changing the build.rs script that is)
• skip the source filtering completely, though any changes to the flake file (for example) will cause rebuilds
• write a source filter which includes the whisperCpp source. This could be a composition filter or you could go with something simpler like lib.sourceByRegex

Hope this helps!

[Avi-D-coder]

Thanks, the whisper-rs flake now builds, but I'm not clear, how this can be applied outside of the whisper-rs flake, where would I apply this when whisper-rs is a dependency.

@yusdacra said:

To fix this in dream2nix we basically set cargoVendorDir = "$TMPDIR/nix-vendor"; and then copy the vendor dir there in dependencies derivation and make it writable (cp -rs --no-preserve=mode,ownership) and in a postInstall we just move that vendor dir to $out, and in the main derivation we use that vendor directory from dependencies derivation out copying it in the same manner. This allows any dependencies build.rs to write stuff to their own source.

Here is the code: https://github.com/nix-community/dream2nix/blob/39c64b454e5f70e1a7b3c69dc595ee71f2d224e4/src/subsystems/rust/builders/crane/default.nix#L47

This feels like it should be the default given how common build.rs, and bindgen usage is.

[ipetkov]

Fix is in #214

[Avi-D-coder]

@ipetkov I tried it, and it's still not working. I'm running into a similar issue with torch-sys wanting to download in build.rs.

error: builder for '/nix/store/c18a5ccn1mippqvzqqmkib68lcvlzw20-local-model-data-server-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       >   cargo:rerun-if-changed=/nix/store/nasgvbblxi25r6y1s82il4m0m4ghib68-glibc-2.35-224-dev/include/bits/types.h
       >   cargo:rerun-if-changed=/nix/store/1kcyy4z7lqz3f1mgmld0yc07fglvv4w0-clang-wrapper-11.1.0/resource-root/include/stdbool.h
       >   gcc  -I.              -O3 -std=c11   -fPIC -pthread -mavx -mavx2 -mfma -mf16c   -c ggml.c -o ggml.o
       >
       >   --- stderr
       >   Assembler messages:
       >   Fatal error: can't create ggml.o: Permission denied
       >   make: *** [Makefile:145: ggml.o] Error 1
       >   thread 'main' panicked at 'Failed to build libwhisper.a', /nix/store/m8xvl3b5qq3agq1dk8bq8rjqr7s15glv-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/whisper-rs-sys-0.2.1/build.rs:55:9
       >   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       For full logs, run 'nix log /nix/store/c18a5ccn1mippqvzqqmkib68lcvlzw20-local-model-data-server-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/27i1pxs6l2abn59d8773jipv2w80d4ij-local-model-data-server-0.1.0.drv' failed to build

error: builder for '/nix/store/zngmfj91wsm67az73cbv6v9f8machxya-llms-deps-0.1.0.drv' failed with exit code 101;
       last 10 log lines:
       > Caused by:
       >   process didn't exit successfully: `/build/68rm40364nwkabr65bbl7rh6yvbfxbm3-source/target/release/build/torch-sys-a18fb4cad92325bd/build-script-build` (exit status: 101)
       >   --- stdout
       >   cargo:rerun-if-env-changed=TORCH_CUDA_VERSION
       >   cargo:rerun-if-env-changed=LIBTORCH
       >
       >   --- stderr
       >   thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: [6] Couldn't resolve host name (Could not resolve host: download.pytorch.org)', /nix/store/ym97izcwzpmadgvk7hxdrhgfx35mpahw-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/torch-sys-0.10.0/build.rs:142:48
       >   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       > warning: build failed, waiting for other jobs to finish...
       For full logs, run 'nix log /nix/store/zngmfj91wsm67az73cbv6v9f8machxya-llms-deps-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/m7m145qvh6h8f9h8yqq7w36wks6n78v3-llms-0.1.0.drv' failed to build

[ipetkov]

You may need to point torch-sys to a precompiled version of libtorch instead of having it try to compile its own. I'd recommend consulting their documentation for the appropriate way to do that (e.g. set the right environment variables)

[Avi-D-coder]

That's sorta the crux of the issue, dealing with setting many sys style crates to use libs compiled by nix is a pain.
Not every library support's it without upstream changes, at some point it becomes supporting nix/crane with Rust rather than crane supporting Rust. There should be some sort of easy workaround in crane for supporting sys style crates.

[ipetkov]

There should be some sort of easy workaround in crane for supporting sys style crates.

What does an "easy workaround" actually look like?

dealing with setting many sys style crates to use libs compiled by nix is a pain.
Not every library support's it without upstream changes, at some point it becomes supporting nix/crane with Rust rather than crane supporting Rust

I understand it can be frustrating getting external projects to play nicely with Nix but I disagree with the characterization here. Crane fully supports building cargo projects. Cargo itself knows nothing about system dependencies, besides doing whatever it is instructed by build scripts. I'm open to exploring ideas but at the end of the day there is only so little that we can do or guess downsteam of Nix and its sandbox, and downstream of whatever build.rs logic is trying to run.

There's plenty of examples of *-sys crates which can build just fine under Nix. I know for a fact the following work as I use them in multiple projects which I build and deploy exclusively with Nix:

• openssl-sys just needs openssl and pkg-config listed as inputs and it can figure out the rest
• libsqlite3-sys is fully able to compile its own vendored source code with no additional inputs besides a C compiler being present

If a build script insists on touching the network at build time, it cannot be Crane's responsibility to guess and know how to satisfy it while running within the sandbox itself.

If a build script insists on expecting a FHS hierarchy, it cannot be Crane's responsibility to simulate a filesystem and guess what dependencies the build script expects to find where.

If a build script insists on running tools like apt-get, dnf, yum, or brew, it cannot be Crane's responsibility to emulate those tools and guess what they ought to be doing.

At the end of the day, it cannot be Crane's responsibility to automagically satisfy the requirements of any arbitrary build script under the sun without any additional intervention from a human who understands the project and its dependencies and how to satisfy them accordingly, especially when said build scripts expect full reign of the system which Nix explicitly takes away.