Skip to content

Commit

Permalink
add bundling feature
Browse files Browse the repository at this point in the history 
nix-portable's mechanism of bundling nix can also be used to bundle any other package.

This change exposes that capability via the nix native bundling API.

To be used like:
nix bundle --bundler github:DavHau/nix-portable nixpkgs#hello

See readme changes for more examples

Also refactor readme entirely
  • Loading branch information
@DavHau
DavHau committed Apr 14, 2024
1 parent d515552 commit e16c708
Show file tree
Hide file tree
Showing 3 changed files with 187 additions and 77 deletions.
222 changes: 153 additions & 69 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,92 +2,159 @@
<img width="400" src="https://gist.githubusercontent.com/DavHau/755fed3774e89c0b9b8953a0a25309fa/raw/fdb8b96eeb94d3b8a79481fa6fad53281e10b15d/nix_portable_2021-04-28_bw.png">
</p>

Nix as a static executable requiring no configuration, privileges, or (user) namespaces.

For binary downloads check the [releases](https://github.com/DavHau/nix-portable/releases) page.

### Goals:
- make it extremely simple to install nix
- make nix work in restricted environments (containers, HPC, ...)
- be able to use the official binary cache (by virtualizing the /nix/store)
- make it easy to distribute nix as a dependency of other projects

### Tested continuously on the following systems/environments:
* Distros (x86_64):
- Arch Linux
- CentOS 7
- Debian
- Fedora
- NixOS
- Ubuntu 22.04
- Ubuntu 23.10
- Ubuntu 24.04
* Distros (aarch64):
- Debian
* Other Environments:
- Github Actions
- Docker (debian image)

### Under the hood:
- The nix-portable executable is a self extracting archive, caching its contents in $HOME/.nix-portable
- Either nix, bubblewrap or proot is used to virtualize the /nix/store directory which actually resides in $HOME/.nix-portable/store
- A default nixpkgs channel is included and the NIX_PATH variable is set accordingly.
- Features `flakes` and `nix-command` are enabled out of the box.


### Drawbacks / Considerations:
If user namespaces are not available on a system, nix-portable will fall back to using proot as an alternative mechanism to virtualize /nix.
Proot can introduce significant performance overhead depending on the workload.
In that situation, it might be beneficial to use a remote builder or alternatively build the derivations on another host and sync them via a cache like cachix.org.
🪩 Use nix on any linux system, rootless and configuration free.

🔥 new: [Create software bundles](#bundle-programs) that work on any linux distribution.

### Missing Features:
- managing nix profiles via `nix-env`
- managing nix channels via `nix-channel`
- support MacOS
[💾 Downloads](https://github.com/DavHau/nix-portable/releases)

---

### Executing nix-portable
After obtaining the binary, there are two options:
1. Specify the nix executable via cmdline argument:
```
./nix-portable nix-shell ...
```
1. Select the nix executable via symlinking, similar to busybox:
### Get nix-portable
```shellSession
curl -L https://github.com/DavHau/nix-portable/releases/latest/download/nix-portable-$(uname -m) > ./nix-portable

chmod +x ./nix-portable
```

### Use nix via nix-portable

There are two ways to run nix:

#### Method 1: Pass nix command line:

```shellSession
./nix-portable nix-shell --help
```

#### Method 2: Symlink against nix-portable:

To create a `nix-shell` executable, create a symlink `./nix-shell` against `./nix-portable`.

```shellSession
ln -s ./nix-portable ./nix-shell
```

Then use the symlink as an executable:

```shellSession
./nix-shell --help
```

This works for any other nix native executable.

### Get and execute programs

Hint: Use [search.nixos.org](https://search.nixos.org/packages) to find available programs.

#### Run a program without installing it

```shellSession
./nix-portable nix run nixpkgs#htop
```

#### Create a temporary environment with multiple programs

1. Enter a temporary environment with `htop` and `vim`:

```shellSession
./nix-portable nix shell nixpkgs#{htop,vim}
```
# create a symlink from ./nix-shell to ./nix-portable
ln -s ./nix-portable ./nix-shell
# execute nix-shell
./nix-shell

2. execute htop

```shellSession
htop
```

### Executing installed programs
All programs installed via nix-portable will only work inside the wrapped environment.
To enter the wrapped environment just use nix-shell:
```
nix-portable nix-shell -p bash
### Bundle programs
nix-portable can bundle arbitrary software into a static executable that runs on [any*](#supported-platforms) linux distribution.

Prerequisites: Your software is already packaged for nix.

**Optional**: If you don't have nix yet, [get nix-portable](#get-nix-portable), then enter a temporary environment with nix and bash:

```shellSession
./nix-portable nix shell nixpkgs#{bashInteractive,nix} -c bash
```

... or use `nix run` to execute a program:
Examples:

#### Bundle gnu hello:


Create a bundle containing [hello](https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=hello) that will work on any machine:

```shellSession
$ nix bundle --bundler github:DavHau/nix-portable -o bundle nixpkgs#hello
$ cp ./bundle/bin/hello ./hello && chmod +w hello
$ ./hello
Hello World!
```
nix-portable nix run {flake-spec}

#### Bundle python + libraries

Bundle python with arbitrary libraries as a static executable

```shellSession
# create the bundle
$ nix bundle --bundler github:DavHau/nix-portable -o bundle --impure --expr \
'(import <nixpkgs> {}).python3.withPackages (ps: [ ps.numpy ps.scipy ps.pandas ])'
$ cp ./bundle/bin/python3 ./python3 && chmod +w ./python3

# try it out
$ ./python3 -c 'import numpy, scipy, pandas; print("Success !")'
Success !
```

### Container Runtimes
To simulate the /nix/store, nix-portable supports the following runtimes, preferred in this order:
- nix (shipped via nix-portable)
- bwrap (existing installation)
- bwrap (shipped via nix-portable)
- proot (existing installation)
- proot (shipped via nix-portable)
### Supported platforms

Potentially any linux system with an **x86_64** or **aarch64** CPU is supported.

nix-portable is tested continuously on the following platforms:

- Distros (x86_64):
- Arch Linux
- CentOS 7
- Debian
- Fedora
- NixOS
- Ubuntu 22.04
- Ubuntu 23.10
- Ubuntu 24.04
- Distros (aarch64):
- Debian
- Other Environments:
- Github Actions
- Docker (debian image)

### Under the hood

- The nix-portable executable is a self extracting archive, caching its contents in $HOME/.nix-portable
- Either nix, bubblewrap or proot is used to virtualize the /nix/store directory which actually resides in $HOME/.nix-portable/store
- A default nixpkgs channel is included and the NIX_PATH variable is set accordingly.
- Features `flakes` and `nix-command` are enabled out of the box.


#### Virtualization

To virtualize the /nix/store, nix-portable supports the following runtimes, preferred in this order:

- nix (shipped via nix-portable)
- bwrap (existing installation)
- bwrap (shipped via nix-portable)
- proot (existing installation)
- proot (shipped via nix-portable)

nix-portable will auto select the best runtime for your system.
In case the auto selected runtime doesn't work, please open an issue.
The default runtime can be overridden via [Environment Variables](#environment-variables).

### Environment Variables
The following environment variables are optional and can be used to override the default behavior of nix-portable
```

The following environment variables are optional and can be used to override the default behavior of nix-portable at run-time.

```txt
NP_DEBUG (1 = debug msgs; 2 = 'set -x' for nix-portable)
NP_GIT specify path to the git executable
NP_LOCATION where to put the `.nix-portable` dir. (defaults to `$HOME`)
Expand All @@ -101,8 +168,25 @@ NP_RUN override the complete command to run nix
```

### Drawbacks / Considerations

Programs obtained outside nix-portable cannot link against or call programs obtained via nix-portable. This is because nix-portable uses a virtualized directory to store its programs which cannot be accessed by other software on the system.

If user namespaces are not available on a system, nix-portable will fall back to using proot as an alternative mechanism to virtualize /nix.
Proot can introduce significant performance overhead depending on the workload.
In that situation, it might be beneficial to use a remote builder or alternatively build the derivations on another host and sync them via a cache like cachix.org.


### Missing Features

- managing nix profiles via `nix-env`
- managing nix channels via `nix-channel`
- support MacOS

### Building / Contributing

To speed up builds, add the nix-portable cache:
```

```shellSession
nix-shell -p cachix --run "cachix use nix-portable"
```
36 changes: 28 additions & 8 deletions default.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,20 @@ with builtins;
nixStatic ? pkgs.pkgsStatic.nix,

buildSystem ? builtins.currentSystem,
# hardcode executable to run. Useful when creating a bundle.
bundledPackage ? null,
...
}@inp:
with lib;
let

pname =
if bundledPackage == null
then "nix-portable"
else lib.getName bundledPackage;

bundledExe = lib.getExe bundledPackage;

nixpkgsSrc = pkgs.path;

pkgsBuild = import pkgs.path { system = buildSystem; };
Expand Down Expand Up @@ -74,7 +83,7 @@ let
zstd = packStaticBin "${inp.zstd}/bin/zstd";

# the default nix store contents to extract when first used
storeTar = maketar ([ cacert nix nixpkgsSrc ]);
storeTar = maketar ([ cacert nix nixpkgsSrc ] ++ lib.optional (bundledPackage != null) bundledPackage);


# The runtime script which unpacks the necessary files to $HOME/.nix-portable
Expand Down Expand Up @@ -433,9 +442,15 @@ let
### select executable
# the executable can either be selected by executing './nix-portable BIN_NAME',
# or by symlinking to nix-portable, in which case the name of the symlink selectes the binary
if [[ "\$(basename \$0)" == nix-portable* ]]; then
# the executable can either be selected by
# - executing './nix-portable BIN_NAME',
# - symlinking to nix-portable, in which case the name of the symlink selects the nix executable
# Alternatively the executable can be hardcoded by specifying the argument 'executable' of nix-portable's default.nix file.
executable="${if bundledPackage == null then "" else bundledExe}"
if [ "\$executable" != "" ]; then
bin="\$executable"
debug "executable is hardcoded to: \$bin"
elif [[ "\$(basename \$0)" == nix-portable* ]]; then\
if [ -z "\$1" ]; then
echo "Error: please specify the nix binary to execute"
echo "Alternatively symlink against \$0"
Expand Down Expand Up @@ -541,7 +556,7 @@ let

runtimeScriptEscaped = replaceStrings ["\""] ["\\\""] runtimeScript;

nixPortable = pkgs.runCommand "nix-portable" {nativeBuildInputs = [unixtools.xxd unzip];} ''
nixPortable = pkgs.runCommand pname {nativeBuildInputs = [unixtools.xxd unzip];} ''
mkdir -p $out/bin
echo "${runtimeScriptEscaped}" > $out/bin/nix-portable.zip
xxd $out/bin/nix-portable.zip | tail
Expand Down Expand Up @@ -575,9 +590,14 @@ let
${zip}/bin/zip -F $out/bin/nix-portable.zip --out $out/bin/nix-portable-fixed.zip
rm $out/bin/nix-portable.zip
mv $out/bin/nix-portable-fixed.zip $out/bin/nix-portable
chmod +x $out/bin/nix-portable
executable=${if bundledPackage == null then "" else bundledExe}
if [ "$executable" == "" ]; then
target="$out/bin/nix-portable"
else
target="$out/bin/$(basename "$executable")"
fi
mv $out/bin/nix-portable-fixed.zip "$target"
chmod +x "$target"
'';
in
nixPortable.overrideAttrs (prev: {
Expand Down
6 changes: 6 additions & 0 deletions flake.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,12 @@
recursiveUpdate
({

bundlers = forAllSystems (system: pkgs: {
default = drv: self.packages.${system}.nix-portable.override {
bundledPackage = drv;
};
});

devShell = forAllSystems (system: pkgs:
pkgs.mkShell {
buildInputs = with pkgs; [
Expand Down

0 comments on commit e16c708

Please sign in to comment.