Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian unstable #151

Merged
merged 5 commits into from
Nov 12, 2023
Merged

Debian unstable #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
16fb138
Update toolRequirements
hasufell Nov 12, 2023
0ae75a2
Use deb11 for unknown debian version
hasufell Nov 12, 2023
10ae84a
Test debian:unstable in bindist tests
hasufell Nov 12, 2023
772e54b
Re-sign
hasufell Nov 12, 2023
f1f8eb8
Test all sigs, not just the main yaml
hasufell Nov 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/bindists.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ on:
metadataFile:
description: Metadata file
required: true
default: ghcup-0.0.7.yaml
default: ghcup-0.0.8.yaml
type: string
channel:
description: Distribution channel (main|prerelease|nightly)
Expand Down Expand Up @@ -48,6 +48,9 @@ jobs:
- image: debian:11
installCmd: apt-get update && apt-get install -y
toolRequirements: build-essential curl libffi-dev libgmp-dev libgmp10 libncurses-dev libncurses5 libtinfo5 libnuma-dev
- image: debian:unstable
installCmd: apt-get update && apt-get install -y
toolRequirements: build-essential curl libffi-dev libgmp-dev libgmp10 libncurses-dev libncurses6 libtinfo6 libnuma-dev
Comment on lines +51 to +53
Copy link

@marinelli marinelli Nov 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a valid configuration for the debian:12 image. If you take a look here https://packages.debian.org/source/stable/ncurses, you will see that libncurses5 and libtinfo5 are just legacy packages. The libncurses-dev package depends on the new ones:

# apt show libncurses-dev 2>&1 | grep ^Depends
Depends: libtinfo6 (= 6.4-4), libncurses6 (= 6.4-4), libncursesw6 (= 6.4-4), libc6-dev | libc-dev, ncurses-bin (>= 6.0+20151017)

I would actually do an other thing: I would completely remove the binary packages from those commands for all the Debian/Ubuntu releases. In those distros every -dev package depends on the corresponding binary library package. So, instead of writing:

toolRequirements: build-essential curl libffi-dev  libgmp-dev libgmp10 libncurses-dev libncurses6 libtinfo6 libnuma-dev

We can just write:

toolRequirements: build-essential curl libffi-dev  libgmp-dev libncurses-dev libnuma-dev

Btw, we are already using this approach with the numa library, we just require libnuma-dev, and apt will install libnuma1 too.

The only difference would be that the packages for the library binaries (eg: libncurses6), will not be marked as automatically installed, and this is not really important, because the binaries and the headers are both required.

We could have something like this:

- image: debian:10
  installCmd: apt-get update && apt-get install -y
  toolRequirements: build-essential curl libffi-dev libgmp-dev libncurses-dev libnuma-dev
- image: debian:11
  installCmd: apt-get update && apt-get install -y
  toolRequirements: build-essential curl libffi-dev libgmp-dev libncurses-dev libnuma-dev
- image: debian:12
  installCmd: apt-get update && apt-get install -y
  toolRequirements: build-essential curl libffi-dev libgmp-dev libncurses-dev libnuma-dev

In this way we will not have to never touch the dependencies :)

I'm not sure it's really useful to have the image for debian unstable. It will change during time, and the ghc packages built on it will be invalid in the medium/long term.
I think we should add debian:12 , and as I suggested in the other PR, we should use the most update version with unknown versions. (EDIT: you already changed the unknown version, I noticed it right now. I'm sorry)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, we should also fix the documentation available here https://www.haskell.org/ghcup/install/#version-12.
I'm pretty in favor of just suggesting build-essential curl libffi-dev libgmp-dev libncurses-dev libnuma-dev, but if you think its better to be more explicit, we should replace libncurses5 and libtinfo5 with the soname 6 versions, and we should also mention libnuma1 and libnuma-dev.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a valid configuration for the debian:12 image.

Then we add a separate section for debian:12.

I would actually do an other thing: I would completely remove the binary packages from those commands for all the Debian/Ubuntu releases. In those distros every -dev package depends on the corresponding binary library package. So, instead of writing:

This is not correct as explained below.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure is correct :) See below

- image: ubuntu:18.04
installCmd: apt-get update && apt-get install -y
toolRequirements: build-essential curl libffi-dev libffi6 libgmp-dev libgmp10 libncurses-dev libncurses5 libtinfo5
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/test-sigs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,7 @@ done
unset key
gpg --verify "${METADATA_FILE}.sig"

for f in ghcup-*.json ghcup-*.yaml hls-metadata-*.json ; do
gpg --verify "${f}.sig"
done

8 changes: 4 additions & 4 deletions ghcup-0.0.7.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@
- build-essential
- curl
- libffi-dev
- libffi6
- libffi8
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can completely remove all the libBlah123 packages if we have the corresponding libBlah-dev package.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not how this works. GHC versions depend on certain specific libraries with specific SONAMEs. We're making a best effort here (should probably also specify libffi7 if it still exists).

Copy link

@marinelli marinelli Nov 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not how this works. GHC versions depend on certain specific libraries with specific SONAMEs. We're making a best effort here (should probably also specify libffi7 if it still exists).

But this dependency would come from the -dev package and the debian/ubuntu base image you are going to use to build ghc. In Debian/Ubuntu, for every release we have just one single soname version, having multiple ones might happen (rarely) with wide used libraries (like ncurses), but just for transition purposes.

Note: we should obviously use the same images for building the ghc tarballs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We support GHC versions back to 8.0.2. The older bindists are linked against older libraries that are not corresponding to what -dev installs. So that is insufficient. That is not a transition period.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-checked experimentally, 'cause I do not use Debian/Ubuntu stable releases since a lot of years (just Debian testing/unstable/experimental and Ubuntu devel releases), with the following command:

$ bash -c 'for image in debian:10 debian:11 debian:12 debian:testing ubuntu:18.04 ubuntu:20.04 ubuntu:22.04 ubuntu:23.10 ; do echo "Testing $image" ; docker run --quiet --rm "$image" sh -c "apt -qy update ; apt -qy upgrade ; apt -qy install build-essential curl libffi-dev libgmp-dev libncurses-dev libnuma-dev ; curl -sSf https://get-ghcup.haskell.org | env BOOTSTRAP_HASKELL_NONINTERACTIVE=1 BOOTSTRAP_HASKELL_MINIMAL=1 sh ; . /root/.ghcup/env ; ghcup upgrade ; ghcup install ghc 9.4.8" ; done'

And it works with all the base images.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We support GHC versions back to 8.0.2. The older bindists are linked against older libraries that are not corresponding to what -dev installs. So that is insufficient. That is not a transition period.

It means that the dev environment used to build that ghc release was broken or badly configured.
I think it would be more helpful to just deprecate those particular tarballs or rebuild them in the correct way and remove all the corner cases, they just create confusion. I understand that this would be probably a not so easy job to do. Anyway, thanks for fixing the main problem.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means that the dev environment used to build that ghc release was broken or badly configured.

Not at all. You can't build GHC against libraries that are not released yet. How does that work?

I think it would be more helpful to just deprecate those particular tarballs or rebuild them in the correct way and remove all the corner cases, they just create confusion. I understand that this would be probably a not so easy job to do. Anyway, thanks for fixing the main problem

I will not do that, unless I get employed by the Haskell Foundation. You can ask GHC devs. A ticket is already open here: haskell/ghcup-hs#903

Copy link

@marinelli marinelli Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means that the dev environment used to build that ghc release was broken or badly configured.

Not at all. You can't build GHC against libraries that are not released yet. How does that work?

I expressed myself in a bad way.
I was just thinking that for creating a binary distribution of ghc for distros that have the concept of a devel package (like Debian and Fedora that use packages like library-dev or library-devel), I would use those devel packages and the related binary packages.
I would not create corner cases where, for using a particular bindist of ghc for a particular distro release, you would need to depend on a library version that is not the main version in the same distro release.
We might have exceptions (ncurses has been an example in Debian), but I would never consider packaging a ghc bindist that depends on exceptions, it would make the releasing process more complicated.

Anyway, as you suggested below, this is not the right place to talk about those ideas.

I think it would be more helpful to just deprecate those particular tarballs or rebuild them in the correct way and remove all the corner cases, they just create confusion. I understand that this would be probably a not so easy job to do. Anyway, thanks for fixing the main problem

I will not do that, unless I get employed by the Haskell Foundation. You can ask GHC devs. A ticket is already open here: haskell/ghcup-hs#903

I'm sorry, I was not asking you for it :)
Nice to know that we already have a ticket. Thanks for sharing.

Copy link
Contributor

@chreekat chreekat Nov 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to removing corner cases, which is why I would recommend against rebuilding old bindists in the first place.

GHC bindists represent snapshots in time, do they not? A particular set of GHC variants build against a particular set of platforms at a particular moment in time. Is that so bad?

Having a different process that continually updates old versions of GHC against new platforms sounds like it would be even better, of course. But to build such a thing would require a lot of resources for development and maintenance that I don't think we have and goes far beyond what I expect of any open source project. Applying that process retroactively to previously-released versions introduces even more ongoing maintenance work. There are more important actions we can take to improve GHC stability and we have to be very careful how we spend our limited resources.

[edit] posted this before seeing @marinelli's latest reply, sorry for any confusion!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chreekat indeed, I also wouldn't expect GHC HQ to do that. In fact, I'd rather be completely independent of their CI and bindists. I did draft a HF proposal about this, but I never submitted it, since I wouldn't be able to accept the funding or dedicate enough time for it: https://gist.github.com/hasufell/18cb5438ce7c2ba388160588d751b32d

- libgmp-dev
- libgmp10
- libncurses-dev
- libncurses5
- libtinfo5
- libncurses6
- libtinfo6
notes: ''
Linux_Ubuntu:
"( >= 20.04 && < 20.10 )":
Expand Down Expand Up @@ -3637,7 +3637,7 @@
dlUri: https://downloads.haskell.org/~ghc/9.4.8/ghc-9.4.8-x86_64-deb11-linux.tar.xz
dlSubdir: ghc-9.4.8-x86_64-unknown-linux
dlHash: 2743629d040f3213499146cb5154621d6f25e85271019afc9b9009e04d66bf6c
unknown_versioning: *ghc-948-64-deb9
unknown_versioning: *ghc-948-64-deb11
Linux_Ubuntu:
unknown_versioning: *ghc-948-64-deb10
'( >= 16 && < 19 )': *ghc-948-64-deb9
Expand Down Expand Up @@ -6235,7 +6235,7 @@
dlHash: 728be2371e257c6960341167192fa704ff1f92ab61657dd4781710a257fae7c1
dlSubdir:
RegexDir: "stack-.*"
# FreeBSD:

Check warning on line 6238 in ghcup-0.0.7.yaml

View workflow job for this annotation

GitHub Actions / Test metadata (8.10.7, 3.6.2.0, ubuntu-22.04) 

6238:9 [comments-indentation] comment not indented like content
# unknown_versioning:
# dlUri: https://downloads.haskell.org/ghcup/unofficial-bindists/stack/2.13.1/stack-2.13.1-freebsd-x86_64.tar.xz
# dlHash: <replace_me>
Expand Down
Binary file modified ghcup-0.0.7.yaml.sig
View file
Open in desktop
Binary file not shown.
8 changes: 4 additions & 4 deletions ghcup-0.0.8.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ toolRequirements:
- build-essential
- curl
- libffi-dev
- libffi6
- libffi8
- libgmp-dev
- libgmp10
- libncurses-dev
- libncurses5
- libtinfo5
- libncurses6
- libtinfo6
notes: ''
Linux_Ubuntu:
"( >= 20.04 && < 20.10 )":
Expand Down Expand Up @@ -3637,7 +3637,7 @@ ghcupDownloads:
dlUri: https://downloads.haskell.org/~ghc/9.4.8/ghc-9.4.8-x86_64-deb11-linux.tar.xz
dlSubdir: ghc-9.4.8-x86_64-unknown-linux
dlHash: 2743629d040f3213499146cb5154621d6f25e85271019afc9b9009e04d66bf6c
unknown_versioning: *ghc-948-64-deb9
unknown_versioning: *ghc-948-64-deb11
Linux_Ubuntu:
unknown_versioning: *ghc-948-64-deb10
'( >= 16 && < 19 )': *ghc-948-64-deb9
Expand Down
Binary file modified ghcup-0.0.8.yaml.sig
View file
Open in desktop
Binary file not shown.
8 changes: 4 additions & 4 deletions ghcup-vanilla-0.0.7.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ toolRequirements:
- build-essential
- curl
- libffi-dev
- libffi6
- libffi8
- libgmp-dev
- libgmp10
- libncurses-dev
- libncurses5
- libtinfo5
- libncurses6
- libtinfo6
notes: ''
Linux_Ubuntu:
"( >= 20.04 && < 20.10 )":
Expand Down Expand Up @@ -3461,7 +3461,7 @@ ghcupDownloads:
dlUri: https://downloads.haskell.org/~ghc/9.4.8/ghc-9.4.8-x86_64-deb11-linux.tar.xz
dlSubdir: ghc-9.4.8-x86_64-unknown-linux
dlHash: 2743629d040f3213499146cb5154621d6f25e85271019afc9b9009e04d66bf6c
unknown_versioning: *ghc-948-64-deb9
unknown_versioning: *ghc-948-64-deb11
Linux_Ubuntu:
unknown_versioning: *ghc-948-64-deb10
'( >= 16 && < 19 )': *ghc-948-64-deb9
Expand Down
Binary file modified ghcup-vanilla-0.0.7.yaml.sig
View file
Open in desktop
Binary file not shown.
8 changes: 4 additions & 4 deletions ghcup-vanilla-0.0.8.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ toolRequirements:
- build-essential
- curl
- libffi-dev
- libffi6
- libffi8
- libgmp-dev
- libgmp10
- libncurses-dev
- libncurses5
- libtinfo5
- libncurses6
- libtinfo6
notes: ''
Linux_Ubuntu:
"( >= 20.04 && < 20.10 )":
Expand Down Expand Up @@ -3461,7 +3461,7 @@ ghcupDownloads:
dlUri: https://downloads.haskell.org/~ghc/9.4.8/ghc-9.4.8-x86_64-deb11-linux.tar.xz
dlSubdir: ghc-9.4.8-x86_64-unknown-linux
dlHash: 2743629d040f3213499146cb5154621d6f25e85271019afc9b9009e04d66bf6c
unknown_versioning: *ghc-948-64-deb9
unknown_versioning: *ghc-948-64-deb11
Linux_Ubuntu:
unknown_versioning: *ghc-948-64-deb10
'( >= 16 && < 19 )': *ghc-948-64-deb9
Expand Down
Binary file modified ghcup-vanilla-0.0.8.yaml.sig
View file
Open in desktop
Binary file not shown.
Loading