Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to OCaml 5.1.1, jsoo 5.8.2 #602

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open

Update to OCaml 5.1.1, jsoo 5.8.2 #602

wants to merge 40 commits into from

Conversation

AltGr
Copy link
Collaborator

@AltGr AltGr commented Jul 25, 2024

(reopen of #601, closed by mistake)

This mainly required time debugging and finding which parts of the code where silently failing with the newer version. Once I found the way to make our version of ppx_metaquot work with the newer ppxlib (very small code change in the end, but it took quite a bit of digging the ppxlib api), most of the effort was actually in updating jsoo rather than OCaml.

We stick at 5.1 for now because ppx_tools (of which we vendor a patched bit) could easily work with it, but isn't ported to 5.2 yet. There has been lots of improvements in jsoo in the 5.x branch, and I expect things to be faster although I haven't run benches.

I expect this to be part of a 1.1 release and bumped the version number in prevision

@AltGr AltGr force-pushed the ocamlup branch 3 times, most recently from 6296e2b to 9da4406 Compare July 25, 2024 16:06
@AltGr
Copy link
Collaborator Author

AltGr commented Jul 26, 2024

Build on OSX fails, but actually at the OCaml compilation stage ; I have no idea about the state of the OCaml 5.1 support, maybe it's been discontinued for x86_64 ?
The CI should probably be updated but we need someone with familiarity with these systems to do so.
Note also that with the new support of opam for Windows, it probably wouldn't be too difficult to propose native Windows builds, as well.

These issues are mostly unrelated to the current PR though.

@AltGr AltGr requested a review from erikmd July 26, 2024 11:50
@AltGr
Copy link
Collaborator Author

AltGr commented Jul 26, 2024

All other checks pass ; some more manual testing is still warranted (I checked the printing, running and grading only on a few ones)

@erikmd
Copy link
Member

erikmd commented Jul 26, 2024

Hi @AltGr, thanks a lot for this meticulous work!

Build on OSX fails, but actually at the OCaml compilation stage ;
I have no idea about the state of the OCaml 5.1 support, maybe it's been discontinued for x86_64 ?

Yes I just saw this failure.
I think it is not an OCaml error, but the fact the macos-latest GHA image now only provide arm support:
actions/runner-images#9741

Given the static-bin-macos CI job manually installs opam*x86_64-macos, I see two approaches:

  • either we use the macos-13 image,
  • or we replace the script with installing an arm version of opam/ocaml (this shouldn't induce much change I guess… but maybe reduce the compatibility for old macOS users, won't it?)

The CI should probably be updated but we need someone with familiarity with these systems to do so.

FWIW I am a macOS user, not a macOS expert but I could test the macOS artifacts anyway!

Note also that with the new support of opam for Windows, it probably wouldn't be too difficult to propose native Windows builds, as well.

Ah OK, great!
anyway in the meantime, users can rely on WSL.

These issues are mostly unrelated to the current PR though.

Yes. Except the macOS static binaries I'd say, as it will block continuous deployment and release.

What do you think of just putting macos-13 here?

runs-on: macos-latest

@erikmd
Copy link
Member

erikmd commented Jul 26, 2024

What do you think of just putting macos-13 here?

on second thought: given master already failed this way because of the arm image, I'll push a separate PR with this change.

Then you'll be able to rebase on this small PR (and maybe add feat:/fix: prefixes to more commits messages in your PR - those that look important to you)

(compilation_mode whole_program)
(flags --no-source-map --opt=2 --enable=use-js-string --target-env=browser)))

(dev (flags (:standard -safe-string -w -32 -warn-error -a+8))
Copy link
Member

@erikmd erikmd Jul 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions:

  1. adding this dev profile, should I change my usual learn-ocaml programming inner loop?

    Namely, coding then building learn-ocaml with a bash alias that runs:
    cd $learn_ocaml && eval $(opam env) && opam install . --deps-only --locked && make && make opaminstall && learn-ocaml build --repo=demo-repository serve

  2. I had the impression that the dev profile had less warnings enabled than in the release profile ? but probably I'm mistaken

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historically it was impossible to build in dev mode because (i) dune didn't allow to customise the jsoo flags manually, it was tied to the built-in profiles, and (ii) source-map was blowing on us and the compilation wouldn't terminate. Thankfully both of these are now fixed :)

In the new setting, the jsoo --opt=2 flag makes release compilation quite a bit slower so I'd advise to use the dev mode when, well, developping 🤯 . The Makefile still defaults to release though, I'll add a PROFILE ?= release variable.
Then you could just run make testrun PROFILE=dev

  1. hm it's quite possible, I haven't taken much attention to the specific warning flags ; I've copied them from another of my projects mainly to disable warn-error in dev mode which I find very hard to work with ("yes I know that this argument is unused, I just commented out the code, will you go on anyways ?" ;) )

Copy link
Member

@erikmd erikmd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To start with, I just reviewed the opam files. I go on

learn-ocaml-client.opam Outdated Show resolved Hide resolved
learn-ocaml-client.opam Show resolved Hide resolved
Copy link
Member

@erikmd erikmd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks a lot @AltGr for the PR!

FYI next week I'm on vacation - I'm not sure at all I'll be able to perform manual tests by then. But surely in August.

src/app/dune Outdated Show resolved Hide resolved
src/app/dune Outdated Show resolved Hide resolved
src/main/learnocaml_client.ml Show resolved Hide resolved
src/ppx-metaquot/genlifter.ml Show resolved Hide resolved
@erikmd erikmd added the kind: feature New user-facing feature. label Jul 26, 2024
@erikmd
Copy link
Member

erikmd commented Jul 26, 2024

Last question: in the end, would you like that we squash-merge your PR ? or keep all the commits in the master's history.

In the latter case, I'd suggest you rebase -i with rewording some commits messages just to add some prefix (you know, this is important because they will automatically populate the release changelog only if there's some prefix).

To make this easier, below are some suggestions for the 31 commits of the PR.

  • 21f51fd déjà OK feat(build): Migrate ocplib-i18n to ppxlib
  • c29d4ae Suggestion=> deps(opam): Use pin-depends until release of newer ocp-indent-nlfork
  • 7e744ff Suggestion=> build(dune): Update stdlib file names for 4.13
  • f69fb6b déjà OK fix(build): Fix/disable some warnings
  • 82c1ca7 Suggestion+Typo=> fix: Update genlifter.ml from upstream ppx_tools
  • 6a4e595 Suggestion=> deps(opam): Remove dependency on ocaml-migrate-parsetree
  • e765106 Suggestion=> feat: Upgrade to OCaml 4.13
  • c40c880 Suggestion+Typo=> feat: Update to OCaml 4.14.2
  • 3b5835a Suggestion=> refactor: Disable Asak for now (it's not compatible with OCaml 5.0 at the moment)
  • 2813944 Suggestion=> build(dune): Update dune-project
  • 97d9783 Suggestion=> deps(opam): ocplib-json-typed is now json-data-encoding
  • f5fd701 Suggestion=> deps(opam): Update opam files for OCaml 5.0
  • 6b4289e Suggestion=> build(dune): ppx_ocplib_i18n doesn't cope well with dune sandboxing
  • 2546d47 Suggestion=> refactor: Disable partitioning functionality for now (asak isn't compatible with 5.0)
  • 3f1d191 Suggestion=> build(duine): Update stdlib cmi list
  • 64b4968 OK Now compiles (but is broken) with 5.0.0
  • b9dc09c Suggestion=> fix: a bunch of deprecation warnings & errors
  • 6032da4 Suggestion=> fix(dune): Adjust jsoo compilation flags
  • 5d555bd Suggestion=> deps(opam): Working with OCaml 5.0.0 / jsoo 5.0.1
  • 0217355 Suggestion=> deps(opam): Update jsoo to 5.1.1
  • 378c951 Suggestion=> fix: Many fixes and update for jsoo 5
  • 3641979 Suggestion=> fix: Actually bind the --dry-run flag
  • 993e286 Suggestion=> refactor: Improve web-worker debug messages
  • b6c9cc0 Suggestion=> feat: Adjust to OCaml 5.1.1
  • ebd0091 déjà OK feat: complete porting to OCaml 5.1 and jsoo 5.8
  • e87ad94 déjà OK feat: bump version to 1.1.0
  • 787440d déjà OK feat: Reenable asak (using pin-depends for compat patch with OCaml 5)
  • fc6fafe déjà OK fix(CI): fix Dockerfile and client dependencies
  • c10038b déjà OK fix(CI): use older dependency releases in lockfile
  • 5c5a133 déjà OK fix(CI): fix static builds and some specific deps
  • 73423d1 déjà OK fix(CI): fix base Alpine version for final Docker images

@AltGr
Copy link
Collaborator Author

AltGr commented Jul 26, 2024

I'd prefer it not to be squashed as this loses a lot of information. In particular, if one wanted a build working with 4.14, it can easily be extracted now by selecting the correct intermediate commit, and that would be completely lost if squashed.

I don't believe, however, that most of these commit belong to the changelog: saying that we updated jsoo and OCaml versions is the gist of it (plus a few aside improvements, like the --dry-run flag fix). For example, it doesn't hold much meaning to have a changelog stating that Asak was disabled and then re-enabled.

@AltGr
Copy link
Collaborator Author

AltGr commented Jul 26, 2024

Thanks a lot for reviewing, this last patch should address your concerns :)

@AltGr AltGr mentioned this pull request Jul 29, 2024
@AltGr
Copy link
Collaborator Author

AltGr commented Jan 9, 2025

This has been long enough, I think this should be merged even with the failure which is limited to the macos scripts;
Unless there are comments within the next days ?

@erikmd
Copy link
Member

erikmd commented Jan 15, 2025

Dear @AltGr,
Best wishes for the new year!

This has been long enough, I think this should be merged even with the failure which is limited to the macos scripts;
Unless there are comments within the next days ?

Apologies for the ridiculously late review for this PR...
Also, thank you Louis for posting this follow-up comment, without forcibly merging the PR without notice.
So I'm working on the remaining issue tonight.

@erikmd

This comment was marked as resolved.

@erikmd

This comment was marked as resolved.

@erikmd erikmd force-pushed the ocamlup branch 3 times, most recently from c3dc438 to 6be6e54 Compare January 18, 2025 18:25
@erikmd
Copy link
Member

erikmd commented Jan 18, 2025

Hi @AltGr, good news! the branch is ready to merge from my point of view. Thanks again for this important PR!

I just let you take a quick look at my patches before that if you wish (otherwise, feel free to merge the PR now).

A few comments meanwhile:

  1. my tests confirmed that the implementation of a previously-discussed idea regarding CI and opam dependencies (namely, having jobs that test-build learn-ocaml (1) with the --locked dependencies & (2) with the latest compatible dependencies) was a great idea, and made it possible to detect several issues
  2. I tried to keep the guideline of atomic commits, while in particular the static-build-linking-flags fix is split in two commits on-purpose, as I suspect the commit 6be6e54 might be reverted later somehow if -lzstd can be dropped when migrating to ocaml 5.2, see also the issue Linking libzstd statically ocaml/ocaml#12562
  3. I added de6c0de just for a debug purpose at first, but I propose to keep it as is, given this verbosity flag is a minimal one, and it may be puzzling to have no output at all with a ~10' build time!
  4. quick question for you: with dune build --display=short, is the displayed step run after it is displayed, or before? (I believe it is "after", but I'd like to be sure ;)
  5. finally regarding performance, I noticed the last js_of_ocaml build steps are a bit long, but given for example that a full build (% make clean ; time make) on Apple M1 + macOS 14.5 with 8 cores takes ~8', I think it's fine anyway!
    % time make
    (...)
    make  952,03s user 682,79s system 344% cpu 7:54,05 total
    

See you soon.

@erikmd
Copy link
Member

erikmd commented Jan 19, 2025

@AltGr Last question for you! → what is the status of these two dependencies?

pin-depends: [
[
"ocp-indent-nlfork.1.5.5"
"git+https://[email protected]/OCamlPro/ocp-indent.git#nlfork"
]
[
"asak.0.5"
"git+https://github.com/AltGr/asak#ocaml5"
]
]

pin-depends: [
[
"asak.0.5"
"git+https://github.com/AltGr/asak#ocaml5"
]
]

@AltGr
Copy link
Collaborator Author

AltGr commented Jan 22, 2025

Thanks @erikmd, for the last review and fixes ! Indeed that zstd linking issue warranted digging deeper than I'd have expected 😅

About 4., I think so, but without certainty... I am not sure about expliciting --display=short as it may override user preferences in ~/.config/dune/config ; but anyway that's the correct setting so I'm fine with it ;)

Yes, js_of_ocaml is a bit slow, but I don't think there is much we can do about it, we are asking a lot of work from it! Maybe we can try fiddling with the options and see if there is no unneeded debug flags left (I remember that, at some point, not disabling source-maps would lead to compilation that wouldn't even terminate -- within my patience timeframe anyway); we should check the weight of the js artifacts as well.

Dependencies:

  • asak ⇒ the PR has been merged upstream and asak 0.5 released so we should switch to that
  • ocp-indent ⇒ I'm afraid we are stuck with the nlfork version, which changed the processing model quite a bit, but which couldn't make it into ocp-indent master -- changing would mean a painful rewrite of the ace interface code (which would probably hit the same difficulties that led to the creation of that fork in the first place). Seeing if more recent changes to ocp-indent can be backported to this branch would be nice though.
    EDIT: nevermind, I had ocp-indent-nlfork published on opam already, so that just needed a new version (Package ocp-indent-nlfork.1.5.5 ocaml/opam-repository#27305) ; I'll guess we can wait for publication before merging this so that the pin-depends can be cleaned up.

AltGr and others added 26 commits January 22, 2025 12:30
it's a bit annoying to have the client depend on asak just because of a datatype
that's not used, but fixing that would require some reorganisation.
(so that it's already in the Docker images)
Fix CI's FTBFS on (macOS, opam/--locked)
href: mirage/ocaml-conduit@f5facd8
TODO-Later: Remove `<` bound, Bump cohttp-lwt-unix's minimal version, Fix build
- hopefully -

Avoid the following error at ld time:

```
ld: Undefined symbols:
  _ZSTD_compressStream2, referenced from:
      _caml_zstd_compress in ocamlcommon.a[114](zstd.npic.o)
  _ZSTD_createCCtx, referenced from:
      _caml_zstd_compress in ocamlcommon.a[114](zstd.npic.o)
  _ZSTD_decompress, referenced from:
      _caml_zstd_decompress in ocamlcommon.a[114](zstd.npic.o)
  _ZSTD_freeCCtx, referenced from:
      _caml_zstd_compress in ocamlcommon.a[114](zstd.npic.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)
      ocamlc src/main/.learnocaml_main.eobjs/byte/dune__exe__Learnocaml_main.{cmi,cmti}
      ocamlc src/repo/.learnocaml_process_repository_lib.objs/byte/learnocaml_process_exercise_repository.{cmo,cmt}
      ocamlc src/repo/.learnocaml_process_repository_lib.objs/byte/learnocaml_process_tutorial_repository.{cmo,cmt}
```
as for some reason a duplicate flag was passed, causing failure:

href: https://github.com/ocaml-sf/learn-ocaml/actions/runs/12846419515/job/35821578447?pr=602
```
 #26 15.79 - File "src/app/dune", line 86, characters 7-30:
 #26 15.79 - 86 |  (name learnocaml_student_view)
 #26 15.79 -             ^^^^^^^^^^^^^^^^^^^^^^^
 #26 15.79 - (cd _build/default/src/app && /home/opam/.opam/5.1/bin/js_of_ocaml --no-source-map --no-source-map --opt=2 --enable=use-js-string --target-env=browser -o learnocaml_student_view.bc.js /home/opam/.opam/5.1/lib/cstruct/cstruct.js /home/opam/.opam/5.1/lib/bigstringaf/runtime.js ../ace-lib/ace_bindings.js learnocaml_student_view.bc-for-jsoo)
 #26 15.79 - js_of_ocaml: option '--no-source-map' cannot be repeated
 #26 15.79 - Usage: js_of_ocaml [COMMAND] …
 #26 15.79 - Try 'js_of_ocaml --help' for more information.
```
using js_of_ocaml 5.9.1
now that asak 0.5 and ocp-indent-nlfork 1.5.5 are published.
@AltGr
Copy link
Collaborator Author

AltGr commented Jan 22, 2025

Rebased and removed the pin-depends ; note that ocp-indent-nlfork is still pending publication (ocaml/opam-repository#27305) so CI is expected to fail until then

@erikmd
Copy link
Member

erikmd commented Jan 22, 2025

Thanks a lot @AltGr for your feedback and for the rebase!

I propose to:

  • merge your PR as soon as the CI is green (thanks for handing the point about the dependencies)
  • check that the docker image for master builds successfully, then release 1.1.0 myself

Fine with you?

After the release, feel free to make the announcement on Discourse OCaml and [caml-list] if you wish (given you're the main contributor for 1.1.0 !) or let me know if you prefer that I look after this - there's an old template here.

So feature-wise, this release will be twofold:

  • support OCaml 5.1 and Jsoo 5.8
  • provide the CLI feature build serve --serve-during-build I had introduced (to complement your build serve --replace feature)

@erikmd
Copy link
Member

erikmd commented Jan 22, 2025

Regarding the last failing CI job, it seems that the image ocamlpro/ocaml:5.1 would need a rebuild or so (?)

Other minor questions (follow-up of the upcoming merge of this PR):

  • regarding the js_of_ocaml performance and your remark "Maybe we can try fiddling with the options and see if there is no unneeded debug flags left":
    would you be OK that we open a side issue, just to remember checking this later? (even if I'm not worried at all about this: I believe the current build time is pretty good);

  • just to understand better the issue I workarounded in 6f6be0f:
    do you have an idea why the final js_of_ocaml command includes a duplicate flag --no-source-map for almost each .js compilation? cf. (cd _build/default/src/app && /home/opam/.opam/5.1/bin/js_of_ocaml --no-source-map --no-source-map --opt=2 … in the dune log; although I don't see the flag duplicated in the dune file…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: feature New user-facing feature.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants