Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: assume BuildKit is available #913

Merged
merged 2 commits into from
Oct 27, 2023

Conversation

kdmccormick
Copy link
Collaborator

Description

BuildKit replaces and improves upon the legacy Docker builder, which was deprecated back in Feb 2023. Assuming that BuildKit is enabled allows us to seriously simplify the Dockerfile, and it also unlocks future build performance improvements based on BuildKit-specific features such as:

  • COPY --link
  • Multiple build contexts
  • More aggressive parallelization during the build process

The Docker versions which Tutor recommends (v20+) all come with BuildKit, so this should be a seamless transition for Tutor users who have been installing the recommended version.

We leave the is_buildkit_enabled template variable in place in order to avoid breaking plugins, but we now just set it to True as a constant. It will be removed some time after Quince, so plugins should remove any uses of the variable from their Dockerfiles.

Relevant discussion:
#868 (comment)

Testing

I've been passively testing this for while. Specifically, I have been using kdmccormick#34 (which is based off of this branch) to work on Open edX features for the past couple months, using the following:

  • both tutor-local and tutor-dev modes
  • both on Ubuntu 22.04 (x86-64) and macOS Ventura (arm64)
  • Docker v24.x + Compose v2.2.x

I haven't been testing the images in production-like tutor-local or tutor-k8s deployments. I don't believe the legacy->BuiltKit switch would have any impact that is specific to those modes, but I welcome additional testing if there is any concern.

@kdmccormick kdmccormick requested a review from regisb October 5, 2023 15:13
@kdmccormick kdmccormick force-pushed the kdmccormick/buildkit branch 2 times, most recently from 4f50e26 to 48ce135 Compare October 5, 2023 15:38
Copy link
Contributor

@regisb regisb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a very minor comment and we should be good to go. Thanks!

tutor/env.py Outdated
@@ -54,7 +54,8 @@ def _prepare_environment() -> None:
("HOST_USER_ID", utils.get_user_id()),
("TUTOR_APP", __app__.replace("-", "_")),
("TUTOR_VERSION", __version__),
("is_buildkit_enabled", utils.is_buildkit_enabled),
# For backwards compatability. Remove after Quince.
Copy link
Contributor

@regisb regisb Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we should add some sort of keyword in the comments to indicate future maintainers when to remove this. Something like DEPR v18. I frequently struggle to re-discover whatever obsolete parts of the code need to be deleted from one major release to the next. What do you think?

That keyword should also be added to tutor.images.build (see change below).

Copy link
Collaborator Author

@kdmccormick kdmccormick Oct 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@regisb Good thinking. Though I think DEPR v18 would leave room for ambiguity--is it deprecated in v18, or should it be removed in v18, or removed after v18?

How about REMOVE-AFTER-Vxx? For example, for this template filter, we're leaving it in for Quince (v16) but it'll be gone in Redwood, so it'd be # REMOVE-AFTER-V16.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"REMOVE-AFTER" looks great to me!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, I just pushed a new commit: 551c631

tutor/templates/build/openedx/Dockerfile Show resolved Hide resolved
BuildKit replaces and improves upon the legacy Docker builder, which was
deprecated back in Feb 2023. Assuming that BuildKit is enabled allows us
to seriously simplify the Dockerfile, and it also unlocks future build
performance improvements based on BuildKit-specific features such as:

* COPY --link
* Multiple build contexts
* More aggressive parallelization during the build process

The Docker versions which Tutor recommends (v20+) all come with
BuildKit, so this should be a seamless transition for Tutor users who
have been installing the recommended version.

We leave the ``is_buildkit_enabled`` template variable in place in order
to avoid breaking plugins, but we now just set it to ``True`` as a
constant. It will be removed some time after Quince, so plugins should
remove any uses of the variable from their Dockerfiles.

Relevant discussion:
overhangio#868 (comment)
@regisb regisb self-assigned this Oct 17, 2023
@kdmccormick kdmccormick requested a review from regisb October 17, 2023 19:39
@kdmccormick
Copy link
Collaborator Author

@regisb Did you want to re-review this, or is it good to go?

Copy link
Contributor

@regisb regisb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks Kyle!

@regisb regisb merged commit c273e7f into overhangio:nightly Oct 27, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

2 participants