Add poetry to image spec

[thomasjpfan]

Tracking issue

Related to flyteorg/flyte#6119

Why are the changes needed?

This allows poetry uses to create a image spec with their dependencies. This PR treats poetry as a "dependency manager". This means that their local project will not be installed. If we see that poetry users want to install their local package, then we can extend this feature with a follow up PR.

What changes were proposed in this pull request?

This PR adds a requirements="poetry.lock" that creates an ImageSpec with poetry.

How was this patch tested?

I created a repo with poetry init with this workflow:

from flytekit import task, ImageSpec
import numpy as np
from random import choice
from string import ascii_letters


image = ImageSpec(
    name="numpy-poetry",
    requirements="poetry.lock",
    registry="localhost:30000",
    env={"A": "".join(choice(ascii_letters) for _ in range(20))},
)


@task(container_image=image)
def add_task() -> np.ndarray:
    x = np.ones(10)
    return x + 10

and ran pyflyte run --remote workflows/main.py.

Summary by Bito

This PR implements Poetry support in ImageSpec with lock file handling capabilities and Docker build integration. It enhances array node capabilities with ReferenceTask support and improves file handling for remote access. The implementation includes comprehensive error handling across components, particularly in file operations and logging systems. Changes include expanded testing infrastructure with improved variable naming and VSCode integration enhancements.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

[flyte-bot]

Code Review Agent Run #73368b

Actionable Suggestions - 5

• tests/flytekit/unit/core/image_spec/test_default_builder.py - 2

  • Consider renaming shadowed lock_file variable · Line 281-281
  • Consider consolidating duplicate pytest parametrize decorator · Line 275-275
• flytekit/image_spec/default_builder.py - 3

  • Consider single $ for PATH variable · Line 58-58
  • Consider moving poetry arg to template · Line 196-197
  • Consider validating pyproject.toml existence · Line 216-217

Review Details

• Files reviewed - 2 · Commit Range: 0813ae8..0813ae8

  • flytekit/image_spec/default_builder.py
  • tests/flytekit/unit/core/image_spec/test_default_builder.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Poetry Support for ImageSpec	- default_builder.py - Added Poetry lock file support and template for Docker builds - test_default_builder.py - Added test cases for Poetry lock file handling and error scenarios

[flyte-bot]

Consider renaming shadowed lock_file variable

The variable name lock_file is being redefined in the function scope, shadowing the parameter with the same name. This could lead to confusion and potential bugs.

Code suggestion

Check the AI-generated fix before applying

Suggested change

	lock_file = tmp_path / lock_file
	lock_file_path = tmp_path / lock_file

Code Review Run #73368b

---

Is this a valid issue, or was it incorrectly flagged by the Agent?

• it was incorrectly flagged

[flyte-bot]

Consider consolidating duplicate pytest parametrize decorator

Consider consolidating the duplicate @pytest.mark.parametrize decorator with the one at line 254 since both test functions use the same parameters.

Code suggestion

Check the AI-generated fix before applying

 @@ -254,7 +254,6 @@
  @pytest.mark.parametrize("lock_file", ["uv.lock", "poetry.lock"])
  @pytest.mark.filterwarnings("ignore::UserWarning")
  def test_lock_errors_no_pyproject_toml(monkeypatch, tmp_path, lock_file):

  @@ -274,7 +274,6 @@
 [email protected]("lock_file", ["uv.lock", "poetry.lock"])
  @pytest.mark.filterwarnings("ignore::UserWarning")
  def test_uv_lock_error_no_packages(monkeypatch, tmp_path, lock_file):

Code Review Run #73368b

---

Is this a valid issue, or was it incorrectly flagged by the Agent?

• it was incorrectly flagged

[flyte-bot]

Code Review Agent Run #edb466

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: 0813ae8..b32a4df

  • tests/flytekit/unit/core/image_spec/test_default_builder.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by

[flyte-bot]

Code Review Agent Run #9afb0f

Actionable Suggestions - 0

Additional Suggestions - 7

• flytekit/clis/sdk_in_container/run.py - 1

  • Consider consolidating JSON serialization logic · Line 478-486
• flytekit/core/array_node.py - 1

  • Consider adding data mode validation check · Line 140-141
• tests/flytekit/unit/cli/pyflyte/test_run.py - 2

  • Consider checking exact message occurrences · Line 480-480
  • Consider more precise output assertions · Line 853-854
• tests/flytekit/integration/remote/test_remote.py - 2

  • Consider using unittest.mock instead of mock · Line 17-17
  • Consider adding cleanup error handling · Line 832-835
• tests/flytekit/unit/bin/test_python_entrypoint.py - 1

  • Consider consolidating repeated timestamp initialization pattern · Line 529-540

Review Details

• Files reviewed - 28 · Commit Range: b32a4df..fa746c8

  • dev-requirements.txt
  • flytekit/clis/sdk_in_container/run.py
  • flytekit/core/array_node.py
  • flytekit/core/array_node_map_task.py
  • flytekit/core/context_manager.py
  • flytekit/core/local_cache.py
  • flytekit/core/tracker.py
  • flytekit/core/workflow.py
  • flytekit/extend/backend/agent_service.py
  • flytekit/extend/backend/base_agent.py
  • flytekit/interactive/vscode_lib/decorator.py
  • flytekit/loggers.py
  • flytekit/models/core/workflow.py
  • flytekit/models/literals.py
  • flytekit/tools/translator.py
  • flytekit/types/directory/types.py
  • flytekit/types/file/file.py
  • plugins/flytekit-airflow/setup.py
  • plugins/flytekit-flyteinteractive/tests/test_flyteinteractive_vscode.py
  • pyproject.toml
  • tests/flytekit/integration/remote/test_remote.py
  • tests/flytekit/integration/remote/utils.py
  • tests/flytekit/integration/remote/workflows/basic/flytefile.py
  • tests/flytekit/integration/remote/workflows/basic/pydantic_wf.py
  • tests/flytekit/unit/bin/test_python_entrypoint.py
  • tests/flytekit/unit/cli/pyflyte/test_run.py
  • tests/flytekit/unit/core/test_local_cache.py
  • tests/flytekit/unit/types/directory/test_listdir.py
• Files skipped - 1

  • .github/workflows/pythonpublish.yml - Reason: Filter setting
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful

---

AI Code Review powered by