additional tools during nodejs processes invocation

[kevin-oliveira-zocdoc]

Description

During nodejs processes execution, such as yarn install, some additional tools might need to be present on the PATH, e.g. make, cp, c++, etc, specially when the npm dependencies uses node-gyp under the hood, so these additional tools are necessary to compile binaries. The current implementation doesn’t allow us to specify more tools than the default.

This adds tools and optional_tools as options to the nodejs subsystem so that these additional tools can be added to the sandbox during nodejs processes execution.

There might be cases where macos specific tools might be needed - mostly when node-gyp is used - such as xcrun and xcodebuild, which are necessary to compile native stuff, and these tools are not available on linux based os, so I've added the optional_tools where these specific tools can be set.

Fixes #21638.

[huonw]

Looks reasonable overall. Thanks for contributing!

[huonw]

The approach we've used with Docker is optional_tools: tools that should be included if they're available. https://www.pantsbuild.org/stable/reference/subsystems/docker#optional_tools

The code that's used for filtering the tools is at:

pants/src/python/pants/backend/docker/util_rules/docker_binary.py

Lines 160 to 177 in c4da187

	if optional_tools:
	optional_tools_requests = [
	BinaryPathRequest(binary_name=binary_name, search_path=search_path)
	for binary_name in optional_tools
	]
	optional_tools_paths = await MultiGet(
	Get(BinaryPaths, BinaryPathRequest, optional_tools_request)
	for optional_tools_request in optional_tools_requests
	)
	all_binary_first_paths.extend(
	[
	cast(BinaryPath, path.first_path) # safe since we check for non-empty paths below
	for path in optional_tools_paths
	if path.paths
	]
	)

Maybe a similar approach could work here and be a bit more flexible?

It is a bit more fiddly to get set-up, though.

[kevin-oliveira-zocdoc]

It makes a lot of sense to use optional_tools instead of macos_tools, thanks for the suggestion. I'll make the change.

[kevin-oliveira-zocdoc]

Changed to use optional_tools: c710ff6

[huonw]

Ah, it'd also be good to add a test for this, to ensure the functionality keeps working in future. Again the docker version might be handy inspiration:

pants/src/python/pants/backend/docker/util_rules/docker_binary_test.py

Lines 164 to 211 in c4da187

	def test_get_docker_with_tools(rule_runner: RuleRunner) -> None:
	def mock_get_binary_path(request: BinaryPathRequest) -> BinaryPaths:
	if request.binary_name == "docker":
	return BinaryPaths("docker", paths=[BinaryPath("/bin/docker")])
	elif request.binary_name == "real-tool":
	return BinaryPaths("real-tool", paths=[BinaryPath("/bin/a-real-tool")])
	else:
	return BinaryPaths(request.binary_name, ())
	def mock_get_binary_shims(request: BinaryShimsRequest) -> BinaryShims:
	return BinaryShims(EMPTY_DIGEST, "cache_name")
	def run(tools: list[str], optional_tools: list[str]) -> None:
	docker_options = create_subsystem(
	DockerOptions,
	experimental_enable_podman=False,
	tools=tools,
	optional_tools=optional_tools,
	)
	docker_options_env_aware = mock.MagicMock(spec=DockerOptions.EnvironmentAware)
	nonlocal mock_get_binary_path
	nonlocal mock_get_binary_shims
	run_rule_with_mocks(
	get_docker,
	rule_args=[docker_options, docker_options_env_aware],
	mock_gets=[
	MockGet(
	output_type=BinaryPaths,
	input_types=(BinaryPathRequest,),
	mock=mock_get_binary_path,
	),
	MockGet(
	output_type=BinaryShims,
	input_types=(BinaryShimsRequest,),
	mock=mock_get_binary_shims,
	),
	],
	)
	run(tools=["real-tool"], optional_tools=[])
	with pytest.raises(BinaryNotFoundError, match="Cannot find `nonexistent-tool`"):
	run(tools=["real-tool", "nonexistent-tool"], optional_tools=[])
	# Optional non-existent tool should still succeed.
	run(tools=[], optional_tools=["real-tool", "nonexistent-tool"])

[huonw]

Sweet, looks great.

Thanks for waiting.

[kevin-oliveira-zocdoc]

@huonw it looks like the macos tests failed when resolving the plugin hdrhistrogram? https://github.com/pantsbuild/pants/actions/runs/12909558410/job/35998335662?pr=21833#step:8:316.
it does not seem related to my changes, can we re-run that step?

[huonw]

Agreed that it looks unrelated.