What it the "correct" way to write a setuptools plugin that generates python source files for a wheel?

[plannigan]

I'm working on implementing a setuptools plugin for gRPC to create the .py files from the .proto. I tried implementing a setuptools.file_finders entrypoint based on the documentation related to generating files based on revision control system. It does work as expected when building an sdist, however when directly building a wheel, the hook seems to be called too late and the generated file does not end up in the .whl file. (On second read, the docs section does says "If the files you want to include in the source distribution...", so this might not being the correct use case.)

The following it the debug output of using my POC attempting to add a simple foo.py the a module. The file gets created and listed in SOURCES.txt, but it looks like the files get staged into a temporary directory first.

$ DISTUTILS_DEBUG=1 python setup.py bdist_wheel
options (after parsing config files):
options (after parsing command line):
option dict for 'aliases' command:
  {}
option dict for 'bdist_wheel' command:
  {}
running bdist_wheel
Distribution.get_command_obj(): creating 'bdist_wheel' command object
Distribution.get_command_obj(): creating 'bdist' command object
Distribution.get_command_obj(): creating 'build' command object
Distribution.get_command_obj(): creating 'build_scripts' command object
Distribution.get_command_obj(): creating 'build_ext' command object
running build
running build_py
Distribution.get_command_obj(): creating 'build_py' command object
creating build
creating build/lib
creating build/lib/my_module
copying my_module/__init__.py -> build/lib/my_module
copying my_module/bar.py -> build/lib/my_module
Distribution.get_command_obj(): creating 'install' command object
Distribution.get_command_obj(): creating 'install_scripts' command object
installing to build/bdist.linux-x86_64/wheel
running install
config vars:
{'abiflags': '',
 'base': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'dist_fullname': 'test-package-setuptools-hook-0.0.1',
 'dist_name': 'test-package-setuptools-hook',
 'dist_version': '0.0.1',
 'exec_prefix': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'platbase': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'platlibdir': 'lib',
 'prefix': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'py_version': '3.9.10',
 'py_version_nodot': '39',
 'py_version_short': '3.9',
 'sys_exec_prefix': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'sys_prefix': '/home/patrick/.pyenv/versions/grpc-test-venv',
 'userbase': '/home/patrick/.local',
 'usersite': '/home/patrick/.local/lib/python3.9/site-packages'}
running install_lib
Distribution.get_command_obj(): creating 'install_lib' command object
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/my_module
copying build/lib/my_module/__init__.py -> build/bdist.linux-x86_64/wheel/my_module
copying build/lib/my_module/bar.py -> build/bdist.linux-x86_64/wheel/my_module
running install_egg_info
Distribution.get_command_obj(): creating 'install_egg_info' command object
Distribution.get_command_obj(): creating 'egg_info' command object
running egg_info
creating test_package_setuptools_hook.egg-info
writing test_package_setuptools_hook.egg-info/PKG-INFO
writing dependency_links to test_package_setuptools_hook.egg-info/dependency_links.txt
writing top-level names to test_package_setuptools_hook.egg-info/top_level.txt
writing manifest file 'test_package_setuptools_hook.egg-info/SOURCES.txt'
------
POC: Generating a Python file
my_module/foo.py
------
exclude_pattern: applying regex r'(^|/)(RCS|CVS|\.svn)/'
writing manifest file 'test_package_setuptools_hook.egg-info/SOURCES.txt'
Copying test_package_setuptools_hook.egg-info to build/bdist.linux-x86_64/wheel/test_package_setuptools_hook-0.0.1-py3.9.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/test_package_setuptools_hook-0.0.1.dist-info/WHEEL
creating 'dist/test_package_setuptools_hook-0.0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'my_module/__init__.py'
adding 'my_module/bar.py'
adding 'test_package_setuptools_hook-0.0.1.dist-info/METADATA'
adding 'test_package_setuptools_hook-0.0.1.dist-info/WHEEL'
adding 'test_package_setuptools_hook-0.0.1.dist-info/top_level.txt'
adding 'test_package_setuptools_hook-0.0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

$ unzip -l dist/test_package_setuptools_hook-0.0.1-py3-none-any.whl 
Archive:  dist/test_package_setuptools_hook-0.0.1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-03-19 13:34   my_module/__init__.py
       21  2022-03-19 13:02   my_module/bar.py
      183  2022-03-19 16:09   test_package_setuptools_hook-0.0.1.dist-info/METADATA
       92  2022-03-19 16:09   test_package_setuptools_hook-0.0.1.dist-info/WHEEL
       10  2022-03-19 16:09   test_package_setuptools_hook-0.0.1.dist-info/top_level.txt
      527  2022-03-19 16:09   test_package_setuptools_hook-0.0.1.dist-info/RECORD
---------                     -------
      833                     6 files

I did some digging in setuptools_scm and it appears that the code path that writes out a file uses finalize_distribution_options. It starts here, which eventually gets to here to call the code that writes files. However, I wasn't able to find documentation on this entrypoint.

[abravalheri]

Hi @plannigan there is some (very little) documentation about this entry-point in https://setuptools.pypa.io/en/latest/userguide/extension.html#customizing-distribution-options.

It is a very generic and very powerful hook. You need to write a function that accepts a Distribution object and can modify it arbitrarily.

Just note that this hook is called very early when no configuration file was read yet...

Have you considered instead adding a new sub-command to build? There is some discussion/examples in #2591.

[plannigan]

The open proposal does seem like it would be the "best" solution. For now I'll be going ahead with finalize_distribution_options

[abravalheri]

@plannigan, I was thinking after the discussion we had this afternoon:

finalize_distribution_options can be used to add subcommands...

It is a bit outside of the box, but I did the following experiment: https://github.com/abravalheri/experiment-setuptools-plugin

[plannigan]

Interesting, I'll play around with that some more. The gRPC maintainers are fairly risk adverse. Where would you say this is on the scale of "this uses documented and supported APIs" to "this is messing around with implementation details"?

[abravalheri]

I think that using finalize_distribution_options to add subcommands should be stable (I don't see why it wouldn't be), the APIs for the subcommands though should change with the PEP 660 implementation, but let's see what the other maintainers think. I will add a comment in the original issue talking about subcommands to see what they think.

[abravalheri]

I asked the question in #2591 (comment)

[abravalheri]

It does work as expected when building an sdist, however when directly building a wheel, the hook seems to be called too late and the generated file does not end up in the .whl file.

Please note the way setuptools currently is supposed to work:

• Users should call python -m build instead of python setup.py (installing build is needed)
• build will create a sdist
• build will create a wheel based on the previous sdist (not from the original folder).