[bug] 'compiled-objects-have-debug-symbols' check fails with a decoding error on numpy windows amd64 wheel

[jameslamb]

What did you expect to happen?

For pydistcheck to only print output it controls, not a Python stacktrace.

What actually happened?

Ran the following on my mac (macOS 12.2.1, intel chip)

curl -O https://files.pythonhosted.org/packages/be/b0/611101990ddac767e54e2d27d1f4576ae1662cca64e2d55ef0e62558ec26/numpy-1.26.3-cp310-cp310-win_amd64.whl

pydistcheck ./numpy-1.26.3-cp310-cp310-win_amd64.whl

yields the following:

==================== running pydistcheck ====================

checking './numpy-1.26.3-cp310-cp310-win_amd64.whl'
------------ check results -----------
Traceback (most recent call last):
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/bin/pydistcheck", line 8, in <module>
    sys.exit(check())
             ^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/pydistcheck/cli.py", line 214, in check
    errors += this_check(distro_summary=summary)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/pydistcheck/checks.py", line 43, in __call__
    has_debug_symbols, cmd_str = _archive_member_has_debug_symbols(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/pydistcheck/shared_lib_utils.py", line 79, in _archive_member_has_debug_symbols
    has_debug_symbols, cmd_str = _look_for_debug_symbols(lib_file=full_path)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/pydistcheck/shared_lib_utils.py", line 53, in _look_for_debug_symbols
    stdout = _run_command(args=[*cmd_args, lib_file])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jlamb/mambaforge/envs/pydistcheck-dev/lib/python3.11/site-packages/pydistcheck/shared_lib_utils.py", line 22, in _run_command
    return stdout.decode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 1749588: invalid continuation byte

How can someone else reproduce this problem?

see above

What version of pydistcheck are you using?

0.5.1

Notes

output of 'conda info' (click me)

     active environment : pydistcheck-dev
    active env location : /Users/jlamb/mambaforge/envs/pydistcheck-dev
            shell level : 1
       user config file : /Users/jlamb/.condarc
 populated config files : /Users/jlamb/mambaforge/.condarc
                          /Users/jlamb/.condarc
          conda version : 23.11.0
    conda-build version : 3.27.0
         python version : 3.9.9.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=23.11.0=0
                          __osx=12.2.1=0
                          __unix=0=0
       base environment : /Users/jlamb/mambaforge  (writable)
      conda av data dir : /Users/jlamb/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/jlamb/mambaforge/pkgs
                          /Users/jlamb/.conda/pkgs
       envs directories : /Users/jlamb/mambaforge/envs
                          /Users/jlamb/.conda/envs
               platform : osx-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.9.9 Darwin/21.3.0 OSX/12.2.1 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.4
                UID:GID : 501:20
             netrc file : None
           offline mode : False

output of 'conda env export'

name: pydistcheck-dev
channels:
  - conda-forge
  - defaults
dependencies:
  - black=23.7.0=py311h6eed73b_1
  - brotli-python=1.1.0=py311hdf8f085_1
  - bzip2=1.0.8=h0d85af4_4
  - ca-certificates=2023.11.17=h8857fd0_0
  - certifi=2023.11.17=pyhd8ed1ab_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - cmakelint=1.4.2=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - coverage=7.3.0=py311h2725bcf_0
  - exceptiongroup=1.1.3=pyhd8ed1ab_0
  - idna=3.6=pyhd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - isort=5.12.0=pyhd8ed1ab_1
  - libcxx=16.0.6=hd57cbcb_0
  - libexpat=2.5.0=hf0c8a7f_1
  - libffi=3.4.2=h0d85af4_5
  - libsqlite=3.43.0=h58db7d2_0
  - libzlib=1.2.13=h8a1eda9_5
  - mypy=1.5.1=py311h2725bcf_0
  - mypy_extensions=1.0.0=pyha770c72_0
  - ncurses=6.4=hf0c8a7f_0
  - openssl=3.2.0=hd75f5a5_1
  - packaging=23.1=pyhd8ed1ab_0
  - pathspec=0.11.2=pyhd8ed1ab_0
  - pip=23.2.1=pyhd8ed1ab_0
  - platformdirs=3.10.0=pyhd8ed1ab_0
  - pluggy=1.3.0=pyhd8ed1ab_0
  - psutil=5.9.5=py311h5547dcb_0
  - pysocks=1.7.1=pyha2e5f31_6
  - pytest=7.4.1=pyhd8ed1ab_0
  - pytest-cov=4.1.0=pyhd8ed1ab_0
  - python=3.11.5=h30d4d87_0_cpython
  - python_abi=3.11=3_cp311
  - pyyaml=6.0.1=py311h2725bcf_1
  - readline=8.2=h9e318b2_1
  - requests=2.31.0=pyhd8ed1ab_0
  - ruff=0.1.3=py311hec6fdf1_0
  - setuptools=68.1.2=pyhd8ed1ab_0
  - tk=8.6.12=h5dbffcc_0
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - typing-extensions=4.7.1=hd8ed1ab_0
  - typing_extensions=4.7.1=pyha770c72_0
  - tzdata=2023c=h71feb2d_0
  - urllib3=2.1.0=pyhd8ed1ab_0
  - wheel=0.41.2=pyhd8ed1ab_0
  - xz=5.2.6=h775f41a_0
  - yaml=0.2.5=h0d85af4_2
  - yamllint=1.32.0=pyhd8ed1ab_0
  - pip:
      - pydistcheck==0.5.1
      - types-requests==2.31.0.2
      - types-urllib3==1.26.25.14
prefix: /Users/jlamb/mambaforge/envs/pydistcheck-dev

[agriyakhetarpal]

Initially, I assumed that this was because Windows (and therefore, a Windows-specific NumPy wheel) needs CP1252 encoding by default, but that didn't work, so I switched decoding with latin1. I think latin1 might be better suited for analysing binary data since it can decode any byte value from 0x00 to 0xFF, and every possible byte has a valid mapping, unlike UTF-8 or CP1252 – so it would be more robust in these cases, even across different platforms and architectures.

In short, it would perfectly preserve ASCII text (0x00-0x7F range) and can represent any byte value without data loss (0x80-0xFF range):

==================== running pydistcheck ====================

checking 'numpy-1.26.3-cp310-cp310-win_amd64.whl'
------------ check results -----------
1. [compiled-objects-have-debug-symbols] Found compiled object containing debug symbols.
2. For details, extract the distribution contents and run 'objdump --all-headers "numpy.libs/libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll"'.
errors found while checking: 1

==================== done running pydistcheck ===============

and I confirmed via running the llvm-readobj --sections numpy-1.26.3-cp310-cp310-win_amd64/numpy.libs/libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll | grep .debug and llvm-objdump -h numpy-1.26.3-cp310-cp310-win_amd64/numpy.libs/libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll commands on my macOS M-series machine that the debug symbols do exist indeed.

Output from dumping NumPy 1.26.3's headers

numpy-1.26.3-cp310-cp310-win_amd64/numpy.libs/libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll:     file format coff-x86-64

Sections:
Idx Name           Size     VMA              Type
  0 .text          01efd5c8 00000003a07b1000 TEXT, DATA
  1 .data          00012cd0 00000003a26af000 DATA
  2 .rdata         000f4cc0 00000003a26c2000 DATA
  3 .pdata         000024a8 00000003a27b7000 DATA
  4 .xdata         00002370 00000003a27ba000 DATA
  5 .bss           00000000 00000003a27bd000 BSS
  6 .edata         00036558 00000003a27c5000 DATA
  7 .idata         00001990 00000003a27fc000 DATA
  8 .CRT           00000060 00000003a27fe000 DATA
  9 .tls           00000010 00000003a27ff000 DATA
 10 .reloc         000056b0 00000003a2800000 DATA
 11 .debug_aranges 00000e20 00000003a2806000 DATA, DEBUG
 12 .debug_info    0001d059 00000003a2807000 DATA, DEBUG
 13 .debug_abbrev  00005f0a 00000003a2825000 DATA, DEBUG
 14 .debug_line    0000f1de 00000003a282b000 DATA, DEBUG
 15 .debug_frame   00003610 00000003a283b000 DATA, DEBUG
 16 .debug_str     00000cdf 00000003a283f000 DATA, DEBUG
 17 .debug_loc     000220ea 00000003a2840000 DATA, DEBUG
 18 .debug_ranges  000029c0 00000003a2863000 DATA, DEBUG`

Fortunately, for NumPy 2.2.1, this is no longer a problem for their Windows amd64 Python 3.10 wheels and was probably fixed upstream :D

---

P.S. This is slightly unrelated, but maybe pydistcheck's output could also recommend platform-specific commands in the _CompiledObjectsDebugSymbolCheck class to users, as it is only LLVM's objdump that can report such issues in DLLs/PE files, so that pydistcheck has more cross-platform compatibility and doesn't need to hardcode "run objdump? i.e., recommend LLVM's objdump on macOS and GNU/Linux's objdump on Linux?

[agriyakhetarpal]

I'm happy to put together a PR if you feel such a solution would be acceptable. :D

[jameslamb]

Wow thank you for the investigation and excellent write-up!

Using a different decoding sounds like a plausible way to handle this, I'd be happy to consider a PR with that switch if you'd like to submit one.

maybe pydistcheck's output could also recommend platform-specific commands ... so that pydistcheck has more cross-platform compatibility and doesn't need to hardcode "run objdump?

I don't want to try to encode that knowledge into pydistcheck's source code. I'm worried about the amount of effort it would take to make that accurate and keep it accurate. I think it's enough for pydistcheck to say, when it finds debug symbols, what command you could run in the same environment to reproduce that finding. Which is exactly what it does.

In case you haven' seen it, here's the set of commands it tries:

pydistcheck/src/pydistcheck/shared_lib_utils.py

Lines 27 to 42 in 85b8af3

	_COMMANDS_TO_PATTERNS = [
	(["dsymutil", "-s"], r"\(N_OSO[\t ]+\)"),
	(["objdump", "--all-headers"], r"[\t ]+\.debug_line[\t ]+"),
	(["objdump", "--macho", "--all-headers"], r"[\t ]+\.debug_line[\t ]+"),
	(["objdump", "-W"], r"^Contents of the \.debug"),
	(["objdump", "--macho", "-W"], r"^Contents of the \.debug"),
	(["objdump", "-g"], r"^Contents of the \.debug"),
	(["objdump", "--macho", "-g"], r"^Contents of the \.debug"),
	(["llvm-objdump", "--all-headers"], r"[\t ]+\.debug_line[\t ]+"),
	(["llvm-objdump", "--macho", "--all-headers"], r"[\t ]+\.debug_line[\t ]+"),
	(["llvm-objdump", "-W"], r"^Contents of the \.debug"),
	(["llvm-objdump", "--macho", "-W"], r"^Contents of the \.debug"),
	(["llvm-objdump", "-g"], r"^Contents of the \.debug"),
	(["llvm-objdump", "--macho", "-g"], r"^Contents of the \.debug"),
	(["readelf", "-S"], r"[\t ]+\.debug_[a-z]+[\t ]+")
	]

pydistcheck/src/pydistcheck/shared_lib_utils.py

Lines 80 to 81 in 85b8af3

	for nm_tool in ["nm", "llvm-nm"]:
	has_debug_symbols, cmd_str = _nm_reports_debug_symbols(

It probably would be faster, safer, and more reliable to forgo all of this and just directly read the various formats' symbol tables without using a subprocess call to look up and run executables on PATH... but I don't personally know how to do that reliably, portably, and without introducing additional runtime dependencies.