s3transfer 0.11.0 is causing issue in library installations.

[vdantuluri-takeda]

It is failing during the installation of dependent libraries with boto core

cannot import name 'DEFAULT_CHECKSUM_ALGORITHM' from 'botocore.httpchecksum' (/databricks/python/lib/python3.8/site-packages/botocore/httpchecksum.py)

Runtime python 3.8.

[jonathan343]

Hey @vdantuluri-takeda, thanks for bringing this issue to our attention!

The botocore-1.36.0 release introduced new default checksum behavior in the s3 client. With these changes we also introduced a new constant (DEFAULT_CHECKSUM_ALGORITHM) which s3transfer-0.11.0 imports. However, because the dependency range for s3transfer is still 'botocore>=1.33.2,<2.0a.0', it's still possible to install a version of botocore that doesn't have this constant and result in the error you're seeing above.

To address this issue I see us taking one of the following paths:

• Update the supported botocore version range to something like 'botocore>=1.36.0,<2.0a.0'.
• Try importing the DEFAULT_CHECKSUM_ALGORITHM from botocore and falling back to CRC32 if this fails (see example below)

try:
    from botocore.httpchecksum import DEFAULT_CHECKSUM_ALGORITHM
except ImportError:
    DEFAULT_CHECKSUM_ALGORITHM = "CRC32"

I expect we'll resolve this issue in time for tomorrows daily release. In the meantime, this issue can be mitigated by using a version of s3transfer < 0.11.0 or using a version of botocore >= 1.36.0.

[jonathan343]

I've opened #326 to resolve this issue. I'll provide any updates in this ticket.

[jonathan343]

#326 is now approved and merged. This fix will be included in tomorrow's release of s3transfer-0.11.1. I'll provide an update in this thread when s3transfer-0.11.1 is released and I verify the issue is resolved.

[jonathan343]

Closing this issue since this was resolved in yesterday's release of s3transfer-0.11.1. Please re-open if you continue seeing these issues.

Thanks again for your patience and bringing this to our attention!