Skip to content

Commit

Permalink
Use lru_cache on costly validation function.
Browse files Browse the repository at this point in the history
This gives a 24% speedup when downloading a folder with 30,000 small files using the CRT via CLI.

When profiling this workload, `_validate_subscriber_methods()` had a big presence in the flame graph. Commenting out the validate function gave a huge speedup. But the validation is useful, so we didn't want to remove it. Using lru_cache provides the same speedup for the 30,000 file workload. It runs 6 times (once per subscriber class) instead of 180,000 times (6 times per file).
  • Loading branch information
graebm committed Nov 29, 2023
1 parent d65d17c commit 24a4ac7
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions s3transfer/subscribers.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
from functools import lru_cache

from s3transfer.compat import accepts_kwargs
from s3transfer.exceptions import InvalidSubscriberMethodError

Expand All @@ -28,6 +30,7 @@ def __new__(cls, *args, **kwargs):
return super().__new__(cls)

@classmethod
@lru_cache()
def _validate_subscriber_methods(cls):
for subscriber_type in cls.VALID_SUBSCRIBER_TYPES:
subscriber_method = getattr(cls, 'on_' + subscriber_type)
Expand Down

0 comments on commit 24a4ac7

Please sign in to comment.