add test to enforce infinite buffer size for all applicable datapipes #5707
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💊 CI failures summary and remediationsAs of commit 7e36fd9 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Please report bugs/suggestions to the (internal) Dr. CI Users group. |
NicolasHug
approved these changes
Mar 30, 2022
There was a problem hiding this comment.
Thanks @pmeier , approving to unblock but I have minor comments
NicolasHug
reviewed
Mar 30, 2022
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 6, 2022
…datapipes (#5707) Summary: * add test to enforce infinite buffer size for all applicable datapipes * use utility function to extract datapipes * check for buffer_size attr rather than type * simplify Reviewed By: NicolasHug Differential Revision: D35393173 fbshipit-source-id: d6e1e28169abec0f206db4f6e8db3d1cd7b91edd
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Given that is not acceptable for our datasets to drop any sample, we need to use an infinite buffer size in all datapipes that have a variable sized buffer. This PR adds a test that enforces this.
Currently we use
vision/torchvision/prototype/datasets/utils/_internal.py
Lines 42 to 43 in 1db8795
but that hopefully changes after pytorch/data#335 is resolved.