Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add a custom storage options param #246

Merged
merged 9 commits into from
Jul 20, 2024

Conversation

csy1204
Copy link
Contributor

@csy1204 csy1204 commented Jul 20, 2024

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #236 (issue).

import litdata as ld

storage_options = {
    "endpoint_url": url,
    "aws_access_key_id": id,
    "aws_secret_access_key": pw,
}

s3_path = "s3://bucket/test_dataset"
dataset = ld.StreamingDataset(s3_path, storage_options=storage_options)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

Copy link

codecov bot commented Jul 20, 2024

Codecov Report

Attention: Patch coverage is 95.23810% with 1 line in your changes missing coverage. Please review.

Please upload report for BASE (main@fd411e5). Learn more about missing BASE report.

Additional details and impacted files
@@          Coverage Diff          @@
##             main   #246   +/-   ##
=====================================
  Coverage        ?    78%           
=====================================
  Files           ?     34           
  Lines           ?   4870           
  Branches        ?      0           
=====================================
  Hits            ?   3781           
  Misses          ?   1089           
  Partials        ?      0           

Copy link
Collaborator

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @csy1204 Looks great. Could you add an example in the README ?

tests/streaming/test_client.py Show resolved Hide resolved
src/litdata/streaming/client.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@csy1204
Copy link
Contributor Author

csy1204 commented Jul 20, 2024

@tchaton Thanks for your reviews.
I've added the example of this right after ✅ Stream large cloud datasets documentation. Feel free to edit my example 😃

Co-authored-by: thomas chaton <[email protected]>
@tchaton tchaton merged commit a2cfe7b into Lightning-AI:main Jul 20, 2024
28 checks passed
@csy1204 csy1204 deleted the feat/custom-s3-connection branch July 20, 2024 15:26
@tchaton
Copy link
Collaborator

tchaton commented Jul 20, 2024

Thanks @csy1204 for your contribution ;) Feel free to continue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Inject custom s3 connection configs (credentials)
2 participants