Add examples for HF datasets #1371
Conversation
facebook-github-bot
added
the
CLA Signed
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/data/1371
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit c82b5fc with merge base aebad0c ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@ramanishsingh Can you add pointers to what all these three notebooks aim to cover in the PR top comment? |
Thanks, done. |
examples/nodes/utils.py
Outdated
| return data | ||
|
|
||
|
|
||
| class MDS_Net(nn.Module): |
There was a problem hiding this comment.
The model like a simple MLP. What does MDS stand for? multidataset?
There was a problem hiding this comment.
Yes, but if it is confusing I can modify the name to something simpler.
| @@ -0,0 +1,2001 @@ | |||
| x,y | |||
There was a problem hiding this comment.
Can you just call this folder "data" or "example_data"?
examples/nodes/utils.py
Outdated
| return x | ||
|
|
||
|
|
||
| def train_mds_model(datasets, weights, batch_size=512): |
There was a problem hiding this comment.
can we have a single "train" function that works for all 3 examples? They all look somewhat similar
|
Not sure how to comment on notebooks so will add comments here: |
|
nit: can you rename |
| @@ -0,0 +1,172 @@ | |||
| { | |||
| "cells": [ | |||
| "\n", | ||
| "\n", | ||
| "# Hyperparameters\n", | ||
| "batch_size = 2 #batch size is kept low so that we can easily see the bacthes when we print them in the later cells\n", |
There was a problem hiding this comment.
Drop the comment it's unnecessary
| "batch_size = 2 #batch size is kept low so that we can easily see the bacthes when we print them in the later cells\n", | ||
| "\n", | ||
| "# Next we batch the inputs, and then apply a collate_fn with another Mapper\n", | ||
| "# to stack the tensors between. We use torch.utils.data.default_collate for this\n", |
There was a problem hiding this comment.
| "# to stack the tensors between. We use torch.utils.data.default_collate for this\n", | |
| "# to stack the tensor. We use torch.utils.data.default_collate for this\n", |
| " node = PinMemory(node)\n", | ||
| "\n", | ||
| "# Since nodes are iterators, they need to be manually .reset() between epochs.\n", | ||
| "# We can wrap the root node in Loader to convert it to a more conventional Iterable.\n", |
There was a problem hiding this comment.
| "# We can wrap the root node in Loader to convert it to a more conventional Iterable.\n", | |
| "# Instead, we can wrap the root node in Loader to convert it to a more conventional Iterable.\n", |
| } | ||
| ], | ||
| "source": [ | ||
| "# Once we have the loader, we can get batches from it over multiple epochs, to train the ML model\n", |
There was a problem hiding this comment.
| "# Once we have the loader, we can get batches from it over multiple epochs, to train the ML model\n", | |
| "# Once we have the loader, we can get batches from it over multiple epochs, to train the model\n", |
| @@ -0,0 +1,148 @@ | |||
| { | |||
| "# Load IMDB dataset from huggingface datasets and select the \"train\" split\n", | ||
| "dataset = load_dataset(\"imdb\", streaming=False)\n", | ||
| "dataset = dataset[\"train\"]\n", | ||
| "# Since dataset is a Map-style dataset, we can setup a sampler to shuffle the data\n", |
There was a problem hiding this comment.
Let's link to the docs similar to other example
examples/nodes/hf_imdb_bert.ipynb
Outdated
| "# a custom map_fn to perform this. Using ParallelMapper allows us to use multiple\n", | ||
| "# threads (or processes) to parallelize this work and have it run in the background\n", | ||
| "max_len = 512\n", | ||
| "batch_size = 2 # Keeping batch size smaller to easily inspect the outputs of a batch\n", |
There was a problem hiding this comment.
| "batch_size = 2 # Keeping batch size smaller to easily inspect the outputs of a batch\n", | |
| "batch_size = 2\n", |
examples/nodes/hf_imdb_bert.ipynb
Outdated
| "dataset = load_dataset(\"imdb\", streaming=False)\n", | ||
| "dataset = dataset[\"train\"]\n", | ||
| "# Since dataset is a Map-style dataset, we can setup a sampler to shuffle the data\n", | ||
| "# Please refer to the migration guide here https://pytorch.org/data/docs/build/html/migrate_to_nodes_from_utils.html\n", |
There was a problem hiding this comment.
I gave you the wrong link, after nightly run this link is live now: https://pytorch.org/data/main/migrate_to_nodes_from_utils.html
* intial_commit * more examples * run precommit * hf mnist * add complete MNIST example * remove old py file * add imdb bert * update mnist example * update hf_bert example * add mds example * update bert example * update function name * run precommit * update bert example * update mnist notebook * update mds * delete ipynb ckpts * remove mds and simplify examples * fix some typos * simplify and remove test train mentiond * remove headings * add titles * fix typo * update url
* intial_commit * more examples * run precommit * hf mnist * add complete MNIST example * remove old py file * add imdb bert * update mnist example * update hf_bert example * add mds example * update bert example * update function name * run precommit * update bert example * update mnist notebook * update mds * delete ipynb ckpts * remove mds and simplify examples * fix some typos * simplify and remove test train mentiond * remove headings * add titles * fix typo * update url
This PR aims to add examples on how to use torchdata.nodes for dataloading and training models.
These examples aim to cover:
nodes: batching, loading, mappingMultiNodeWeightedSamplerinnodesto show the tools available innodes.Fixes #1352