Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasets module #492

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Datasets module #492

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b3750ff
feat: added initial files for philadelphia crime dataset
mskaa3 Jan 2, 2025
0bb6263
feat: added initial files for chciago crime dataset
mskaa3 Jan 2, 2025
e240707
feat: added initial files for police incidents dataset
mskaa3 Jan 2, 2025
b8344dc
feat: added initial files for airbnb dataset & updated load function …
mskaa3 Jan 3, 2025
cbf930a
feat: added initial files for house sales in king county dataset
mskaa3 Jan 3, 2025
bd9ad65
Merge branch 'main' into develop
piotrgramacki Jan 3, 2025
3afa07e
feat: added required libraries for dataset module
mskaa3 Jan 3, 2025
9f0e979
feat: remove HF loader
piotrgramacki Jan 3, 2025
daa05f8
fix: update lockfile with new pdm
piotrgramacki Jan 3, 2025
1c1b0b5
fix: update lockfile platform
piotrgramacki Jan 3, 2025
eea4e88
fix: pdm lock now working?
piotrgramacki Jan 3, 2025
ed5a0d0
chore: simplify directory structure for datasets
piotrgramacki Jan 15, 2025
00e6daa
fix: fixed loading train/test splits & corrected dataset attributes
mskaa3 Jan 17, 2025
4306636
fix: deleted unneccesary code
mskaa3 Jan 17, 2025
9593a42
fix: fixed train test split
mskaa3 Jan 17, 2025
f63fe1a
feat: added funtion to retrieve h3 indexes with target labels
mskaa3 Jan 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 104 additions & 0 deletions examples/datasets/airbnb_multicity.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from srai.datasets import AirbnbMulticityDataset\n",
"\n",
"%load_ext dotenv\n",
"%dotenv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"airbnb_multicity = AirbnbMulticityDataset()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading default version"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"hf_token = os.getenv(\"HF_TOKEN\")\n",
"gdf = airbnb_multicity.load(hf_token=hf_token)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading raw, full data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_all = airbnb_multicity.load(hf_token=hf_token, version=\"all\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_all.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
106 changes: 106 additions & 0 deletions examples/datasets/chicago_crime.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from srai.datasets import ChicagoCrimeDataset\n",
"\n",
"%load_ext dotenv\n",
"%dotenv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"chicago_crime = ChicagoCrimeDataset()\n",
"hf_token = os.getenv(\"HF_TOKEN\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load default data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf = chicago_crime.load(hf_token=hf_token)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load data from 2022"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_2022 = chicago_crime.load(hf_token=hf_token, version=\"2022\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_2022.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
99 changes: 99 additions & 0 deletions examples/datasets/house_sales_in_king_county.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from srai.datasets import HouseSalesInKingCountyDataset\n",
"\n",
"%load_ext dotenv\n",
"%dotenv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"hskc_dataset = HouseSalesInKingCountyDataset()\n",
"hf_token = os.getenv(\"HF_TOKEN\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load default version of dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf = hskc_dataset.load(hf_token=hf_token)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load raw version of dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_all = hskc_dataset.load(hf_token=hf_token, version=\"all\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"gdf_all.head()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading
Loading