Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: reproduce issue (#3) #6

Merged
merged 7 commits into from
Mar 6, 2024
Merged

bug: reproduce issue (#3) #6

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a4ede1b
try making pyspark consistent
caldempsey Mar 2, 2024
e68f596
newlines
caldempsey Mar 2, 2024
f9849ec
Update Dockerfile
caldempsey Mar 3, 2024
a8a0344
Update Dockerfile
caldempsey Mar 3, 2024
68c18e6
remove unused deps
caldempsey Mar 3, 2024
fa78ce1
notebook logging
caldempsey Mar 3, 2024
4be964e
add data back
caldempsey Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ healthcheck env:
./infra-scripts/health_check.sh {{env}}

# deploys to a local environment, passes arguments to docker-compose
deploy args:
deploy *args:
docker rm -f $(docker ps -a -q) || true
just build
mkdir -p notebook-data-lake/src
Expand Down
1 change: 0 additions & 1 deletion infra-data-lake/localhost/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@ services:
ports:
- '8890:8888'
volumes:
caldempsey marked this conversation as resolved.
Show resolved Hide resolved
- ./../../notebook-data-lake/data:/data
Copy link
Owner Author

@caldempsey caldempsey Mar 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this causes the issue, adding this line back everything works fine. shouldn't be this way because this is the jupyter notebook. not the cluster.

- ./../../infra-mock-blob-storage:/home/jovyan/mock-blob-storage
- ./../../notebook-data-lake/notebooks:/home/jovyan/notebooks
- ./../../notebook-data-lake/data:/home/jovyan/data
Expand Down
4 changes: 3 additions & 1 deletion infra-data-lake/notebook/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,6 @@ ENV PATH=$PATH:$SPARK_HOME/bin
RUN pip install findspark

# Install PySpark
RUN pip install pyspark
RUN pip install pyspark==3.3.1
RUN pip install delta
RUN pip install delta-spark==3.1.0
3 changes: 2 additions & 1 deletion infra-data-lake/spark/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ FROM bitnami/spark:3.5.1
USER root
RUN install_packages curl
USER 1001
COPY _build/spark_jars /opt/bitnami/spark/jars/
COPY _build/spark_jars /opt/bitnami/spark/jars/
RUN pip install pyspark==3.3.1
Loading
Loading