Skip to content

Commit

Permalink
refinement: Update README.md (#2)
Browse files Browse the repository at this point in the history
  • Loading branch information
caldempsey authored Mar 2, 2024
2 parents 9fdc984 + 874496a commit 6c857c8
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This repository provides a local experimental environment for data lakes and mock blob storage, leveraging PySpark and Spark clusters. It allows you to mimic Blob Storage locally and manage it with an Jupyter Notebook connected to a Spark Cluster closely emulating a real but simple environment.

This setup uses `mvn` to pull artefacts and transitive dependencies for Spark, e.g. [Databricks Delta Lake](https://delta.io), directly into the Spark's jars without any requirement for network requests from Spark, providing an effective template for the CI deployment for data processing pipelines and analytics in a secure or controlled setting.
This setup uses `mvn` to pull artefacts and transitive dependencies for Spark, e.g. [Databricks Delta Lake](https://delta.io), used as an example in this template, directly into the Spark's jars without any requirement for network requests from Spark, providing an effective template for the CI deployment for data processing pipelines and analytics in a secure or controlled setting.

Effortlessly dive in and unleash your data's potential, today!

Expand Down

0 comments on commit 6c857c8

Please sign in to comment.