How can we provide easy Jupyter Notebook access to PUDL?

[zaneselvans]

One type of user that we would like to serve with PUDL data is folks who are comfortable doing semi-programmatic, exploratory, interactive analysis using Python in a Jupyter Notebook, but who aren't necessarily comfortable dealing with Python environment setup, or running a local notebook server. Ideally we'd be able to drop somebody into an example notebook that has access to all of the data and appropriate software packages with nothing more than a URL.

We've had a "pilot" JupyterHub managed by 2i2c for quite a while, but it hasn't seen a lot of use by us or others, for a couple of reasons:

• You have to sign up for an account, and the account creating process is manual and thus laggy.
• The process of updating the software environment (Docker container) and data on the server has been manual.
• Because it's a pilot setup that was originally free, it has limited compute resources -- 6GB of memory and 2CPUs -- which aren't actually sufficient for working with our data.
• We could pay for more resources, but that will cost more, and the overall cost will scale with usage, and we don't have a steady source of funding to support this service.

The original idea was that it would be very useful to have scalable computational resources (e.g. backed by a Dask cluster maybe on GKE) that were easily accessible for ourselves and others, with always fresh data + software ready to go.

2i2c is understandably wondering what we actually want to do longer term, and I think we are wondering whether this is the best way to provide interactive Notebook access to PUDL. So what are the other options out there these days, and what do we actually need?

Desired setup

• 16-32GB of memory
• 2+ CPUs
• fast read access to the PUDL data
• automatic updates to the PUDL data
• ability to choose what version of the PUDL data you're working with
• ability to save the notebooks and data outputs and return to them later
• ability to sign up for the service without any human in the loop
• minimal costs for us or the users
• Not a proprietary platform that extends Jupyter notebooks in a way that makes them incompatible with the open tooling (e.g. Mode or Hex)
• Python software environment with lots of standard data science & visualization packages.

Questions

• Can fast read access to the data be working from a cloud bucket? Or do we need a local copy?
• To what degree would the setup benefit from having more scalable RAM / CPU / GPU? Is that a future requirement?
• Once we go to data-only distributions, do we need to be able to define or tweak the Python environment?

Notebook + Data Hosting Options

2i2c

2i2c is a non-profit interactive computation infrastructure provider run by many of the same folks behind Project Jupyter.

Advantages

• 2i2c is aligned with our open data / open source / open science / reproducibility values.
• Definitely compatible with other Jupyter notebook situations
• Aimed at a research / compute intensive user base. We can have as much compute as we want to pay for.
• We can define the software environment ourselves, or use off-the-shelf data science Docker images.

Questions

• How easy can we make new user onboarding?
• How easy can we make data updates / remote access to our data releases?
• How much would it cost for us to cover the required RAM / CPU resources?
• Is there any way that users can be responsible for paying for more resources?
• Could we provide a free tier and a paid tier with more resources?

Binder

For lightweight live Jupyter Notebooks, mybinder.org has been the go-to solution for a while. Unfortunately
the Binder Project (which is run by many of the same people as 2i2ic) is running out of compute credits and is scaling down. The volume of data that we’re trying to provide access to, and the CPU/RAM resources that are required to effectively work with our data are also well beyond what mybinder offers I think.

JupyterLite

JupyterLite runs a notebook server in-browser using Pyodide. We could create and maintain a JupyterLite deployment containing our data and a Python environment. Folks would go to the deployed URL and the data / environment would be downloaded into their browser's sandboxed filesystem.

Advantages

• It's free and open source.
• No infrastructure costs b/c it uses local compute.
• Runs in-browser so users don't need to think about local environment setup.

Questions

• Do users have enough local memory / CPU for it to work smoothly?
• Will downloading 10GB be a big barrier?
• Is the performance and compatibility of Pyodide good enough?
• Would this result in the same data being downloaded over and over again by users with significant cost?
• Could we make the deployment (or the data at least) downloadable from the AWS Open Data Registry somehow?
• If users do want to have access to the data on disk somewhere outside the browser sandbox, is that possible?

Kaggle

Kaggle provides a remarkable quantity of resources for free. I created a PUDL dataset on there to test it out.

Advantages

• 100GB of free storage for open datasets.
• Notebooks can be associated with the data.
• 30 GB of memory and 4 CPUs, plus GPU/TPU quotas.
• Kaggle notebooks can be connected to (paid) GCP resources.
• Lots of tutorials / courses on using data science libraries & techniques.
• Automatic data updates can be scheduled and pulled from cloud storage.
• The site can read and display rich metadata when it's provided in a datapackage.json and data packages now support annotating SQLite DBs.
• Kaggle already has a huge community of data science practitioners, and very little open climate & energy data.
• We might be able to run volunteer challenges to solve some of our data science issues.

Questions

• Kaggle is a for-profit company, so what's the catch?
• How are they monetizing this service?
• How long will it be free?
• You can't tweak the python environment, but can suggest new packages to be included in the stock environment, or install packages from within the notebook if need be.

Google Colab

Google Colab provides access to Jupyter notebooks running on GCP resources. They offer a free tier, and two paid tiers, with increasing resources. They are vague about what the resources backing the various plans, but this post suggests ~16/32/52GB of memory for the free/Pro/Pro+ subscriptions, which cost $0/$10/$50 per month.

Advantages

• Easy to share notebooks on GitHub or via Google Drive
• Free tier is probably good enough for most day-to-day usage.
• We are already completely in thrall to The Google.
• Presumably reading data from GCP buckets would be very fast.
• If folks need more resources for their workflow, they can upgrade themselves at modest cost.

Questions

• How do you hook it up to remote data?
• How do we define a python environment for the notebook? It seems like the stock environment is a big pile of common data science tooling.
• Are there features in Colab that make their notebooks incompatible with other ways of running Jupyter notebooks?
• How does the integration with GitHub / version control work?

[TrentonBush]

Not much to add, but wanted to second Kaggle and Colab as the places that have always stood out for free compute resources. Also it's Google all the way down; Kaggle was bought by Google back in 2017.

As far as product longevity, I'd speculate Kaggle out lasts Colab because free-for-users compute directly helps grow their platform and thus their core business. Colab on the other hand is probably a quite expensive branding exercise for Google. But as long as "AI" is hot it's hard to imagine them letting go of it. Or of Kaggle.

[zaneselvans]

I think I'm leaning towards Kaggle, since they have infrastructure for automatically updating the datasets, provide substantial notebook resources, can parse rich metadata about SQLite DBs, and provide a built-in ecosystem for learning / sharing data and analysis.

I haven't seen an obvious way to also publish and automatically update notebooks so I asked this question. I also noticed that they install what seem to be incompatible versions of pandas and SQLAlchemy in their default environment (which seems like it must be a common issue, or am I missing something here?) and so opened this issue.

[zaneselvans]

We do need to get back to 2i2c in a timely fashion about our current collaboration and what to do with it. I think it probably makes sense to discontinue it because:

• We primarily want to provide public access to the notebooks, where we are not responsible for the underlying cloud resources -- either technically or financially.
• Because significant resources are required to work with our data, free services like Binder probably won't be sufficient.
• We are moving away from wanting or needing to have a complex custom computation environment.
• The scalable computation that 2i2c is overkill for most of what we do -- a beefy laptop is plenty.
• All this means that a commodity notebook service where there are either freebie credits from a cloud provider (e.g. Colab) or the user can BYO compute (either by buying more resources from a cloud platform for themselves or by using their local machine) is probably both technically and financially a better fit for our use case.

2i2c seems like a better fit for an institutional user that is paying for the cloud resources themselves, and potentially needs more than a basic software environment or a whole cluster worth of compute resources.

Both Kaggle and Colab seem like they fit our needs better. Kaggle seems to provide a richer ecosystem for collaboration and learning and updating hosted open datasets that integrates well with the data we're publishing. Once all of our data is being processed by Dagster and written into the DB, it seems like we only need to provide a file path or URL pointing ate the data (and potentially metadata) in order for a notebook to work on a variety of platforms.

[bendnorman]

What do y'all think about continuing our 2i2c relationship until we set up an alternative? We can give our handful of 2i2c users a heads up we are sunsetting 2i2c in favor of another option.

[zaneselvans]

The current situation doesn't really work for a few reasons:

• The data isn't updated frequently.
• Signups are laggy because a human is in the loop (me, I'm the human)
• Our hub doesn't have enough memory to be usable with our data.
• Our example notebooks aren't tested in CI so they are often broken.

Because of this I think maintaining the existing setup has pretty limited utility, and may even be a bit misleading to folks. So I'm not particularly attached to maintaining continuity between the old and new setup (I think we've functionally failed at that already).

Things we need to do to move to a Kaggle / Colab setup that really works well:

• Finish the Dagster transition and just distribute data w/o a bespoke Python environment.
• Re-vamp our example notebooks to reflect the new situation and test them in CI.
• Publish datapackage.json files for each of our database outputs.

[TrentonBush]

The context of this problem seems like it will change significantly for the better once we move to distributing data instead of code, which we intend to do in the next few months.

Once we make that switch, will the 2i2c hardware limitations still be a concern? 6GB is not enough to run the whole ETL, but it seems like plenty to mess around with a SQLite database. CEMS processing runs in dask so is out-of-core capable, which should meet the needs of the few people who really want to dive into the weeds of full-granularity CEMS data.

Also who is the intended audience for this? Someone has to be comfortable with python/R/SQL but uncomfortable with downloading a SQLite db and working locally. I'm sure somebody is in the intersection of that Venn diagram but I can't see it being that big? I'm wondering if we should just keep 2i2c for now for the sake of simplicity, and defer the switching costs until hosting actually becomes expensive, if it ever does.

I agree that switching to a platform like Kaggle would be preferable, but in a world of budget priorities, I'd vote for spending this time fixing our data distribution system instead. I think that would be more appreciated by our users.

[bendnorman]

Thanks for doing all of this research @zaneselvans!

Imposing my bias on the user here, but I think the main goal of these hosted notebooks is to provide potential users a way to instantly play with PUDL data in a notebook. Like @TrentonBush said, I'm not convinced there is a large portion of potential users who want to interact with data in a notebook but aren't comfortable setting up a local python environment with pandas and jupyter. While exploring other python tools I'll typically quickly play with the tool or data in the binder or google collab notebook they provide. Once I'm convinced the tool solves my problem, I'll bring it into my own development environment. While it would be nice to have the ability to save the notebooks and data outputs and return to them later, I don't think it should be a top priority.

The manual signup process for 2i2c doesn't allow for instantaneous data exploration and binder resources are disappearing, which leaves us with Colab, Kaggle and JupyterLite. JupyterLite seems like a cool option because we'll never have to pay for resources or be stressed about free resources disappearing. Downloading 10 GB won't allow for instantaneous data exploration so I don't think it's a great option for us right now. If free resources like Colab and Kaggle dry up we could provide a smaller subset of the data on JupyterLite.

Between Colab and Kaggle, Kaggle seems to be more feature rich and adds PUDL to a very popular data registry. Both provide free resources that could be taken away (same as our AWS bucket). Assuming this access mode is for prospective users to do initial data exploration (they won't be using it on a recurring basis), I say we take advantage of the free resources while they're available. Pivoting to JupyterLite or a paid solution won't disrupt users' workflow.