Run pytest github action on a windows VM

[bendnorman]

Our users and contributors on windows are running into environmental issues. See #1742. Should we expand our tox-pytest action to run on a Windows VM in addition to an ubuntu VM? Have we tried this before @zaneselvans?

[zaneselvans]

Once upon a time, I think probably in response to @grgmiller's early eGrid work, we did make an effort to get all this working under Windows, but eventually I gave up. IIRC there were several issues:

• ogr2ogr which we use to convert the Census DP1 geodatabase to SQLite can't be installed on Windows using pip.
• IIRC there was another Windows incompatibility having to do with the way we access the filesystem structures inside zipfiles in the datastore archives, which we couldn't resolve.
• Debugging without an actual Windows machine to play with was extremely tedious, and none of us were familiar with Windows.
• We were (and are) moving away from expecting users to run the ETL.
• The Windows Subsystem for Linux exists and works. Using a development Docker container with VS Code also works.
• Most of the tools we're using aren't really designed to work on Windows and it seemed like a substantial amount of additional debugging and maintenance work to try and keep it all functional.

[bendnorman]

That all makes sense. I think we should update the docs to recommend windows users use WSL.

Our new docker image isn't really ideal for development because you need to rebuild the image after each code change. We could create another docker image that mounts the code instead.

@grgmiller @arengel @TrentonBush what do y'all think is best for Windows users?

[zaneselvans]

Yeah I think a separate pudl-dev image would definitely be the way to go, with a little docker-compose file to arrange things appropriately along the lines of this example from Arjan Codes.

I thought we already said to use WSL, but I guess we don't quite.

[arengel]

We have a few people on our team that use Windows and that need to use PUDL. But when I say use PUDL I don't mean to run the ETL, the functionality they use is more limited:

• Access the downloaded sqlite databases and 861 using PudlTabl.
• Process CEMS data using convert.epacems_to_parquet and output.epacems
  At the moment, we are able to install PUDL directly into a conda environment on Windows and these parts of PUDL work, at least to the best of my knowledge. (We use pip to install PUDL but have mamba handle installing most of the packages before installing PUDL because it's faster and I don't entirely trust pip.)

My preference would be to maintain this level of functionality on Windows directly using mamba/pip as we do now. The reason is that we have some team members on Windows that are new to Python, terminals and such and so it's nice to be able to avoid WSL, docker or anything fancy that I will have to support and troubleshoot for this limited use case.

For anything more serious, I think WSL and/or docker make sense.

Our issues do not justify the effort of getting PUDL fully functional on Windows and adding it to the tox-pytest action to ensure this is the case. If there's a way to very easily create an action that makes sure that PUDL and its dependencies are installable on Windows, that would be nice, but I realize that the fact that some of the dependencies must be installed by conda probably makes this a lot harder.

[zaneselvans]

It's also possible to have tox manage its environments using conda or mamba instead of pip and virtualenv via tox-conda, which would address some of these issues, and I've been tempted to do it before. I've hesitated because it wasn't obvious to me how to avoid needing to specify PUDL's requirements twice -- once in setup.py and then again in an environment.yml that would be used by mamba. And having all those packages specified in more than one place seemed like asking for stuff to get out of sync. The @dependabot also doesn't support updating environment.yml files.

I imagine it's doable, and might be a better arrangement, but whenever I look at it it seems like more work than I expected.

[grgmiller]

I think that if there were clear instructions on how to get WSL set up that would be helpful. I've never used Linux before so it sounds like there might be a bit of a learning curve, but I'm sure there are resources out there.

The only potential challenge is we are currently using some of the pudl analysis functions as a dependency for another project we are developing, and for development/debugging, it is really convenient to be able to reference my local version of the pudl code by installing it in my conda environment using a pip install of --editable ../pudl. I'm guessing that if my local copy of pudl was on WSL (and if I'm doing my other code development on windows instead of WSL), there would not be an easy way to do this, unless I pushed my changes from my WSL version of pudl to my fork, then pulled it to my local windows version of pudl.

[zaneselvans]

@TrentonBush is there a good guide to working with WSL / Python that you've referred to? It seems like there should be documentation that exists already that'll be more complete and up-to-date than anything we could cobble together.

Yeah I don't think you'd want to be working with a mix of Windows and Linux python environments. You'd probably want to look at running it all inside WSL. With many PyData things I think you'll find it easier to get everything working together in Linux, and with VS Code and Jupyter I don't think you'll be forced to really "use Linux" for much directly. It's just the code execution environment. But Trenton can definitely say more about what the experience is like as I've never used Windows.

[TrentonBush]

I've never used Linux before so it sounds like there might be a bit of a learning curve

WSL makes it so you don't have to interact with Linux very much if you don't want to. This lets you spread the learning curve out over a long and mostly voluntary timeline.

You can access/manage all your files (on both windows and linux) via the regular file explorer. Networking and disk I/O is all passed through so you don't have to configure anything to access local storage or the internet. Because VSCode is a windows product, it integrates nicely with WSL too (not sure if you use VSCode). As long as you're using pip/conda, I don't think there is much different on the python side of things.

I think the only things you would have to do in a linux shell are command line things like managing conda or starting the jupyter server. I very rarely run into something OS specific like file permissions. One performance note: transferring data between Linux/Windows via WSL is slower than reading it natively on either OS. This only matters if you have to read a lot of data (like EPA CEMS). But in that case you will probably want to move or copy the data over to the linux side instead of reading it across from Windows.

From what I remember, the main startup hurdle is simply remembering what existing code is written for the conda env in linux vs in windows. For me that problem resolved itself over time, because I put all new code on the linux side. But you could be more proactive than I was and deliberately move things over.

The official setup docs from Microsoft look good to me. The default install options (WSL2, Ubuntu) are the ones you want. My only additions would be:

1. bookmark ("pin to Quick access") the WSL filesystem in File Explorer so it is easier to navigate to. The path in File Explorer looks like \\wsl$\Ubuntu\home\<your_username>\
2. Redirect the WSL web browser (which, without additional setup, has neither a browser nor a means to display GUIs) to use your browser on the windows side. This is convenient when opening jupyter notebooks, for example. To do this:
   1. open /home/<your_username>/.bashrc in a text editor and add this line at the bottom: export BROWSER=/mnt/c/Program\ Files/Mozilla\ Firefox/firefox.exe (or whatever your browser path is. Use backslashes to escape spaces. The prefix /mnt/c is the linux path to the windows C drive).
   2. For jupyter specifically, run jupyter notebook --generate-config, then edit the .py file it creates. Uncomment the line referring to c.NotebookApp.use_redirect_file and set it to False. This works for both notebook and lab.