Using python virtual environments for a clean and easy dbt development experience across multiple projects and versions

[gwenwindflower]

As we've shifted towards recommending folks install adapter specific packages with pip, it will be extremely worthwhile to make a high quality guide to installing dbt inside of a virtual environment within your project. Global python situations can become a nightmare, I find it much better to not install global binaries when it can be avoided!

via archived developer blog repo

[VersusFacit]

Will you be recommending basic virtual envs, tools like pyenv which manage multiple python packages, or some other tool? There's a lot of different ways this can go. I find for more tricky things like adapter testing, pyenv is a lifesaver. The average user could probably get by with a basic virtual environment. That said, I've seen cases where people use them to install but don't actually use the separation feature. Practically, they'll use one virtual env for everything that's always active rather than activating and deactivating them when jumping around their dev machines. This leads to all sorts of compatibility issues almost as if they'd never used on a venv in the first place.

Just some thoughts I've been having I do more development with dbt and alllll the adapters.

[gwenwindflower]

yep, good q, i'm imagining this leaning towards my setup (the correct way ofc 😛) which is to use pyenv for managing your global python versions and make switching that up easy, but basic python -m venv etc virtual environments per project for actually doing work, installing dependencies like dbt, etc. completely agree that the pattern i've seen of people using "global" virtual environments to manage different versions of dbt feels bad/antipattern/more confusing than just using actual global binaries , and is extremely limiting if you have more dependencies than just dbt.

i will be leaning heavily on diagramming the concepts here and explaining this really, really clearly vs 'type this code and do this thing'. the goal here is actual deep understanding, there are plenty half decent guides on doing this stuff in a way that will work, few that i think make virtual environments really click. we want to write the latter.

[VersusFacit]

Love the approach and the desire to actually instruct about environments as project dependency containers. For what it's worth, I am a pyenv virtualenv person since it's all in one integrated, and I really never want to go back to manually managing python -m venv virtual environments because it requires you -- as far as I know, maybe I'm just a crappy user 😅 -- to specify full paths to activate/deactivate [which is tooooo much overhead for me, even with aliases]. I use virtual environments per project too :D

I'll bow out so as not to clog the airways. But also, I'm happy to read over and be a CLI editor of sorts if you ever need it. Tooling like this is a favorite thing of mine 😊 Will love to see what you put together 💥

[jaypeedevlin]

I’m still wrapping me head around how this works — is this something you’re accepting submissions on?

I’d love to write something on dbtenv, which we built to solve the same problem.