Why I Still Use Python Virtual Environments in Docker
Whenever I publish something about my Python Docker workflows, I invariably get challenged about whether it makes sense to use virtual environments in Docker containers. As always, it’s a trade-off, and I err on the side of standards and predictability.
You're viewing a single thread.
I started to use Nix to build containers that contain just my app and nothing else. The benefit of it is that it makes containers smaller, removes unused components (less potential attack vectors) and a container from a specific checked out version will always be identical (Dockerfile on its own (without extra work) doesn't provide such guarantee). I also have the ability to customize python and dependencies to remove additional pieces that I don't need (this unfortunately requires some experience with Nix, to know how to do it)
I wrote my own abstraction on top of poetry2nix and nix2container to remove need for boilerplate: https://github.com/takeda/nix-cde
The example shows how a hello world application can be packed and then how I can reduce its size further from 178MB to 68.9MB. This doesn't include using musl to get the size even lower than that.
Though I totally agree with author about venv and that's what I did before and still do in situations where I can't use Nix. Venv is standardized and is much more predictable and prevents surprises.
The team I'm part of wants to ditch Nix in favour of just about anything, because no one wants to maintain Nix and everyone sees it as just source of problems :(
I agree that it was complicated to learn Nix for me, too, but now I see benefits in it but I can't make them change their mind and tired of trying. Nix could've been much easier to advocate for if the language itself wasn't this esoteric
I see that too. Despite what most people say they aren't truly interested in learning new things (at least things that would force them out of their comfort zones).
I mean if team tries to move out then there's not much one can do.
Maybe they can look into using some tooling that whole isn't nix, it uses nix under the hood and still prices some benefits.
I heard about DevBox and Flox. Those at least try to provide a reproducible dev environment (note, I haven't used them myself as I feel that the abstraction they do places limits on nix functionality, but then others might see it as a benefit)
I also am getting impression that as time progresses things are getting smoother over time. With poetry2nix for example the big problem are packages that depend on C libraries, as those are not specified as python dependencies, so poetry2nix has a override file which adds them.
Previously I very frequently had to update and contribute new packages there. I was a bit away from python as was assigned to work on a Go project for half a year and now starting to work on another python project and when tried to use it and things just worked. All I had to do was to use latest poetry2nix and my project then compiled to a working container.
The dockerfile does not guarantee this, but the docker image or any OCI image does.
That's true, but also misleading.
OCI image is like having an jpeg image. While Dockerfile is like the text prompt you write to ChatGPT to generate the image.
Yes every time you look at the jpeg, it is the same exact image, but that's kind of obvious, the real problem is if you try the text query to ChatGPT you will get something slightly different every time.
Nix brings a true reproducibility. So in this analogy the same prompt brings the exact same image. This allows you to check on that prompt in your source control and if you mess up something there's always a way back.
This is something docker promised, but never delivered.
Dockerfile should not be confused with the artifact.
It should not, but artifacts never had problem with mutating before we had docker. If you generate an rpm package and store it in an artifactory it always was the same exact package (unless someone overwrote it, lol)
Operationally we usually expect a dockerfile to be identical across many builds of different releases and know the artifact produced will have different code
But that's basically the problem docker claimed to fix. This is also the problem that you frequently encounter with a pipeline that worked fine one day suddenly stopped working next day, because something that your Dockerfile referenced changed (maybe a new image was updated that broke something, you can lock things to specific hashes, but you need to be very conscious about that and in the wild I never seen anyone really doing it).
Anything you are doing with nix to make the lock files perfect is the same amount of work you’d be doing to any method of producing an OCI artifact.
It is not. Hashes are and lock files are built-in and Nix uses them by default.
If for example I use a flake, the flake.lock will hold the exact version of nixpkgs (package repo) in time. That happens without any additional effort. The poetry2nix converts poetry.lock file to nix packages that are once again locked in time, and that also happens behind the scenes.
The result is that all dependencies (python dependencies - from poetry.lock as well as the rest of the system (python, c libraries etc) - from flake.lock are all locked and in my repo. So everything is repeatable without effort on my side.
To repeat that with Dockerfile is much more challenging.
I do think your approach is interesting though. Certainly less effort than manually packing an OCI with something like buildpaks or trying to run through bazel to get your way through a distroless build (two other methods that don’t make massive images with a Debian base). And obviously ‘From:scratch’ in docker build land is a nightmare.
If you get your app build with Nix. The whole thing, including all of app's dependencies are explicitly referenced so you can wrap it into a docker, an rpm file, OS image etc.
It's controversial, but IMO nix is actually easier than what we are doing now. I think the problem is that it is a massive paradigm shift and what most people know what to do with existing technologies will generally be not useful, so you have to relearn everything.
But IMO it pays off. For example when starting a new project I can package the whole thing in 5 minutes. poetry2nix translates the project and it's dependencies into nix packages and then since nix understands dependencies for my project it can package it automatically.
You make a good point in that Docker promised to make dev environments reproducible so that everyone on the team would have the same environment. They even succeeded in that, but either intentionally or accidentally omitted reproducibility over time due to the introduction of non-locked dependencies.
It does. it does to this. That’s the docker image not the docker file. You are confusing the spec with the artifact. If you want reproducible dev envs you use a system like compose or any rad of other tools to launch images from your artifact store.
You use them, make sure they are always pristine and cleaned after use, don't have network connectivity and other things that could affect the build.
Or you could use Nix which builds everything this way.
Notice that you mentioned additional systems to achieve that, you wouldn't need them if docker was truly providing it.
LOL. We always have this problem if you have people only using spec files and not the artifacts. You are comparing apples to oranges by comparing the dockerfile to a build rpm package. Let me help you:
An rpm package == docker image
An rpm .spec file == dockerfile
You if you only give people spec files and have them rebuild the package you will get different hashes of the rpm file. Similarly you would likely not change your spec file between releases and know your rpm file is going to be different.
But that's the whole point. A developer wants spec file to ALWAYS generate the same artifact. And most devs even believe that and get frustrated when it doesn't (like in your example).
Nix basically solves that. It even removes the need for tools like artifactory, because there's no longer need for it. The code fully defines the final binary. Of course you don't want to rebuild everything every time, so a cache is introduced.
Before you say that it is just renaming artifactory. It really isn't. It actually works like a cache. I can remove any piece of it, and the missing pieces will be rebuild if they are needed. It is also used by the builder, so it doesn't repeat itself. I especially like it when working on feature branch and it completes the code. I eventually merge it, and if my merge did not modify code it won't waste time rebuilding the same thing.