Currently I’m running some services though Docker on a Proxmox VM. Before I had Proxmox, I thought containers were a very clean way of organizing my system. I’m currently wondering if I can just install the services I always use on the VM directly. What are the pros and cons of that?
You must log in or register to comment.
Pros:
- better validation of installation and content, as long as you’re otherwise set for that.
- easier setup
- easier management
- easier teardown
- simpler firewalling
Cons: (vs docker-on-vm)
- your homelab isn’t someone’s resume
deleted by creator
Always use containers.
I wouldn’t recommend it
Containers are just processes with flags. Those flags isolate the process’s filesystem, memory [1], etc.
The advantages of containers is that the software dependencies can be unique per container and not conflict with others. There are no significant disadvantages.
Without containers, if software A has the same dependency as software B but need different versions of that dependency, you’ll have issues.
[1] These all depend on how the containers are configured. These are not hard isolation but better than just running on the bare OS.
Thanks for this - the one advantage I’m noticing is that to update the services I’m running, I have to rebuild the container. I can’t really just update from the UI if an update is available. I can do it, it is just somewhat of a nuisance.
How often are there issues with dependencies? Is that a problem with a lot of software these days?
But rebuilding your container is pretty trivial from the command line all said and done. I have something like this alias’d in my .bashrc to smooth it along:
Docker compose pull; docker compose down; docker compose up -d
I regularly check on my systems and go through my docker dirs and run my alias to update everything fairly simply. Add in periodic schedule image cleanups and it has been humming along for a couple years for the most part (aside from one odd software issues and hardware failures).
How often are there issues with dependencies? Is that a problem with a lot of software these days?
I started using docker 3-4 years ago specifically because I kept having issues with dependencies of one app breaking others, but I also tend to run a lot of services per VM. Honestly, the overhead of container management is infinitely preferable to the overhead that comes with managing OS level stuff. But I’m also not a Linux expert, so take that for what you will.
Is there a specific reason you’re taking the services down before bringing them back up? Just
docker compose pull && docker compose up -drecreates all services that had a new image pulled, but leaves the others running.Probably just a hold over from when I was first learning. Had issues with a couple services not actually updating without it, so I just do it to be absolutely sure. Also, I only ever run one app per compose, so that forces a “reboot” of the whole stack when I update.
I know rebuilding containers is trivial, but updating a service in the UI is more trivial than that. I’m just trying to make my life as trivial as possible 😁. It seems like containers may be worth the little bit of extra effort.
I mean, for anything where you’re willing to trust the container provider not to push breaking changes, you can just run Watchtower and have it automatically update. That’s how most of my stuff runs.
If you’re not using some sort of automatic updates, you’re not too seriously trying to make your life as trivial as possible. 😂 Just use fixed major version tags where possible in order to avoid surprise breakage.
I beg to disagree about the disadvantages. An important one is that you cannot easily update shared libraries globally. This is a problem with things like libssl or similar. Another disadvantage is the added complexity both wrt. to operation but also in general the amount of code running. It can also be problematic that many people just run containers without doing any auditing. In general containers are pretty opaque compared to os packaged software which is usually compiled individually for the os.
This being said, systemd offers a lot of isolation features that allows similar isolation to containers but without having to deal with docker.
I see containers as having a couple of advantages:
- Separation of dependencies - while not as big of issue as it used to be, just knowing that you won’t end up with the requirements for one application conflicting with another is one less issue to worry about. Additionally, you can do anything you want to one container, without having an effect on another container. You don’t get stuck wanting to reboot or revert the system, but not wanting to break a different running service.
- Portability - Eventually, you are going to replace the OS of that VM (at least, you should). Moving a container to a new OS is dead simple. Re-installing an application on a new OS, moving data and configs can be anywhere from easy to a pain in the arse, depending on the software.
- Easier fall back - Have you ever upgraded an application and had everything go to shit? In my years working as a sysadmin, I lost way too many evenings to this sort of bullshit. And while VM snapshots should make reverting easy, sometimes it just didn’t work out that way. Containers force enough separation of applications that you can do just about anything to one container and not effect others.
- Less dependency on a single install - Have you ever had a system just get FUBAR, and after a few hours of digging the answer seems to be, just format the drive and start over? Maybe you tried some weird application out and the uninstall wasn’t really clean. By having all that crap happen in containers, you can isolate the damage. Nuke the container, nuke the image, and the base OS is still clean.
- Easier version testing - Want to try out upgrading to version 2 of an application, but worried that it may not be fully baked yet or the new configs may take a while to get right? Do it off in a separate container on a copy of the data. You can do this with VMs and snapshots; but, I find containers to be less overhead.
That all said, if an application does not have an official container image, the added complexity of creating and maintaining your own image can be a significant downside. One of my use cases for containers is running game servers (e.g. Valheim). There isn’t an official image; so, I had to roll my own. The effort to set this up isn’t zero and, when trying to sort out an image for a new game, it does take me a while before I can start playing. And those images need to be updated when a new version of the game releases. Technically, you can update a running container in a lot of cases; but, I usually end up rebuilding it at some point anyway.
I’d also note that, careful use of VMs and snapshots can replicate or mitigate most of the advantages I listed. I’ve done both (decade and a half as a sysadmin). But, part of that “careful use” usually meant spinning up a new VM for each application. Putting multiple applications on the same OS install was usually asking for trouble. Eventually, one of the applications would get borked and having the flexibility to just nuke the whole install saved a lot of time and effort. Going with containers removed the need to nuke the OS along with the application to get a similar effect.
At the end of the day, though. It’s your box, you do what you are most comfortable with and want to support. If that’s a monolithic install, then go for it. While I, or other might find containers a better answer for us, maybe it isn’t for you.
Man back when I played there was a community image at least
I’m sure there are several out there. But, when I was starting out, I didn’t see one and just rolled my own. The process was general enough that I’ve been able to mostly just replace the SteamID of the game in the Dockerfile and have it work well for other games. It doesn’t do anything fancy like automatic updating; but, it works and doesn’t need anything special.
Cons of containers are slightly worse disk and memory consumption.
Pros:
- ease of installation
- declarative configuration
- security
- dependency management is solved
Stick with the containers
security
with containers, software maintainers also need to keep their image up-to-date with latest security fixes (most of them don’t) - whereas these are usually handled by unattended-upgrades or similar in a VM. Then put out a new release and expect users to upgrade ASAP. Or rebuild and encourage redeploying the
latestimage every day or so, which is bad for other reasons (no warning for breaking changes, the software must be tested thoroughly after every commit tomaster).In short this adds the burden of proper OS/image maintenance for developers, something usually handled by distro maintainers.
trivy is helpful in assessing the maintenance/vulnerability level of OCI images.
Wait, ease of installation? As someone who had to walk away from a semi-homebrew, mildly complicated cloud storage setup recently, that’s not the experience I had. Networks within networks, networks next to networks not talking to each other, mapped volumes, even checking logs is made more complicated by containerising. Sure, I’m a noob, but that only reinforces my point.
I definitely see your point, but the difference is that it’s one thing to learn. Once you know docker, you can deploy and manage anything.
Just to throw another option in: Lxc are containers too. And they are the other major option proxmox comes with.
It feels more like bare metal installations, but are more lightweight and share there ressources they do not use.
I never got why having Proxmox and one VM with several docker containers except I absolutly don’t want to deal with installations at all.
On the other hands I wanted to learn about linux and the basics of handling proxmox.
I wish the phrase “bare metal” would die…
I’ve been asking myself the same question for a while. The container inside a VM is my setup too. It feels like the container in the VM in the OS is a bit of an onion approach which has pros and cons. If u are on low powered hardware, I suspect having too many onion layers just eat up the little resources you have. On the other hand, as Scott@lem.free.as suggests, it easier to run a system, update and generally maintain. It would be good to have other opinion on this. Note that not all those that have a home lab have good powered labs. I’m still using two T110’s (32GB ECC ram) that are now quite dated but are sufficient for my uses. They have Truenas scale installed and one VM running 6 containers. It’s not fast, but its realiable.
Container overhead is near zero. They are not virtualized or anything like that, they are just processes on your host system that are isolated. Its functionally not much more different to chroot.
Personally, I always like to use containers when possible. Keep in mind that unlike virts, containers have very minimal overhead. So there really is no practical cost to using them, and they provide better (though not perfect) security and some amount of sandboxing for every application.
Containers mean that you never have to worry about whether your VM is running the right versions of certain libraries. You never have to be afraid of breaking your setup by running a software update. They’re simpler, more robust and more reliable. There are almost no practical arguments against using them.
And if you’re running multiple services the advantages only multiply because now you no longer have to worry about running a bespoke environment for each service just to avoid conflicts.
Copying a response I wrote on another comment -
Thanks for this - the one advantage I’m noticing is that to update the services I’m running, I have to rebuild the container. I can’t really just update from the UI if an update is available. I can do it, it is just somewhat of a nuisance.
How often are there issues with dependencies? Is that a problem with a lot of software these days?
Ok but containers generally have a lot less dependencies. If you are making your own images, then you know exactly how to rebuild them. In the event something happens, it makes it much easier to get up and running again and also remember what you did to get the service running. The only other thing that would be better is Nix.
If you use an image that someone is maintaining, this makes it even easier and there are services out there that will keep your containers up to date when a new image is available. You can also just automate your image builds to run nightly and keep it up to date.
There’s no good answer to that because it depends entirely on what you’re running. In a magical world where every open source project always uses the latest versions of everything while also maintaining extensive backwards compatibility, it would never be a problem. And I would finally get my unicorn and rainbows would cure cancer.
In practice, containers provide a layer of insurance that it just makes no sense to go without.
