From Docker to Podman; how it started

For years, I ran my containerized services with Docker, still as root. I know, crazy, right? But "back in the day," the concept of rootless Docker was just as wild as rootfull docker is now. After a while, though, I decided it was time to move to something more modern and secure. My plan was to migrate my container stack to Podman.

This is a breakdown of the challenges I faced and, more importantly, the solution I found to one of the biggest roadblocks.

My Migration Setup

My home lab runs on a Proxmox host. I've been running my Docker stack in a dedicated Linux Container (LXC). For this new endeavor, I spawned a fresh LXC to be the new home for my Podman stack.

I ran into my first obstacle almost immediately. I couldn't get any containers to spawn using Podman as a regular user inside the LXC. I eventually learned from a Proxmox forum that I needed to use a privileged LXC for this to work. This might be a Topic for further investigations to figure out what exact prerequisits are out for podman to run on a LXC.

The Podman-Compose Conundrum

With a privileged LXC in place, I moved my docker-compose stack over to Podman. I was pleasantly surprised that it started up with relatively minor adjustments. My initial stack consisted of Portainer, a Speedtest Tracker, and a container for some network tools. This was a perfect setup for my tests.

This is when I discovered an important distinction: there is a difference between podman compose and podman-compose. The former is a newer, integrated command, while the latter is a community-maintained Python utility. I opted for podman-compose since, in my experience, it acted as a more direct drop-in replacement for docker-compose.

The Annoying Reboot Problem

After getting everything up and running, I stumbled upon a major annoyance: after every reboot, my entire container stack was dead. My containers weren't starting up automatically.

I spent a lot of time with a friend trying to fix this. We found other people online describing the same problem, but with no clear solution. We considered using systemd to manage the services, but we didn't initially think about using it at the user level.

This is a key difference between Podman and Docker. Unlike Docker, Podman is daemonless. It doesn't have a background process constantly running to manage services. To have a daemonless and rootless Podman stack that can survive a reboot, you need something to start and manage those services for you. The solution was to use systemd at the user level.

The Systemd Solution

I needed to create a systemd user service to manage my Podman stack. The first step was to create the service file in my home directory.

Bash
vim.tiny /home/volker/.config/systemd/user/podman-compose.service

After doing some extensive research, I came up with the following configuration.

[Unit]
Description=Podman via podman-compose to fix reboot issue
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Environment=PODMAN_USERNS=keep-id
Restart=always
TimeoutStartSec=50
TimeoutStopSec=50
ExecStart=/usr/bin/podman-compose up --remove-orphans
ExecStop=/usr/bin/podman-compose stop
Type=simple
WorkingDirectory=%h

[Install]
WantedBy=default.target

Let's break down a few of the important lines in this file:

Wants=network-online.target #This ensures that the service waits until the network is active before it tries to start. This is crucial for containers that need internet access.

ExecStart=/usr/bin/podman-compose up --remove-orphans #This is the core command that starts your container stack.

WorkingDirectory=%h #This tells the service to run the command from my home directory, which is where my compose file is located.

Of course, just creating the file wasn't enough. I still needed to tell systemd to start and enable the new service.

Bash

# This is a critical step for user-level services in some environments
export XDG_RUNTIME_DIR=/run/user/$UID

# Reload systemd to recognize the new service file
systemctl --user daemon-reload

# Start the service
systemctl --user start podman-compose

# Enable the service to start on every boot
systemctl --user enable podman-compose

I hope my little journey helps someone else out there facing the same issue. With these steps, I was able to successfully migrate my container stack and have it automatically start up after every reboot.

For more details on systemd units and their options, the official documentation was a great help.

I start to form opinions on why one would replace a daemon that does "one thing well" with another daemon thst does "everything" from starting the kernel to loging access to migration of Homedirectories. But apparently this might be the way to go, who cares that i am sceptical.