Hi all,
waiting for the publication of the docker/incus translator, i’m facing some issue that i can’t solve.
Apparently almost all my problems seem related to the /var/run directory
For example netalertx doesn’t start with this error
ERROR: unable to bind listening socket for address '/run/php/php8.3-fpm.sock': No such file or directory (2)
[...]
[04-Feb-2025 16:51:55] ERROR: FPM initialization failed
setpgid: Operation not permitted
nginx: [emerg] open() "/run/nginx/nginx.pid" failed (2: No such file or directory)
Paperless-ngx refuses to start with that error message
To detach from the console, press: <ctrl>+a q
Waiting for Redis...
Connected to Redis broker.
Apply database migrations...
Operations to perform:
Apply all migrations: account, admin, auditlog, auth, authtoken, contenttypes, django_celery_results, documents, guardian, mfa, paperless, paperless_mail, sessions, socialaccount
Running migrations:
No migrations to apply.
Running Django checks
System check identified no issues (0 silenced).
Executing /usr/local/bin/paperless_cmd.sh
Error: The directory named as part of the path /var/run/supervisord/supervisord.pid does not exist
For help, use /usr/local/bin/supervisord -h
Same images are running well into an Alpine container.
incus start paperless-ngx --console
To detach from the console, press: <ctrl>+a q
Waiting for Redis...
Connected to Redis broker.
Apply database migrations...
Operations to perform:
Apply all migrations: account, admin, auditlog, auth, authtoken, contenttypes, django_celery_results, documents, guardian, mfa, paperless, paperless_mail, sessions, socialaccount
Running migrations:
No migrations to apply.
Running Django checks
System check identified no issues (0 silenced).
Executing /usr/local/bin/paperless_cmd.sh
Error: The directory named as part of the path /var/run/supervisord/supervisord.pid does not exist
For help, use /usr/local/bin/supervisord -h
The current configuration is messed up by my failed attempt.
It looks like docker images built on alpine work well, while those built on Debian suffer the /run or /var/run issue
The handling of /run in application containers is a bit of a mess overall…
We’ve seen cases where if /run is persistent, things have a tendency to only work properly once, then as soon as the container restarts or crashes, the leftover files break everything.
So we’re respecting standard Linux setups and mount /run as a tmpfs, but that then causes some images, like the ones you’re dealing with to fail because they don’t create intermediary paths on boot (in your case, /run/nginx, /run/php and /run/supervisord) which then causes startup failures…
This is all a bit frustrating as there doesn’t appear to really be a good answer to this.
I think our behavior of making /run a tmpfs as it normally is on Linux is probably the correct move, but that means that on occasion fixes to images will be needed…
Any chance you can have the netalertx image have its startup script or entrypoint perform:
mkdir -p /run/nginx /run/php
That won’t affect any environments where the paths already exist and should fix running in environment where /run is a tmpfs.
Similarly for paperless-ngx, this one would need a mkdir -p /var/run/supervisord which should similarly take care of this issue.
I think it should be easily enough to justify as nobody should really expect anything to persist in /run, /var/run (symlink to /run) or /tmp on a Linux system. So it should be good project hygiene to ensure that any needed paths in there is created on startup, even if other container runtimes don’t hit this issue.
The handling of /run in application containers is a bit of a mess overall…
As much as I truly hate to say this, may I suggest adding an option to mount /run as tmpfs-supported overlayfs?
I really wish Docker would just mount tmpfs on /run by default but it doesn’t; and those broken images are now everywhere it seems.
To add a bit of context to this for casual readers, /run is meant to be cleared at boot time on full-fledged distro. There are many good reasons for that (esp. it’s much easier to start from a clean slate at boot rather than trying to figure out whether some random files in /run are actually artifacts from a previous boot, which is impossible to guarantee). Most distros just mount tmpfs over /run and link the (legacy) /var/run folder to it: it’s in RAM, it naturally gets cleared on reboots and it doesn’t grow much, being meant for minimal runtime info and unix sockets.
Container images like those above prepopulate /run. As init is fully controlled and only the relevant service is started anyway, this can be made to “just work”, especially given Docker’s propensity to use overlayfs pretty much everywhere. But it’s not clean and breaks the like of Incus for no real reason (the alternative, populating /run in the entrypoint.sh script, is as easy). There is no real solution for Incus: either implement the workaround above (or something along those lines) or mount tmpfs at the cost of ignoring what is in there (current solution) or wipe /run on boot (which would make the container fail the second time it’s started).
The third alternative, at least for Paperless, is to redefine the relevant environment variable to put the offending pidfile in a folder that is guaranteed to exist (e.g. directly under /run).
to netalertx/init, then rebuilt container with netalertx-dev image to fix a setpgid issue.
I don’t know if there is some more general and elegant solution, but rebuilding docker images after add the mkdir commands to entrypoint didn’t work for me.
Today I learned other lessons, the most relevant one is that this is definitively not my game, however I’ll keep playing.
We certainly could do the overlayfs with a tmpfs underlay trick, though I don’t know how easy it is to get liblxc to set that up for us and Incus itself doesn’t really have a good mechanism for doing complex mounts outside of what liblxc supports. We really want any tmpfs mount to originate from within the container as if it’s not performed by a process within the container, the memory usage of that tmpfs will not be accounted within the container’s allocation, allowing for the container to potentially cause a DoS through tmpfs writes (or at least exceed its memory allocation).
Maybe I’m a bit too optimistic here, but so far for the stuff I’ve been running, I’ve used the approach of sending any needed entrypoint fixes to the relevant upstream repo and have generally not had any pushback in making their logic behave.
but I’m not familiar with lxc’s capabilities and if people are willing to fix their images, the existing approach (a plain, simple tmpfs over /run) is just superior!
If people are receptive to amending their entrypoints, it’s for the best!