Incus unable to start/run (docker) container natively: 's6-overlay-suexec' error

dwlfrth · July 16, 2024, 8:04pm

Hello there

So I am trying to run a couple of (docker) containers natively in Incus. Some of them works but some others refuse to start…
After some investigation, it seems like the ones who are having issues are based with an ‘init environment’ ?
Anyway, here’s the error I get from the console.log after a start attempt

s6-overlay-suexec: fatal: can only run as pid 1

Is there anyway the behavior of the container can be overriden by some configuration flags that I am not aware of ?

Cheers !

simos · July 16, 2024, 8:14pm

Hi!

Can you post the name such an example image, preferably somewhat small in size?

dwlfrth · July 16, 2024, 8:41pm

Yup sure !

docker.io/pihole/pihole

simos · July 16, 2024, 8:53pm

I gave it a go. There was an issue when you use --console. Error in incus monitor --pretty output: err=“read /dev/pts/ptmx: input/output error”.

$ incus launch docker:pihole/pihole --ephemeral --console
Launching the instance
Instance name is: blessed-coyote                                  
To detach from the console, press: <ctrl>+a q
Error: Failed running forkconsole: "container is not running: \"blessed-coyote\""
                                                                                 Error: stat /proc/-1: no such file or directory
$ incus launch docker:pihole/pihole --ephemeral
Launching the instance
Instance name is: busy-stork
$

--show-log did not show any error to me.

dwlfrth · July 16, 2024, 8:59pm

It “launch” but quit right after. Maybe I should say “unable to use” rather than “unable to start” ?

Not sure what I should do of a container that exits right away !

dwlfrth · July 16, 2024, 9:13pm

Here’s what I get on my end

root@ipster:/var/log/incus# incus launch docker:pihole/pihole --ephemeral --storage=incus-sp
Launching the instance
Instance name is: splendid-jay

The instance you are starting doesn't have any network attached to it.
  To create a new network, use: incus network create
  To attach a network to an instance, use: incus network attach

root@ipster:/var/log/incus# cat splendid-jay/console.log 
s6-overlay-suexec: fatal: can only run as pid 1

simos · July 16, 2024, 9:15pm

One aspect with Docker support in Incus, is the ability to launch and not get any error message or something similar. I would call it the ephemeral test. You launch the OCI image, it does not get any instructions to keep it running, and most likely it stops. As soon as it stops, the instance gets deleted due to the –ephemeral flag.

$ incus launch docker:pihole/pihole --ephemeral
Launching the instance
Instance name is: busy-stork
$

In your case you would need to further add configuration similar to what is shown in @stgraber’s announcement. Like

stgraber@dakara:~$ incus launch docker:mysql mysql \
> -c environment.MYSQL_DATABASE=wordpress \
> -c environment.MYSQL_USER=wordpress \
> -c environment.MYSQL_PASSWORD=wordpress \
> -c environment.MYSQL_RANDOM_ROOT_PASSWORD=1
Launching mysql
...

I am not familiar this specific image. I believe the image just exited because it did not get any configuration.
https://hub.docker.com/r/pihole/pihole

dwlfrth · July 16, 2024, 9:19pm

I agree with the ephemeral concept
Thing is, with the error being thrown, it seems to me to be more than missing environment variables / configurations

I’ll give it a try with some values but some searches seems to point to init context not being allowed in Incus the same way it is for docker. Again, not all containers are quitting unexpectly, nginx works per example

Also, can you point me to the announcement in question ?

dwlfrth · July 16, 2024, 9:28pm

Again, not quite sure of the lead I am following but, something like this would probably “solve” the error.
(That is what is used to disable the default docker ‘init’ process injection with Dokku)

3.2. Disable default --init process injection:

dokku scheduler-docker-local:set pihole init-process false

stgraber · July 16, 2024, 9:55pm

Yeah, we may be able to add a property for that, or if there’s metadata in the container’s config.json that we can react to, we can do that too.

dwlfrth · July 16, 2024, 9:56pm

Hey @stgraber ! Glad to see you again !

So, am I on the right path or I’m completely lost here ?

stgraber · July 16, 2024, 10:13pm

stgraber · July 16, 2024, 10:14pm

That will handle the pihole init issue, but note that even though this lets me create a pihole container and login through the web UI, the FTL service it runs appears to be trying to lock memory in a way that’s not allowed in unprivileged containers, causing it to fail to start.

It may be a small tweak for the pihole folks to do to allow running in environments with less privileges than they’re used to though, so maybe something that can be reported to them and be sorted out there?

stgraber · July 16, 2024, 10:16pm

stgraber@dakara:~$ incus launch docker:pihole/pihole pihole -c environment.FTLCONF_LOCAL_IPV4=172.17.250.1 --console
Launching pihole
To detach from the console, press: <ctrl>+a q
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
        * CAP_NET_ADMIN
        * CAP_SYS_NICE
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf

  [i] Installing latest logrotate script...
	[i] Existing logrotate file found. No changes made.
  [i] Assigning random password: xw-N8TGS
  [✓] New password set
  [i] Added ENV to php:
                    "TZ" => "",
                    "PIHOLE_DOCKER_TAG" => "",
                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                    "CORS_HOSTS" => "",
                    "VIRTUAL_HOST" => "pihole",
  [i] Using IPv4 and IPv6

  [✓] Installing latest Cron script
  [i] setup_blocklists now setting default blocklists up: 
  [i] TIP: Use a docker volume for /etc/pihole/adlists.list if you want to customize for first boot
  [i] Blocklists (/etc/pihole/adlists.list) now set to:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [i] Existing DNS servers detected in setupVars.conf. Leaving them alone
  [i] Applying pihole-FTL.conf setting LOCAL_IPV4=172.17.250.1
  [i] FTL binding to default interface: eth0
  [i] Enabling Query Logging
  [i] Testing lighttpd config: Syntax OK
  [i] All config checks passed, cleared for startup ...
  [i] Docker start setup complete

  [i] pihole-FTL (no-daemon) will be started as pihole

s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
Stopping pihole-FTL
  [i] Neutrino emissions detected...
pihole-FTL: no process found
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [✓] Parsed 156669 exact domains and 0 ABP-style domains (ignored 1 non-domain entries)
      Sample of non-domain entries:
        - "0.0.0.0"

  [i] List has been updated

  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available
  [i] Number of gravity domains: 156669 (156669 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [i] FTL is not running
  [✓] Cleaning up stray matter
  [✓] Restarting DNS server

  [✗] DNS service is NOT running

Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
  Pi-hole version is v5.18.3 (Latest: v5.18.3)
  web version is v5.21 (Latest: v5.21)
  FTL version is v5.25.2 (Latest: v5.25.2)
  Container tag is: 2024.07.0

Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found
Stopping pihole-FTL
pihole-FTL: no process found

2024-07-16 22:15:44.104 1176M] Using log file /var/log/pihole/FTL.log
[2024-07-16 22:15:44.104 1176M] ########## FTL started on pihole! ##########
[2024-07-16 22:15:44.104 1176M] FTL branch: master
[2024-07-16 22:15:44.104 1176M] FTL version: v5.25.2
[2024-07-16 22:15:44.104 1176M] FTL commit: 8943e260
[2024-07-16 22:15:44.104 1176M] FTL date: 2024-05-08 20:59:50 +0100
[2024-07-16 22:15:44.104 1176M] FTL user: pihole
[2024-07-16 22:15:44.104 1176M] Compiled for x86_64 (compiled on CI) using gcc (Debian 8.3.0-6) 8.3.0
[2024-07-16 22:15:44.104 1176M] Starting config file parsing (/etc/pihole/pihole-FTL.conf)
[2024-07-16 22:15:44.104 1176M]    SOCKET_LISTENING: only local
[2024-07-16 22:15:44.104 1176M]    AAAA_QUERY_ANALYSIS: Show AAAA queries
[2024-07-16 22:15:44.104 1176M]    MAXDBDAYS: max age for stored queries is 365 days
[2024-07-16 22:15:44.104 1176M]    RESOLVE_IPV6: Resolve IPv6 addresses
[2024-07-16 22:15:44.104 1176M]    RESOLVE_IPV4: Resolve IPv4 addresses
[2024-07-16 22:15:44.104 1176M]    DBINTERVAL: saving to DB file every minute
[2024-07-16 22:15:44.104 1176M]    DBFILE: Using /etc/pihole/pihole-FTL.db
[2024-07-16 22:15:44.104 1176M]    MAXLOGAGE: Importing up to 24.0 hours of log data
[2024-07-16 22:15:44.104 1176M]    PRIVACYLEVEL: Set to 0
[2024-07-16 22:15:44.104 1176M]    IGNORE_LOCALHOST: Show queries from localhost
[2024-07-16 22:15:44.104 1176M]    BLOCKINGMODE: Null IPs for blocked domains
[2024-07-16 22:15:44.104 1176M]    ANALYZE_ONLY_A_AND_AAAA: Disabled. Analyzing all queries
[2024-07-16 22:15:44.104 1176M]    DBIMPORT: Importing history from database
[2024-07-16 22:15:44.104 1176M]    PIDFILE: Using /run/pihole-FTL.pid
[2024-07-16 22:15:44.104 1176M]    SOCKETFILE: Using /run/pihole/FTL.sock
[2024-07-16 22:15:44.104 1176M]    SETUPVARSFILE: Using /etc/pihole/setupVars.conf
[2024-07-16 22:15:44.104 1176M]    MACVENDORDB: Using /macvendor.db
[2024-07-16 22:15:44.104 1176M]    GRAVITYDB: Using /etc/pihole/gravity.db
[2024-07-16 22:15:44.104 1176M]    PARSE_ARP_CACHE: Active
[2024-07-16 22:15:44.104 1176M]    CNAME_DEEP_INSPECT: Active
[2024-07-16 22:15:44.104 1176M]    DELAY_STARTUP: No delay requested.
[2024-07-16 22:15:44.104 1176M]    BLOCK_ESNI: Enabled, blocking _esni.{blocked domain}
[2024-07-16 22:15:44.104 1176M]    NICE: Cannot change niceness to -10 (permission denied)
[2024-07-16 22:15:44.104 1176M]    MAXNETAGE: Removing IP addresses and host names from network table after 365 days
[2024-07-16 22:15:44.104 1176M]    NAMES_FROM_NETDB: Enabled, trying to get names from network database
[2024-07-16 22:15:44.104 1176M]    EDNS0_ECS: Overwrite client from ECS information
[2024-07-16 22:15:44.104 1176M]    REFRESH_HOSTNAMES: Periodically refreshing IPv4 names
[2024-07-16 22:15:44.105 1176M]    RATE_LIMIT: Rate-limiting client making more than 1000 queries in 60 seconds
[2024-07-16 22:15:44.105 1176M]    LOCAL_IPV4: Using IPv4 address 172.17.250.1 for pi.hole and hostname
[2024-07-16 22:15:44.105 1176M]    LOCAL_IPV6: Automatic interface-dependent detection of address
[2024-07-16 22:15:44.105 1176M]    BLOCK_IPV4: Automatic interface-dependent detection of address
[2024-07-16 22:15:44.105 1176M]    BLOCK_IPV6: Automatic interface-dependent detection of address
[2024-07-16 22:15:44.105 1176M]    SHOW_DNSSEC: Enabled, showing automatically generated DNSSEC queries
[2024-07-16 22:15:44.105 1176M]    MOZILLA_CANARY: Enabled
[2024-07-16 22:15:44.105 1176M]    PIHOLE_PTR: internal PTR generation enabled (pi.hole)
[2024-07-16 22:15:44.105 1176M]    ADDR2LINE: Enabled
[2024-07-16 22:15:44.105 1176M]    REPLY_WHEN_BUSY: Drop queries when the database is busy
[2024-07-16 22:15:44.105 1176M]    BLOCK_TTL: 2 seconds
[2024-07-16 22:15:44.105 1176M]    BLOCK_ICLOUD_PR: Enabled
[2024-07-16 22:15:44.105 1176M]    CHECK_LOAD: Enabled
[2024-07-16 22:15:44.105 1176M]    CHECK_SHMEM: Warning if shared-memory usage exceeds 90%
[2024-07-16 22:15:44.105 1176M]    CHECK_DISK: Warning if certain disk usage exceeds 90%
[2024-07-16 22:15:44.105 1176M] Finished config file parsing
[2024-07-16 22:15:44.105 1176M] FATAL: create_shm(): Failed to create shared memory object "FTL-lock": Permission denied
[2024-07-16 22:15:44.105 1176M] Initialization of shared memory failed.
root@pihole:~#

stgraber · July 16, 2024, 10:16pm

It could also be our fault as we mostly ignore the mount requests from OCI containers at this point

stgraber · July 16, 2024, 10:17pm

Yeah, so it’s asking for /dev/shm to be mounted as a tmpfs, that may be the problem. Let me try to manually do it.

stgraber · July 16, 2024, 10:19pm

Yep, that’s it. With my fix applied, doing incus exec pihole -- mount -t tmpfs tmpfs /dev/shm fixes it.

I’ll open an issue to have us come up with a mount hook which handles most of the common mount requests from config.json. We’re not going to be able to allow all of them for security reasons, but basic things like tmpfs are fine.

stgraber · July 16, 2024, 10:21pm

github.com/lxc/incus

Add (limited) support for mount requests from OCI config.json

opened 10:21PM - 16 Jul 24 UTC

stgraber

Bug Easy

Currently we're completely ignoring the mount section from `config.json`. Tha…t's obviously not exactly ideal as there may occasionally be reasonable requests in there which are actually required for a container :) We should add logic to ignore all the stuff we mount ourselves (/dev, /proc, /sys, /sys/fs/cgroup, /dev/mqueue, /dev/pts, ...) but then for the stuff that we're not handling, effectively turn it into liblxc config so long as it's something that looks safe (would probably initially restrict to tmpfs).

dwlfrth · July 16, 2024, 10:21pm

Awesome, as usual @stgraber
I’ll test that

stgraber · July 17, 2024, 12:34am

I’ve now added an extra commit to incusd/instance/lxc: Handle OCI containers using real PID1 by stgraber · Pull Request #1005 · lxc/incus · GitHub which handles the Mounts config.

stgraber@dakara:~$ incus launch docker:pihole/pihole pihole -c environment.FTLCONF_LOCAL_IPV4=172.17.250.1 --console
Launching pihole
To detach from the console, press: <ctrl>+a q
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
        * CAP_NET_ADMIN
        * CAP_SYS_NICE
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf

  [i] Installing latest logrotate script...
	[i] Existing logrotate file found. No changes made.
  [i] Assigning random password: ckClKhH7
  [✓] New password set
  [i] Added ENV to php:
                    "TZ" => "",
                    "PIHOLE_DOCKER_TAG" => "",
                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                    "CORS_HOSTS" => "",
                    "VIRTUAL_HOST" => "pihole",
  [i] Using IPv4 and IPv6

  [✓] Installing latest Cron script
  [i] setup_blocklists now setting default blocklists up: 
  [i] TIP: Use a docker volume for /etc/pihole/adlists.list if you want to customize for first boot
  [i] Blocklists (/etc/pihole/adlists.list) now set to:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [i] Existing DNS servers detected in setupVars.conf. Leaving them alone
  [i] Applying pihole-FTL.conf setting LOCAL_IPV4=172.17.250.1
  [i] FTL binding to default interface: eth0
  [i] Enabling Query Logging
  [i] Testing lighttpd config: Syntax OK
  [i] All config checks passed, cleared for startup ...
  [i] Docker start setup complete

  [i] pihole-FTL (no-daemon) will be started as pihole

s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
  [✓] Parsed 156669 exact domains and 0 ABP-style domains (ignored 1 non-domain entries)
      Sample of non-domain entries:
        - "0.0.0.0"

  [i] List has been updated

  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available
  [i] Number of gravity domains: 156669 (156669 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [i] Pi-hole blocking will be enabled
  [i] Enabling blocking
  [✓] Reloading DNS lists
  [✓] Pi-hole Enabled

  Pi-hole version is v5.18.3 (Latest: v5.18.3)
  web version is v5.21 (Latest: v5.21)
  FTL version is v5.25.2 (Latest: v5.25.2)
  Container tag is: 2024.07.0

So looks like that’s taking care of the pihole issue at least