Port Mapping Issue with Incus OCI Container

Incus
1

@stgraber I am trying to map a port to my Incus server from an Incus OCI container and I am getting the following error:

 scott  demo-1  ~  incus list
+--------+---------+--------------------+-----------------------------------------------+-----------------+-----------+
|  NAME  |  STATE  |        IPV4        |                     IPV6                      |      TYPE       | SNAPSHOTS |
+--------+---------+--------------------+-----------------------------------------------+-----------------+-----------+
| Draw-3 | RUNNING | 10.85.91.32 (eth0) | fd42:d6fd:c813:b493:216:3eff:fec4:dab4 (eth0) | CONTAINER (APP) | 0         |
+--------+---------+--------------------+-----------------------------------------------+-----------------+-----------+

 scott  demo-1  ~  incus profile create proxy-8080
Profile proxy-8080 created

 scott  demo-1  ~  incus profile device add proxy-8080 hostport8080 proxy connect="tcp:127.0.0.1:8080" listen="tcp:0.0.0.0:8080" 
Device hostport8080 added to proxy-8080

 scott  demo-1  ~  incus profile add Draw-3 proxy-8080 
Error: Failed to start device "hostport8080": Error occurred when starting proxy device: Error: Permission denied - Failed setns to connector network namespace

 scott  demo-1  ~  incus stop Draw-3

 scott  demo-1  ~  incus profile add Draw-3 proxy-8080 
Profile proxy-8080 added to Draw-3

 scott  demo-1  ~  incus start Draw-3
Error: Error occurred when starting proxy device: Error: Permission denied - Failed setns to connector network namespace
Try `incus info --show-log Draw-3` for more info

 scott  demo-1  ~  incus info --show-log Draw-3
Name: Draw-3
Status: STOPPED
Type: container (application)
Architecture: x86_64
Created: 2024/07/22 19:04 UTC
Last Used: 2024/07/22 19:19 UTC

Log:

lxc Draw-3 20240722191907.530 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc Draw-3 20240722191907.530 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc Draw-3 20240722191907.531 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc Draw-3 20240722191907.531 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
2

That’s interesting, it may be an issue because of what we do with the DHCP client, causing some kind of startup race. I’ll have to try it here to see if I can reproduce it and sort that out.

3

Thanks. Let me know if you want me to provide additional information.

4

Sadly no luck so far…

stgraber@dakara:~$ incus launch docker:nginx nginx
Launching nginx
stgraber@dakara:~$ incus profile create proxy-8080
Profile proxy-8080 created
stgraber@dakara:~$ incus profile device add proxy-8080 hostport8080 proxy connect="tcp:127.0.0.1:8080" listen="tcp:0.0.0.0:8080"
Device hostport8080 added to proxy-8080
stgraber@dakara:~$ incus stop nginx
stgraber@dakara:~$ incus profile add nginx proxy-8080 
Profile proxy-8080 added to nginx
stgraber@dakara:~$ incus start nginx
stgraber@dakara:~$

What Docker image are you using?
And what package of Incus are you using on what distro?

5

The host server OS is Ubuntu 24.04 LTS. Incus is the latest patch of 6.3. I am using this image:

docker:ghost:alpine
6

Hmm, so here this image fails to start with a database error, I wonder if that’s also what’s hitting you, causing the container to be dead by the time the proxy device starts?

7 
[2024-07-25 21:22:16] INFO Ghost is running in production...
[2024-07-25 21:22:16] INFO Your site is now available on http://localhost:2368/
[2024-07-25 21:22:16] INFO Ctrl+C to shut down
[2024-07-25 21:22:16] INFO Ghost server started in 0.466s
[2024-07-25 21:22:16] ERROR connect ECONNREFUSED 127.0.0.1:3306

connect ECONNREFUSED 127.0.0.1:3306

"Unknown database error"

Error ID:
    500

Error Code: 
    ECONNREFUSED

----------------------------------------

Error: connect ECONNREFUSED 127.0.0.1:3306
    at /var/lib/ghost/versions/5.88.1/node_modules/knex-migrator/lib/database.js:57:19
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16)

[2024-07-25 21:22:16] WARN Ghost is shutting down
[2024-07-25 21:22:16] WARN Ghost has shut down
[2024-07-25 21:22:16] WARN Your site is now offline
[2024-07-25 21:22:16] WARN Ghost was running for a few seconds
8

Ah, passing NODE_ENV=development takes care of that one.

9 
stgraber@dakara:~$ incus create docker:ghost:alpine ghost -c environment.NODE_ENV=development
Creating ghost
stgraber@dakara:~$ incus config device add ghost hostport8080 proxy connect="tcp:127.0.0.1:2368" listen="tcp:0.0.0.0:8080"
Device hostport8080 added to ghost
stgraber@dakara:~$ incus start ghost

That gives me a working website on http://127.0.0.1:8080

10

The thing that occurs to me here is that we use your environment variable parameter to pass variables:

incus launch docker:ghost:alpine  -c environment.NODE_ENV=development

Then we use a device proxy to remap a port:

incus profile create proxy-8080
incus profile device add proxy-8080 hostport8080 proxy connect="tcp:127.0.0.1:80" listen="tcp:0.0.0.0:8080"

Then we use device definitions to map persistent data:

incus config device add Container-1 ghost-data disk source=/home/scott/ghost-data path=/var/lib/ghost/content

I think we need some type of consistency of interface. For example, we still need a way to pass the docker restart policy for a container.

We need a CLI for creating docker OCI apps as simple as a docker run command:

 docker run -d \
	--name some-ghost \
    --restart unless-stopped \
	-e NODE_ENV=development \
	-e database__connection__filename='/var/lib/ghost/content/data/ghost.db' \
	-p 80:2368 \
	-v /path/to/ghost/blog:/var/lib/ghost/content \
	ghost:alpine

Not a criticism, just my thoughts.

11

I don’t see anything inconsistent there: both are adding devices, and you could add the proxy device using incus config device add ... instead of putting it in a profile.

Indeed, things which go in a profile are only really useful if they are being shared between multiple containers - but obviously, multiple containers can’t bind to the same local port on the host for proxying, so I’d recommend not putting that setting in a profile.

As for incus launch .. -c environment.NODE_ENV=development, you could do

incus create docker:ghost:alpine Container-1
incus config set Container-1 environment.NODE_ENV=development

Now that’s a setting which perhaps could usefully sit in a profile, if you have multiple containers which all make use of an environment variable NODE_ENV - although different OCI images will generally consume different variables.

EDIT: I’m sticking on incus 6.0 LTS so I can’t test this properly, but regarding your docker command

docker run -d \
	--name some-ghost \
    --restart unless-stopped \
	-e NODE_ENV=development \
	-e database__connection__filename='/var/lib/ghost/content/data/ghost.db' \
	-p 80:2368 \
	-v /path/to/ghost/blog:/var/lib/ghost/content \
	ghost:alpine

the main problem is that AFAICS the -d (--device) flag to incus launch only lets you override the parameters of an existing device from a profile, not create a new device.

I guess what you’d want is something like this (which won’t work today):

incus launch ghost:alpine some-ghost \
  -c environment.NODE_ENV=development \
  -c environment.database__connection__filename=/var/lib/ghost/content/data/ghost.db \
  -d hostport2368,type=proxy,connect=tcp:0.0.0.0:80,listen=tcp:0.0.0.0:2368 \
  -d ghost-data,type=disk,source=/home/scott/ghost-data,path=/var/lib/ghost/content

(and I guess a restart policy would be another -c setting)

1 Like
12

Yeah, it should generally be possible by using -d for this stuff.
Restart policy currently doesn’t exist by once it does, it will work with -c.

13

Sorry to warm up an older thread but it seems like this trick doesn’t work with latest stable release 6.8?

I just created a new bookworm server and installed incus 6.8, added the remote entries and started to deploy some OCI images and get

Error: Error occurred when starting proxy device: Error: Permission denied - Failed setns to connector network namespace

Also tried @stgraber trick:

during “incus init”… but no joy.

Interesting is that the trick still works for “ghost” but not for “authentik” for example. I have an older system where I installed “authentik” on 6.3 and there it worked. Now with 6.8 it doesn’t. I’m able to start the instance without proxy, however adding the proxy afterwords (during runtime) doesn’t do the trick either.

Any ideas how to debug this further are more than welcome.

14

This means that there is something specific with that other OCI image that you mentioned.
That is, since the docker:ghost:alpine with the parameter environment.NODE_ENV=development allows to create successfully a proxy device, then something is different with the other OCI image that you are trying.

In that case, you would need to post some instructions on how you try to setup authentik so that someone else that does not know what is authentik, can build on and perhaps figure out what is needed there. The instructions are somewhat involved.

15

Right, didn’t send any instructions. They are pretty straight forward:

incus remote add ghcr https://ghcr.io --protocol=oci
incus init ghcr:goauthentik/server auth-server -c environment.NODE_ENV=development
incus config device add auth-server web proxy connect=tcp:127.0.0.1:9443 listen=tcp:0.0.0.0:9443

You need of course have a redis and postgresql server up and running and provide the environment variables as documented. But even without them you get the error message.

I’m not sure but it might be related to the recent OCI DHCP enhancement? My next try will be to install an older Incus release < 6.7 to check if this is true. As mentioned I have installed it with 6.3 and I tried older Authentik versions to.

Update follows…

16

Think I found the commit which causes the issue.

As mentioned I was able to deploy Authentik with version 6.3. As such I setup an Incus build VM, followed the instructions “how to compile from source” (btw. if you want to use OCI it requires skopeo and umoci to be installed which is missing in the documentation) downloaded 6.3 sources, compiled, run the minimum init followed by the steps above and the container started without complaining.

Created another VM and done the exact same steps for 6.4 and the container failed to start with exactly the same error message

Error: Error occurred when starting proxy device: Error: Permission denied - Failed setns to connector network namespace

This indicates the issue was introduced between 6.3 and 6.4. Going through the list of commits one major OCI change was commit 5d8b9c8 which changes how the container initializes. Reversing the commit, recompile and restart incus allowed the start the container including proxy.
After the container has started once you can re-apply the change, recompile and restart and it still all works.

In summary with the improvement of commit 5d8b9c8 adding a proxy to an OCI container fails with the error

Error: Error occurred when starting proxy device: Error: Permission denied - Failed setns to connector network namespace

Knowing which code change introduces the issue explains or better clarifies the error message itself.

As I don’t have such deep knowledge about the Incus internals I’m not quite sure how to change the code but I believe with the findings from above it will help @stgraber to sort this out quickly.

I’m now able to move forward to get my new server setup with a custom deployment build.

Happy to run more tests if required.

17

I fixed that one on Friday :slight_smile:

18
19

Thanks @stgraber, I’m impressed!

Just want to confirm that it works. Compiled github head, run a few tests and all the errors are gone.

Looking forward for the next release.