Docker swarm in Incus OCI?

Lesley_Richards · September 19, 2024, 11:11am

Hi all
I’ve been testing incus, running docker containers without installing docker. It works perfectly and I’m really excited.
However, I have an existing cluster of pc’s running docker swarm. So my obvious question is: Can docker swarm run in incus using OCI?
Thank you

simos · September 19, 2024, 11:46am

When we use the word Docker, we mean the container management tool that is used to run container images as those found on Docker Hub (https://hub.docker.com/). Those container images are OCI container images.

Incus is able to use its own container management service to run those OCI container images, whether they come from Docker Hub or some other repository of such OCI container images. This is quite crucial; Incus is using its own service to run those OCI container images (not Docker’s).

Docker Swarm is this. Docker Swarm requires a way to run Docker images. It is likely either easy or somewhat easy to get swarmkit to work with OCI images running in Incus. swarmkit should be somehow modify to get Incus to launch the images instead of invoking the docker command. I think anyone can give it a go and report back what is working and what’s missing.

abiosoft · September 19, 2024, 11:54am

I think the easier way out would be to run Docker Swarm inside Incus containers, rather than getting Incus to act like Docker Swarm.

simos · September 19, 2024, 12:32pm

I have not tried any of these.
Can it work in an Incus container with security.nesting=true? Does it require security.nesting=true to run Docker in an Incus container? Or, just use an Incus VM instead?

abiosoft · September 19, 2024, 12:42pm

Yeah, it should work fine with security.nesting=true.

Arnaud_F · March 18, 2025, 12:59pm

Hi,

Result of my testing on this topic:

I’ve created a debian 12 container with :

config:
  security.nesting: "true"
  security.syscalls.intercept.mknod: "true"
  security.syscalls.intercept.setxattr: "true"

Installed docker using apt
Run docker swarm init ...
Launched my first stack

Result : it fails when dealing with overlay networks.
Below the syslog content :

Mar 18 11:20:31 app-docker dockerd[515]: time="2025-03-18T11:20:31.312575608Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint hfvzohopa9k3483rsgkm0acrd 8>
Mar 18 11:20:31 app-docker dockerd[515]: time="2025-03-18T11:20:31.588131577Z" level=error msg="fatal task error" error="error creating external connectivity network: cannot restrict inter-container communicat>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322398608Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322540401Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322601439Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322678987Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322725622Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322885901Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322955295Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.322737877Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323060825Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323129725Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323164353Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323242477Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323242912Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323376575Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323444855Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323504017Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323566145Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.323642786Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.372617513Z" level=warning msg="failed to deactivate service binding for container cda-diee-inte_maildev.1.xxrzlpyk4y52gvkuqxw7frqpc" error="No>
Mar 18 11:20:32 app-docker dockerd[515]: time="2025-03-18T11:20:32.881824143Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint ufr9hbfjwqfcmijsqp7yqp3q0 0>
Mar 18 11:20:33 app-docker dockerd[515]: time="2025-03-18T11:20:33Z" level=info msg="level=info ts=2025-03-18T11:20:33.625793884Z caller=driver.go:77 msg=\"starting logging driver for container\" id=eadad43dff>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.494885605Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.495072053Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.495157884Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.495417469Z" level=error msg="error reading the kernel parameter net.ipv4.vs.conn_reuse_mode" error="open /proc/sys/net/ipv4/vs/conn_reuse_mode>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.495512010Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodes>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34.495591971Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_quiescent_template" error="open /proc/sys/net/ipv4/vs/expir>
Mar 18 11:20:34 app-docker dockerd[515]: time="2025-03-18T11:20:34Z" level=info msg="level=info ts=2025-03-18T11:20:34.637124647Z caller=driver.go:77 msg=\"starting logging driver for container\" id=75ade0c83f>

Networks are created in swarm using the ‘overlay’ driver.

Any idea how to solve this ?

Arnaud_F · March 18, 2025, 1:17pm

Answering myself, any issue was posted to github and a solution proposed which seems to work on my side.

The issue : Cannot disable userland-proxy in LXC container even though it works without it · Issue #49240 · moby/moby · GitHub

The solution :

create /etc/systemd/system/docker.service.d/ignore-br-netfilter-error.conf
with content :

[Service]
Environment="DOCKER_IGNORE_BR_NETFILTER_ERROR=1"

systemctl daemon-reload
systemctl restart docker

After that, the stack is starting correctly and the issue doesn’t appear anymore in the logs

Arnaud_F · March 18, 2025, 4:37pm

@simos : after playing around Docker behaves poorly by setting the environment variable because it exposes all services to ipv6 endpoints and no more ipv4.

Impossible to change this.

I wanted to rollback the env settings and putting the sysctl it argues like :

  raw.lxc: |2

    lxc.sysctl.net.ipv4.neigh.default.gc_thresh1=128
    lxc.sysctl.net.ipv4.neigh.default.gc_thresh2=512
    lxc.sysctl.net.ipv4.neigh.default.gc_thresh3=1024
    lxc.sysctl.net.ipv4.vs.conn_reuse_mode=0
    lxc.sysctl.net.ipv4.vs.expire_nodest_conn=1
    lxc.sysctl.net.ipv4.vs.expire_quiescent_template=1
    lxc.sysctl.net.ipv6.bindv6only=0

however I can’t run my container anymore :

Log:

lxc app-docker 20250318163244.376 ERROR    conf - ../src/lxc/conf.c:setup_sysctl_parameters:3126 - No such file or directory - Failed to setup sysctl parameters net.ipv4.neigh.default.gc_thresh1 to 128
lxc app-docker 20250318163244.376 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4011 - Failed to setup sysctl parameters
lxc app-docker 20250318163244.376 ERROR    start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "app-docker"
lxc app-docker 20250318163244.376 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc app-docker 20250318163244.388 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "vmbr0" to its initial name "veth9e44bdf8"
lxc app-docker 20250318163244.399 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "vmbr3" to its initial name "veth9065cf73"
lxc app-docker 20250318163244.399 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc app-docker 20250318163244.399 ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "app-docker"
lxc app-docker 20250318163244.399 WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 873879
lxc 20250318163244.681 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20250318163244.681 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

any idea how to set the asked sysctl by Docker ?