Ubuntu 16.04 container loses IPv4 address immediately after launch and cannot be stopped

chevrox · November 3, 2021, 9:53pm

I created several containers using stock Ubuntu 16.04 image on a two-host (Debian 11) LXD cluster that uses macvlan. The newly launched containers loses IPv4 right after executing the first lxc exec, and lxc subsequently hangs upon any attempt at manipulating the container, including lxc stop.

Using another host, also running Debian 11 that uses LXD bridge and is not part of any cluster, i created another Ubuntu 16.04 container using the stock image to see if the issue is present with default setup. In this setup, the container doesn’t get IPv4 address at all, and hangs upon any command after creation.

There are production containers running on the cluster so I’d rather not bring the cluster offline (these are computing resources without redundancy) and reinstall LXD, but the Ubuntu 16.04 containers won’t shut down so I can delete them or free up the resources they take (which are albeit minimal). Ideas?

Edit: here’s part of the log (just repeats ad nauseum)

lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:disarm_handler:61 - Disarmed 25 for "lxc_cmd_handler" handler
lxc 20211103220224.328 TRACE    commands - commands.c:lxc_cmd:516 - Opened new command socket connection fd 62 for command "get_state"
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_cleanup_handler:1945 - Closing client fd 25 for "lxc_cmd_cleanup_handler"
lxc 20211103220224.328 DEBUG    commands - commands.c:lxc_cmd_get_state:1053 - Container "ubuntu-16-04" is in "RUNNING" state
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:delete_handler:75 - Deleted 25 for "lxc_cmd_handler" handler
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_accept:2042 - Accepted new client as fd 25 on command server fd 9
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_handler:2005 - Processed command get_limit_cgroup_fd; cleaning up client fd 25
lxc 20211103220224.328 DEBUG    commands - commands.c:lxc_cmd_rsp_recv_fds:136 - Received exact number of file descriptors 1 == 1
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:disarm_handler:61 - Disarmed 25 for "lxc_cmd_handler" handler
lxc 20211103220224.328 TRACE    commands - commands.c:lxc_cmd:516 - Opened new command socket connection fd 62 for command "get_limit_cgroup_fd"
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_cleanup_handler:1945 - Closing client fd 25 for "lxc_cmd_cleanup_handler"
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:delete_handler:75 - Deleted 25 for "lxc_cmd_handler" handler
lxc 20211103220224.328 TRACE    cgfsng - cgroups/cgfsng.c:cgroup_get:3567 - Reading pids.current from unified cgroup hierarchy
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_accept:2042 - Accepted new client as fd 25 on command server fd 9
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_handler:2005 - Processed command get_state; cleaning up client fd 25
lxc 20211103220224.328 DEBUG    commands - commands.c:lxc_cmd_rsp_recv_fds:147 - Command "get_state" received response
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:disarm_handler:61 - Disarmed 25 for "lxc_cmd_handler" handler
lxc 20211103220224.328 TRACE    commands - commands.c:lxc_cmd:516 - Opened new command socket connection fd 62 for command "get_state"
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_cleanup_handler:1945 - Closing client fd 25 for "lxc_cmd_cleanup_handler"
lxc 20211103220224.328 DEBUG    commands - commands.c:lxc_cmd_get_state:1053 - Container "ubuntu-16-04" is in "RUNNING" state
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:delete_handler:75 - Deleted 25 for "lxc_cmd_handler" handler
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_accept:2042 - Accepted new client as fd 25 on command server fd 9
lxc ubuntu-16-04 20211103220224.328 TRACE    commands - commands.c:lxc_cmd_handler:2005 - Processed command get_limit_cgroup_fd; cleaning up client fd 25
lxc 20211103220224.328 DEBUG    commands - commands.c:lxc_cmd_rsp_recv_fds:136 - Received exact number of file descriptors 1 == 1
lxc ubuntu-16-04 20211103220224.328 TRACE    mainloop - mainloop.c:disarm_handler:61 - Disarmed 25 for "lxc_cmd_handler" handler

stgraber · November 3, 2021, 10:03pm

And by lxc stop, I’m assuming that includes a lxc stop --force?

chevrox · November 3, 2021, 11:45pm

@stgraber yes, and the container does not respond to any command.

stgraber · November 3, 2021, 11:53pm

Can you show ps fauxww on the system in question, at least the chunk related to that container (starting from [lxc monitor] ... NAME and including all processes for the container).

chevrox · November 8, 2021, 8:41pm

@stgraber I was eventually able to force stop the containers cross-host for the cluster, so please consider this issue resolved. Note to self: do not create Ubuntu 16.04 containers.