Forkproxy child process hangs after adding proxy device to a container

LXD
1

Hi,

We have an EC2 Ubuntu instance which runs several containers.
When I add a proxy device to a container with:

lxc config device add <container_name> myport2201 proxy listen=tcp:<ec2_internal_ip>:2201 connect=tcp:127.0.0.1:22

It executes and adds a device just fine. But, it also leaves one process hanging:

$ ps -ef | grep defunct
root     1527033 1527030  0 15:20 ?        00:00:00 [lxd] <defunct>

Which parent appears to be forkproxy process:

$ ps -ef | grep 1527030
1000000  1527030 2418824  0 15:20 ?        00:00:00 /snap/lxd/current/bin/lxd forkproxy -- 2418824 4 tcp:<ec2_internal_ip>:2201 1272039 3 tcp:127.0.0.1:22   0644
root     1527033 1527030  0 15:20 ?        00:00:00 [lxd] <defunct>

Please advise how to deal with this. Is this a normal thing for forkproxy process to be alive or should it exit upon creation of a device? LXD version is 4.0.4

Thanks!

2

forkproxy will stick around so long as the container is running.
I’m not sure what the defunct child process is about and I’m not seeing it here on a more recent LXD version, so there’s a reasonable chance that the upcoming 4.0.5 may fix that for you.

3

Thank you for the reply. It is a strange behaviour, I’ve just upgraded LXD to the latest stable version 4.10 - same issue


Defunct process is still there upon stopping and starting the container.

4

And the defunct process is definitely tied to forkproxy? So adding more proxy devices leads to more of those showing up?

5

Exactly. Adding ‘x’ amount of proxy devices to a container leads to ‘x’ defunct processes.

I am 100% parent is forkproxy process. As pasted in my first post, forkproxy process with ID 1527030

The logs look ok, I don’t see any error in forkstart.log for that container nor in lxd.log.

6

@brauner can you take a look at this one?

1 Like
7

forkproxy seems to leave a zombie behind.

8

Thanks for confirming. But, is that something we need to worry about? I.e. if they pile up over time can they cause issues with the host or containers? Could the host run out of memory?

We have to run 30+ containers on the host. Each container has to have 3 proxy devices. That will leave 90 forkproxy processes up and running - each having a zombie. We are worried it could drain the memory out or similar.

Could we just kill the forkproxy process with zombie child itself? We’ve noticed there aren’t any consequences when we do that, container runs smoothly after that and device is still attached. Or do you have any kind of workaround for this?

9

This should be fixed by

2 Likes
10

Thanks for confirming. But, is that something we need to worry about? I.e. if they pile up over time

It’s nothing to really worry about but we will obviously fix this! :slight_smile:

can they cause issues with the host or containers? Could the host run out of memory?

We have to run 30+ containers on the host. Each container has to have 3 proxy devices. That will leave 90 forkproxy processes up and running - each having a zombie. We are worried it could drain the memory out or similar.

Apart from polluting the process table zombies don’t have any negative side effects. Essentially, a zombie is a process that has already exited but it’s exit status hasn’t been retrieved by the parent. So the kernel already has released all resources associated with that process but just keeps the task struct around (which is a bit of memory but not all that much). It’s almost impossible to drain your host of resources because of that. One problem could be that if you were to create an insane amount of zombies you could run out of PID numbers since they aren’t recycled. But that’s unlikely too.

Could we just kill the forkproxy process with zombie child itself? We’ve noticed there aren’t any consequences when we do that, container runs smoothly after that and device is still attached. Or do you have any kind of workaround for this?

You sure could do that but killing the forkproxy process would mean no data will be shoveled which I’d assume is what you would want to use the forkproxy for.

But in any case, the branch I sent should fix the issue you reported and get rid of any zombies.

1 Like
11

Thank you for resolving this so quickly and for thorough explanation! My team is new to working with LXD so very much appreciated :slight_smile:

1 Like
12

No worries and you’re very welcome! I hope you enjoy it. :slight_smile: