LXC 3.0.3 processes killed by nginx restart


#1

Hello all,

I have recently been upgrading a set of config managed servers from Ubuntu 14 -> 16 -> 18

This required upgrades to nginx/passenger and lxc (from 2 to 3.0.3)

Having updated my lxc containers’ configs, most things are working nicely but there is one unexpected problem in that the containers are destroyed on all but 1 server when nginx is restarted.

The difference between the 1 “working” and other “failing” servers appears to be an entry in the journalctl:

nginx.service: Killing process 13743 (lxc-start)

As far as I can tell, all the relevant configs (nginx,passenger,lxc) match and I have ran out of ideas of where to look next.

Has anyone else run into (and hopefully solved) into this issue? I am sure this must be a config issue somewhere but I am not sure where else to look so any advice will be greatly appreciated.

Relevant versions:
Nginx: 1.14
Phusion Passenger: 6.0.2
LXC: 3.0.3
OS: Ubuntu 18.04.2 Bionic Bear


(Stéphane Graber) #2

What’s starting your LXC container?


#3

Hey, thanks for taking an interest.

It’s a Node app that just executes lxc commands as a child-process. The Node app is maintained using nginx+passenger


(Stéphane Graber) #4

Ok, so I think your probably is systemd.

Back when you were on 14.04, you were on upstart which would manage the main process (nginx) but not care about subprocesses so much.

Systemd is different as it uses a cgroup per service and then restarts then entirety of that cgroup on restarts, which in your case, includes your LXC containers.

You should be able to fix that by doing a systemd override for that unit (systemctl edit nginx) and modifying the KillMode to process (you may want to look at systemd man for details).


#5

I think you are definitely onto something there

I don’t see any difference between the working and failing systemctl files for nginx but setting KillMode to process resolves the issue.

If I run systemctl status nginx I still see some references to lxc that aren’t present in the other working server.

e.g:

   CGroup: /system.slice/nginx.service
           ├─ 8234 [lxc monitor] /var/lib/lxc 302323
           ├─ 8814 [lxc monitor] /var/lib/lxc 302085
           ├─ 9364 [lxc monitor] /var/lib/lxc 302089
           ├─ 9531 [lxc monitor] /var/lib/lxc 302321
           ├─ 9871 [lxc monitor] /var/lib/lxc 302115

I’ve no idea how I have managed to set up different cgroups but at least for now I have a workaround and plenty of reading. Thanks for the help Stéphane :+1: