Apache inside lxd stops serving pages after a while

LXD
#1

I have 3 containers running Ubuntu 22.04. The host is also Ubuntu 22.04.

Those 3 containers are running Apache, which runs well in the beginning, but then after a while (could be a few hours or a few days), it just stops serving pages. Each time another container fails.

The service is still running, port 443 is still being listened to by Apache, and the logs don’t show anything relevant.

The only way to get out of this situation is to restart Apache.

It could very much not be related to LXD, but it seems odd that it occurs with all the containers.

Any suggestions or guiding questions would be helpful.

#2

I tried to debug Apache using strace (started before the hang, stopped a few seconds after), but couldn’t make sense of anything meaningful in the log.

What else do you think I should try?

#3

Is it worth trying Ubuntu 20.04 for the containers?

#4

Maybe, if anything that may give you another datapoint to sort out what’s going on.

#5

I downgraded 2 of the 3 containers to Ubuntu 20.04, left one as 22.04 for comparison.

So far the 22.04 Apache has stopped once, while the others were running continuously. It’s still too soon to know for sure, let’s give it a few more days, but it’s beginning to appear that the problem is with the 22.04 container image.

Maybe it’s worth for whoever is in charge to launch a 22.04 container, install Apache and run a small site, then check it out a few days later to see if it’s still responding to requests.

#6

Yeah, I wonder what’s going on there. Would be interesting to run with maximum debug in apache2 to see what it’s doing.

#7

Let me know if you want me to help.
If so, please tell me exactly what configuration to set.

#8

I’ve not really played with apache2 in a while but Google says you can set LogLevel to Debug which should cause a lot more output to go to /var/log/apache2/

#9

Okay, I’ve set the log level to debug, and will post the log here next time it happens.

#10

I have some great findings!

Thanks to the debug log, I was able to find a discussion linking this exact problem to the mpm_event module in Apache 2.4.52 and 2.4.53 (the former is the version offered by apt on Ubuntu 22.04).

Those patches should solve it:
https://svn.apache.org/viewvc?view=revision&revision=1899858
https://svn.apache.org/viewvc?view=revision&revision=1899865
https://svn.apache.org/viewvc?view=revision&revision=1899884

I’m not sure if it is officially fixed in 2.4.54.

So the problem was always with Apache and not the container image.
Thanks for your patience guidance.

1 Like