ELAPSED time of the process inside incus container is wrong

Incus
, ,
1

Hello. Please help me understand.

Using these images: images:debian/bookworm/amd64 and images:debian/bookworm/cloud.

I want to find out how much time since when the particular process is running.

Example command:
ps -eo pid,etimes,start_time,args --sort=start_time

root@testtime2:/proc/1# ps -eo pid,etimes,start_time,args --sort=start_time
    PID ELAPSED START COMMAND
      1     739 14:36 /sbin/init
    120     738 14:36 /lib/systemd/systemd-journald
    127     738 14:36 /lib/systemd/systemd-udevd
    131     738 14:36 /lib/systemd/systemd-resolved
    138     738 14:36 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
    140     738 14:36 /lib/systemd/systemd-logind
    144     738 14:36 /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 linux
    146     737 14:36 /lib/systemd/systemd-networkd
    147     733 14:37 bash
    170 4123168224 14:55 ps -eo pid,etimes,start_time,args --sort=start_time
root@testtime2:/proc/1#

user@host-1:~$ uptime
 14:58:47 up 28 min,  2 users,  load average: 0.00, 0.00, 0.02
root@testtime2:/proc/1# uptime
 14:58:24 up 21 min,  0 user,  load average: 0.00, 0.00, 0.02

As you see above, the host and containers booted just recently, but elapsed time is way too high, and it does not change. I have checked my other older setups (like debian 11 and LXD), on those containers I get constantly reported 0 even though the process has been running for some time.

ps command looks up information in /proc/ sub directories, and probably something is reported there incorrectly. Does incus container require special kernel privileges for this? If so, then what permissions need granting? I have alrady made container privileged, that did not help.

Thank you.

2

Most likely it’s a timezone issue. By default your containers and VMs get the timezone Etc/UTC, unless you configure otherwise.

Run in the container timedatectl to determine the current timezone.

Run timedatectl list-timezones to list the available timezones. Such as Europe/Berlin.

Finally, run

$ sudo timedatectl set-timezone Europe/Berlin 
$ timedatectl 
               Local time: Fri 2024-05-31 16:50:12 CEST
           Universal time: Fri 2024-05-31 14:50:12 UTC
                 RTC time: n/a
                Time zone: Europe/Berlin (CEST, +0200)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no
$

You may configure the appropriate timezone through the Incus default profile.

3

Thanks, I tried both with UTC and Europe/London (BST).

At the time of test, both container and host were in the same time zone.

user@host-1:~$ date
Fri 31 May 16:03:03 BST 2024
user@host-1:~$ timedatectl
               Local time: Fri 2024-05-31 16:03:05 BST
           Universal time: Fri 2024-05-31 15:03:05 UTC
                 RTC time: Fri 2024-05-31 15:03:05
                Time zone: Europe/London (BST, +0100)
System clock synchronized: no
              NTP service: n/a
          RTC in local TZ: no
user@host-1:~$ incus exec testtime2 bash
root@testtime2:~# date
Fri May 31 16:03:13 BST 2024
root@testtime2:~# timedatectl
               Local time: Fri 2024-05-31 16:03:16 BST
           Universal time: Fri 2024-05-31 15:03:16 UTC
                 RTC time: n/a
                Time zone: Europe/London (BST, +0100)
System clock synchronized: no
              NTP service: n/a
          RTC in local TZ: no
root@testtime2:~# ps -eo pid,etimes,start_time,args --sort=start_time
    PID ELAPSED START COMMAND
      1    4794 14:36 /sbin/init
    120    4793 14:36 /lib/systemd/systemd-journald
    127    4793 14:36 /lib/systemd/systemd-udevd
    131    4793 14:36 /lib/systemd/systemd-resolved
    138    4793 14:36 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
    140    4793 14:36 /lib/systemd/systemd-logind
    144    4793 14:36 /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 linux
    146    4792 14:36 /lib/systemd/systemd-networkd
    180 4123168224 16:03 bash
    183 4123168224 16:03 /lib/systemd/systemd-timedated
    184 4123168224 16:03 ps -eo pid,etimes,start_time,args --sort=start_time
root@testtime2:~#
4

is there anything that I need to enable with

incus config set testtime2 security.syscalls.allow
or
incus config set testtime2 raw.apparmor ?

5

You could try setting security.syscalls.intercept.sysinfo=true in case ps relies on the uptime value from that data to calculate the elapsed time somehow.

Otherwise, the /proc data comes straight from the kernel, Incus doesn’t really have any control over there.

LXCFS which Incus runs for you does mask a few files in /proc which maybe are also confusing ps somehow? You can easily find those with grep lxcfs /proc/mounts and could try to unmount them to see if that fixes things, though note that you’ll also start seeing the host resources rather than whatever limits you may have set on the container.

6

umounting /sys/uptime helped. You can see that my vim has been open for few seconds.

Will this be safe to do? It is just hat container will show the hosts uptime.

root@testtime2:~# ps -eo pid,etimes,start_time,args --sort=start_time
    PID ELAPSED START COMMAND
      1      87 19:28 /sbin/init
    114      87 19:28 /lib/systemd/systemd-journald
    119      87 19:28 /lib/systemd/systemd-resolved
    125      87 19:28 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
    126      87 19:28 /lib/systemd/systemd-logind
    128      87 19:28 /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 vt220
    130      86 19:28 /lib/systemd/systemd-networkd
    133      70 19:28 bash
    148      23 19:29 bash
    153       6 19:29 vim
    160       0 19:29 ps -eo pid,etimes,start_time,args --sort=start_time

FYI, security.syscalls.intercept.sysinfo=true did not help.

7

Yeah, it’s safe to unmount /proc/uptime though as you’ve no doubt noticed, it now means that your container will report the same uptime as the host it’s running on.

8

Thank you. I am trying to figure out how to not mount this.

This config did not help:

config:
  raw.lxc: lxc.mount.entry = dev/null proc/uptime none optional,noauto

neither this did, excluding proc

config:
  raw.lxc:
    lxc.mount.auto: sys cgroup

adding this in the container /etc/fstab did not help either

/dev/null /proc/uptime none default 0 0

should I write a custom script i.e. systemd unit to unmount /proc/uptime every time the container is started?

9

Yeah, it’s mounted by the LXCFS hook so you won’t have much luck preventing it from being mounted. Your best bet is indeed a unit inside of the instance that will unmount it on boot.

10

Thank you, I shall do that with systemd unit, fairly simple.

Is there anything working on the backgroud in incus process that may mount it back during container runtime?

This is needed for a production application that employs ps command to calculate how long are the processes running. If this fix is not reliable then we will have to work around and just subtract this random value that is reported by ps command, it is not ideal but promising to be stable workaround.

11

Nope, it won’t be mounted back. The fact that we don’t have a good kernel API to inject this kind of mounts after startup is actually a problem for us at times, but in this case guarantees that it won’t come back :slight_smile: