Nested incus breaking out of configured limits

Incus
1

I’m running incus nested inside another incus container. The outside container has limits set on memory and cpu usage. I noticed the inner container breaks out of those limits and has full access to the host’s resources. Is this expected behaviour?

Outside container:

pargo@bastion:~$ incus config show amd01-r --expanded
architecture: x86_64
config:
  cloud-init.vendor-data: "#cloud-config\npackage_update: true\npackage_upgrade: true\npackage_reboot_if_required:
    true\npackages: \n- htop\ntimezone: America/Fortaleza\nusers:\n- gecos: Default
    pargo user\n  groups: sudo, video, render\n  name: pargo\n  lock_passwd: true\n
    \ sudo: ALL=(ALL) NOPASSWD:ALL\n  shell: /bin/bash\n"
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20241217_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20241217_07:42"
  image.type: squashfs
  image.variant: cloud
  limits.cpu: 6-7,14-15
  limits.memory: 8GB
  security.nesting: "true"
  security.syscalls.intercept.mount: "true"
  volatile.base_image: 7ea3a7e80da1af4fd9724a4dc3ee6cb82351940bf8189b4b8797fac271d7c151
  volatile.cloud-init.instance-id: 5ab26b9b-4a9c-406f-97ac-d5e8e231b20e
  volatile.eth0.host_name: vethf69f8141
  volatile.eth0.hwaddr: 00:16:3e:04:74:5c
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: abc7dd06-1184-4b20-bb0d-853532160fd4
  volatile.uuid.generation: abc7dd06-1184-4b20-bb0d-853532160fd4
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: local
    type: disk
ephemeral: false
profiles:
- default
- amd-5700g
stateful: false
description: Restricted instance on amd01
pargo@bastion:~$ incus exec amd01-r -- free -h
               total        used        free      shared  buff/cache   available
Mem:           7.5Gi       219Mi       5.7Gi       464Ki       1.6Gi       7.2Gi
Swap:             0B          0B          0B

On the inner container the memory usage sees the entire host’s memory. Not only that, this container was just created. The amount of memory used indicates it’s seeing the host’s used memory.

root@amd01-r:~# incus config show d1 --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20241217_05:24)
  image.os: Debian
  image.release: bookworm
  image.serial: "20241217_05:24"
  image.type: squashfs
  image.variant: default
  security.privileged: "true"
  volatile.base_image: 583ae31313a3eb7e87b389b163fc503f1511bc78a398ff8e027a43d194e07a64
  volatile.cloud-init.instance-id: 9213a5e3-9f59-4603-a35d-a6f635b17777
  volatile.eth0.host_name: vethc79449d7
  volatile.eth0.hwaddr: 00:16:3e:8c:4e:a2
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 93bb198a-05aa-43df-8d36-31c90d80bba0
  volatile.uuid.generation: 93bb198a-05aa-43df-8d36-31c90d80bba0
devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
root@amd01-r:~# incus exec d1 -- free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi       4.0Gi       280Mi       2.0Gi        28Gi        26Gi
Swap:          7.7Gi        83Mi       7.6Gi

Previously, I was seeing it breaking out of cpu limits. It seems to be ok for now. Must have been some of the tests I was running, but I’ll try to reproduce breaking out of cpu limits again.

2

It’s unlikely that it broke out of the resource limit as that would be a rather serious kernel bug.
Most likely what’s going on is that the nested container isn’t getting access to LXCFS so it sees the host resources rather than its limits.

1 Like
3

I’ll test it further and report.

How can I check if it has access to lxcfs?

4

Sorry for the false alarm. Indeed the limits are enforced, just not a good way to check them with tools like htop. Is there a way to fix this?

5

Is there anything to fix? The fact that the container is somewhat “clueless” about it’s containerization limits strikes me as being a good thing in many ways.

6

Why a good thing?

To start, it’s odd it can see the host’s resources. And if any application checks memory usage to see if it can use more RAM, it’ll not check correctly. Memory might be full and it can just keep trying to get more since the host has a lot available until the application crashes without memory. I can’t even see a way in which this would be a good thing.

7

Another issue that I just tested right now. An application that scales to the number of available cpu cores. It checks the number of cpu cores and launches a number of threads equivalent to the number of cores. On the first level, it reads correctly the number of cores and launches 4 cores. On the nested instance, it launches 16 threads and starves the container of resources.

8

Did you try restarting the Incus container (parent one) after installing Incus inside of it?

We have some limited logic for passing through LXCFS into nested containers but that requires that the host pass a clean LXCFS tree onto the first level container. This is only possible after it’s rebooted following the installation of Incus inside the container.

9

I did just now. Doesn’t help. Some interesting info I was looking at right now from the parent container’s perspective.

It sees 4 cores, like it’s configured to through /proc/cpuinfo

root@amd01-r:~# cat /proc/cpuinfo                                                                                                                                                                                                            
processor       : 0                                                                                                                                                                                                                          
vendor_id       : AuthenticAMD                                                                                                                                                                                                               
cpu family      : 25                                                                                                                                                                                                                         
model           : 80                                                                                                                                                                                                                         
model name      : AMD Ryzen 7 5700G with Radeon Graphics                                                                                                                                                                                     
stepping        : 0                                                                                                                                                                                                                          
microcode       : 0xa50000d                                                                                                                                                                                                                  
cpu MHz         : 1397.187                                                                                                                                                                                                                   
cache size      : 512 KB                                                                                                                                                                                                                     
physical id     : 0                                                                                                                                                                                                                          
siblings        : 16                                                                                                                                                                                                                         
core id         : 6                                                                                                                                                                                                                          
cpu cores       : 8                                                                                                                                                                                                                          
apicid          : 12                                                                                                                                                                                                                         
initial apicid  : 12                                                                                                                                                                                                                         
fpu             : yes                                                                                                                                                                                                                        
fpu_exception   : yes                                                                                                                                                                                                                        
cpuid level     : 16                                                                                                                                                                                                                         
wp              : yes                                                                                                                                                                                                                        
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl 
pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext 
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
 cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpi
d overflow_recov succor smca fsrm                                                                                                                                                                                                            
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso                                                                                                                                                               
bogomips        : 7586.00                                                                                                                                                                                                                    
TLB size        : 2560 4K pages                                                                                                                                                                                                              
clflush size    : 64                                                                                                                                                                                                                         
cache_alignment : 64                                                                                                                                                                                                                         
address sizes   : 48 bits physical, 48 bits virtual                                                                                                                                                                                          
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]                                                                                                                                                                               
                                                                                                                                                                                                                                             
processor       : 1                                                                                                                                                                                                                          
vendor_id       : AuthenticAMD                                                                                                                                                                                                               
cpu family      : 25                                                                                                                                                                                                                         
model           : 80                                                                                                                                                                                                                         
model name      : AMD Ryzen 7 5700G with Radeon Graphics                                                                                                                                                                                     
stepping        : 0                                                                                                                                                                                                                          
microcode       : 0xa50000d                                                                                                                                                                                                                  
cpu MHz         : 1397.344                                                                                                                                                                                                                   
cache size      : 512 KB
physical id     : 0
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 14
initial apicid  : 14
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl 
pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext 
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
 cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpi
d overflow_recov succor smca fsrm
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
bogomips        : 7586.00
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

processor       : 2
vendor_id       : AuthenticAMD
cpu family      : 25
model           : 80
model name      : AMD Ryzen 7 5700G with Radeon Graphics
stepping        : 0
microcode       : 0xa50000d
cpu MHz         : 1397.281
cache size      : 512 KB
physical id     : 0
siblings        : 16
core id         : 6
cpu cores       : 8
apicid          : 13
initial apicid  : 13
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl 
pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext 
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
 cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpi
d overflow_recov succor smca fsrm
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
bogomips        : 7586.00
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

processor       : 3
vendor_id       : AuthenticAMD
cpu family      : 25
model           : 80
model name      : AMD Ryzen 7 5700G with Radeon Graphics
stepping        : 0
microcode       : 0xa50000d
cpu MHz         : 1396.862
cache size      : 512 KB
physical id     : 0
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl 
pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext 
perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total
 cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpi
d overflow_recov succor smca fsrm
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass srso
bogomips        : 7586.00
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

It sees 8GB of memory through /proc/meminfo

root@amd01-r:~# cat /proc/meminfo                                                                                                                                                                                                    [1/1972]
MemTotal:        7812500 kB                                                                                                                                                                                                                  
MemFree:         7159380 kB
MemAvailable:    7679300 kB
Buffers:               0 kB
Cached:           520204 kB
SwapCached:            0 kB
Active:            26524 kB
Inactive:         582632 kB
Active(anon):        268 kB
Inactive(anon):    88968 kB
Active(file):      26256 kB
Inactive(file):   493664 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         88952 kB
Mapped:                0 kB
Shmem:               284 kB
KReclaimable:     380252 kB
Slab:                  0 kB
SReclaimable:          0 kB
SUnreclaim:            0 kB
KernelStack:       10288 kB
PageTables:        18536 kB
SecPageTables:      4172 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    24156348 kB
Committed_AS:    6351388 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      113052 kB
VmallocChunk:          0 kB
Percpu:            45440 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      398152 kB
DirectMap2M:    19914752 kB
DirectMap1G:    12582912 kB

But incus info --resources sees differently. It sees all the cores and some weird values for memory.

root@amd01-r:~# incus info --resources                                                                                                                                                                                                       
System:                                                                                                                                                                                                                                      
  Vendor: Gigabyte Technology Co., Ltd.                                                                                                                                                                                                      
  Product: A520M K V2                                                                                                                                                                                                                        
  Family: A520 MB                                                                                                                                                                                                                            
  Version: -CF                                                                                                                                                                                                                               
  SKU: Default string                                                                                                                                                                                                                        
  Type: container                                                                                                                                                                                                                            
  Chassis:                                                                                                                                                                                                                                   
      Vendor: Default string                                                                                                                                                                                                                 
      Type: Desktop                                                                                                                                                                                                                          
      Version: Default string                                                                                                                                                                                                                
  Motherboard:                                                                                                                                                                                                                               
      Vendor: Gigabyte Technology Co., Ltd.                                                                                                                                                                                                  
      Product: A520M K V2                                                                                                                                                                                                                    
      Version: x.x                                                                                                                                                                                                                           
  Firmware:                                                                                                                                                                                                                                  
      Vendor: American Megatrends International, LLC.                                                                                                                                                                                        
      Version: F1                                                                                                                                                                                                                            
      Date: 12/29/2022                                                                                                                                                                                                                       
                                                                                                                                                                                                                                             
Load:                                                                                                                                                                                                                                        
  Processes: 23                                                                                                                                                                                                                              
  Average: 0.12 0.18 0.17                                                                                                                                                                                                                    
                                                                                                                                                                                                                                             
CPU:           
  Architecture: x86_64
  Vendor: AuthenticAMD
  Name: AMD Ryzen 7 5700G with Radeon Graphics
  Caches:
    - Level 1 (type: Data): 32KiB
    - Level 1 (type: Instruction): 32KiB
    - Level 2 (type: Unified): 512KiB
    - Level 3 (type: Unified): 16MiB
  Cores: 
    - Core 0           
      Frequency: 1400Mhz       
      Threads:                  
        - 0 (id: 0, online: true, NUMA node: 0)
        - 1 (id: 8, online: true, NUMA node: 0)
    - Core 1
      Frequency: 1400Mhz
      Threads:
        - 0 (id: 1, online: true, NUMA node: 0)
        - 1 (id: 9, online: true, NUMA node: 0)
    - Core 2
      Frequency: 3711Mhz
      Threads:
        - 0 (id: 2, online: true, NUMA node: 0)
        - 1 (id: 10, online: true, NUMA node: 0)
    - Core 3
      Frequency: 3712Mhz
      Threads:
        - 0 (id: 3, online: true, NUMA node: 0)
        - 1 (id: 11, online: true, NUMA node: 0)
    - Core 4
      Frequency: 3309Mhz
      Threads:
        - 0 (id: 4, online: true, NUMA node: 0)
        - 1 (id: 12, online: true, NUMA node: 0)
    - Core 5
      Frequency: 3712Mhz
      Threads:
        - 0 (id: 5, online: true, NUMA node: 0)
        - 1 (id: 13, online: true, NUMA node: 0)
    - Core 6
      Frequency: 3711Mhz
      Threads:
        - 0 (id: 6, online: true, NUMA node: 0)
        - 1 (id: 14, online: true, NUMA node: 0)
    - Core 7
      Frequency: 4640Mhz
      Threads:
        - 0 (id: 7, online: true, NUMA node: 0)
        - 1 (id: 15, online: true, NUMA node: 0)
  Frequency: 3199Mhz (min: 1400Mhz, max: 4672Mhz)

Memory:                                                                                                                                                                                                                            [278/1998]
  Free: 7.31GiB
  Used: 24.19GiB
  Total: 31.50GiB

GPU:
  NUMA node: 0
  PCI address: 0000:05:00.0
  Driver: amdgpu (6.1.0-26-amd64)
  DRM:
    ID: 0
    Card: card0 (226:0)
    Control: controlD64 (226:0)
    Render: renderD128 (226:128)

NIC:
  NUMA node: 0
  PCI address: 0000:03:00.0
  Driver: r8169 (6.1.0-26-amd64)

Disks:
  Disk 0:
    NUMA node: 0
    ID: nvme0n1
    Device: 259:0
    Model: ADATA SWORDFISH
    Type: nvme
    Size: 465.76GiB
    WWN: eui.324d34303239414b4ce0001835484452
    Read-Only: false
    Removable: false
    Partitions:
      - Partition 1
        ID: nvme0n1p1
        Device: 259:1
        Read-Only: false
        Size: 953.00MiB
      - Partition 2
        ID: nvme0n1p2
        Device: 259:2
        Read-Only: false
        Size: 464.83GiB
  Disk 1:
    NUMA node: 0
    ID: sda
    Device: 8:0
    Model: ST4000VN006-3CW1
    Type: scsi
    Size: 3.64TiB
    Read-Only: false
    Removable: false
  Disk 2:                                                                                                                                                                                                                          [227/1998]
    NUMA node: 0
    ID: sdb
    Device: 8:16
    Model: ST4000VN006-3CW1
    Type: scsi
    Size: 3.64TiB
    Read-Only: false
    Removable: false

PCI devices:
  Device 0:
    Address: 0000:00:00.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1630
    NUMA node: 0
    IOMMU group: 0
    Driver: 
  Device 1:
    Address: 0000:00:00.2
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1631
    NUMA node: 0
    IOMMU group: 0
    Driver: 
  Device 2:
    Address: 0000:00:01.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1632
    NUMA node: 0
    IOMMU group: 0
    Driver: 
  Device 3:
    Address: 0000:00:02.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1632
    NUMA node: 0
    IOMMU group: 1
    Driver: 
  Device 4:                                                                                                                                                                                                                        [180/1998]
    Address: 0000:00:02.1
    Vendor:              
    Vendor ID: 1022
    Product:       
    Product ID: 1634
    NUMA node: 0    
    IOMMU group: 2
    Driver: pcieport
  Device 5:
    Address: 0000:00:02.2
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1634
    NUMA node: 0
    IOMMU group: 3
    Driver: pcieport
  Device 6:
    Address: 0000:00:08.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1632
    NUMA node: 0
    IOMMU group: 4
    Driver: 
  Device 7:
    Address: 0000:00:08.1
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1635
    NUMA node: 0
    IOMMU group: 5
    Driver: pcieport
  Device 8:
    Address: 0000:00:14.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 790b
    NUMA node: 0
    IOMMU group: 6
    Driver: 
  Device 9:
    Address: 0000:00:14.3
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 790e
    NUMA node: 0
    IOMMU group: 6
    Driver: 
  Device 10:                                                                                                                                                                                                                       [126/1998]
    Address: 0000:00:18.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166a
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 11:
    Address: 0000:00:18.1
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166b
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 12:
    Address: 0000:00:18.2
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166c
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 13:
    Address: 0000:00:18.3
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166d
    NUMA node: 0
    IOMMU group: 7
    Driver: k10temp
  Device 14:
    Address: 0000:00:18.4
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166e
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 15:
    Address: 0000:00:18.5
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 166f
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 16:                                                                                                                                                                                                                        [72/1998]
    Address: 0000:00:18.6
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1670
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 17:
    Address: 0000:00:18.7
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1671
    NUMA node: 0
    IOMMU group: 7
    Driver: 
  Device 18:
    Address: 0000:01:00.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 43ec
    NUMA node: 0
    IOMMU group: 8
    Driver: xhci_hcd
  Device 19:
    Address: 0000:01:00.1
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 43eb
    NUMA node: 0
    IOMMU group: 8
    Driver: ahci
  Device 20:
    Address: 0000:01:00.2
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 43e9
    NUMA node: 0
    IOMMU group: 8
    Driver: pcieport
  Device 21:
    Address: 0000:02:03.0
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 43ea
    NUMA node: 0
    IOMMU group: 8
    Driver: pcieport
  Device 22:                                                                                                                                                                                                                        [18/1998]
    Address: 0000:03:00.0
    Vendor: 
    Vendor ID: 10ec
    Product: 
    Product ID: 8168
    NUMA node: 0
    IOMMU group: 8
    Driver: r8169
  Device 23:
    Address: 0000:04:00.0
    Vendor: 
    Vendor ID: 1cc1
    Product: 
    Product ID: 5766
    NUMA node: 0
    IOMMU group: 9
    Driver: nvme
  Device 24:
    Address: 0000:05:00.0
    Vendor: 
    Vendor ID: 1002
    Product: 
    Product ID: 1638
    NUMA node: 0
    IOMMU group: 10
    Driver: amdgpu
  Device 25:
    Address: 0000:05:00.1
    Vendor: 
    Vendor ID: 1002
    Product: 
    Product ID: 1637
    NUMA node: 0
    IOMMU group: 11
    Driver: snd_hda_intel
  Device 26:
    Address: 0000:05:00.2
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 15df
    NUMA node: 0
    IOMMU group: 12
    Driver: ccp
  Device 27:
    Address: 0000:05:00.3
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1639
    NUMA node: 0
    IOMMU group: 13
    Driver: xhci_hcd
  Device 28:
    Address: 0000:05:00.4
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 1639
    NUMA node: 0
    IOMMU group: 14
    Driver: xhci_hcd
  Device 29:
    Address: 0000:05:00.6
    Vendor: 
    Vendor ID: 1022
    Product: 
    Product ID: 15e3
    NUMA node: 0
    IOMMU group: 15
    Driver: snd_hda_intel

When checking on the child container, /proc/cpuinfo and /proc/meminfo show all the system’s resources.

EDIT: The parent container is now running debian/12/cloud. Not sure it matters. It was ubuntu/24.04/cloud before when I was testing.

10

I suppose it depends on who owns the host/container and what you are trying to confine. I have multiple machines with quite different resources and it’s sometimes useful for me to see the host resources from inside the container. For my externally facing instances, I strictly use VM’s and tightly control what they have, so I hadn’t given this a lot of thought from a multi-user perspective.

On memory, if it’s useful, an “incus config set (instance) limits.memory=2GiB” gives the instance visibility only on the configured memory, not the full system (tested on Deb12 host/container):

andrew@Yoda:~$ incus shell cx
root@cx:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           501Gi        35Mi       501Gi       112Ki        70Mi       501Gi
Swap:             0B          0B          0B
root@cx:~# exit
logout
andrew@Yoda:~$ incus config set cx limits.memory=2GiB
andrew@Yoda:~$ incus shell cx
root@cx:~# free -h
               total        used        free      shared  buff/cache   available
Mem:           2.0Gi        35Mi       1.9Gi       112Ki        70Mi       2.0Gi
Swap:             0B          0B          0B
root@cx:~#

But your point on cpu’s is certainly a good one, especially if you restrict the container to having few cores - that still shows up in full, like disk devices.

I can also now see that if you want to expose such containers to multi-users, it is providing “to much information” for no benefit, so I’ll stand corrected. :slight_smile:

V/R

Andrew

11

For some reason, my child container can’t be bounded. This is what I got:

root@amd01-r:~# incus config set d1 limits.memory=2GB
root@amd01-r:~# incus exec d1 -- free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi       4.0Gi       5.6Gi       2.0Gi        23Gi        26Gi
Swap:          7.7Gi       101Mi       7.6Gi
root@amd01-r:~# incus config show d1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20241219_05:42)
  image.os: Debian
  image.release: bookworm
  image.serial: "20241219_05:42"
  image.type: squashfs
  image.variant: default
  limits.memory: 2GB
  volatile.base_image: 25a043e17b9965ae088164455f1c62b111b1d913ae5b94f10da236130e2139e4
  volatile.cloud-init.instance-id: f015cb98-3fbe-44ec-b68f-4ce178cd1221
  volatile.eth0.host_name: vethe592d7e6
  volatile.eth0.hwaddr: 00:16:3e:09:8b:61
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: aa704302-4576-4659-9a6c-1a038dc37834
  volatile.uuid.generation: aa704302-4576-4659-9a6c-1a038dc37834
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
root@amd01-r:~# incus restart d1
root@amd01-r:~# incus exec d1 -- free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi       4.0Gi       5.6Gi       2.0Gi        23Gi        26Gi
Swap:          7.7Gi       101Mi       7.6Gi

Have no idea why mine is not working like yours. Any clue on what I can try?

12

Gosh no, why is that different? That is…unexpected, given that this is an old feature.

I just tried it again, I even used GB instead of GiB in desperation, but I still got what I expected:-

andrew@Yoda:~$ incus launch images:debian/12 cx
andrew@Yoda:~$ incus exec cx -- free -h
               total        used        free      shared  buff/cache   available
Mem:           501Gi        29Mi       501Gi       100Ki        20Mi       501Gi
Swap:          975Mi          0B       975Mi
andrew@Yoda:~$ incus config set cx limits.memory=2GB
andrew@Yoda:~$ incus exec cx -- free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi        27Mi       1.8Gi        96Ki        20Mi       1.8Gi
Swap:             0B          0B          0B
andrew@Yoda:~$ incus delete -f cx
andrew@Yoda:~$

I running Incus 6.8 on Debian 12 AMD Epyc Milan hardware, I even tried it on EPYC Naples because I was curious and I still got the same result. This one’s out of my league, hopefully Stephane or one of the other smart dudes can advise. Weird. GL.

Addendum - another difference is that I get a different output from config show (this one from my Naples system):-

andrew@lando:~$ incus config show cx
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20241220_05:24)
  image.os: Debian
  image.release: bookworm
  image.serial: "20241220_05:24"
  image.type: squashfs
  image.variant: default
  limits.memory: 1GiB
  volatile.base_image: 3c3fc3df6d159cc658671da25cfa64ef114edadfaeff277e38d466eb57af3f97
  volatile.cloud-init.instance-id: 89ce936c-a5ec-4744-8037-01f2abb2f64e
  volatile.eth0.host_name: vetha885b77b
  volatile.eth0.hwaddr: 00:16:3e:10:31:82
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8bca3708-b9ba-42fa-b195-0c637fbcdfea
  volatile.uuid.generation: 8bca3708-b9ba-42fa-b195-0c637fbcdfea
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Andrew