Where did my memory go?

jono · May 19, 2022, 5:12am

Hi Guys,

I wanted to check how I can trace where all my memory goes after a week of uptime of a VM in our lxd cluster, it’s an older version of stable: (planning to update to 5.1)

  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: true
  server_name: lxdvm-dmz-02.uat-01.example.com
  server_pid: 3588
  server_version: 4.0.1
  storage: zfs
  storage_version: 0.8.3-1ubuntu12.1

root@lxdvm-dmz-02.uat-01.example.com:/home/jonathan# free -m
              total        used        free      shared  buff/cache   available
Mem:          32370       31175         250          19         944         845
Swap:         16384        4487       11897

root@lxdvm-dmz-02.uat-01.example.com:/home/jonathan# lxc list|grep dmz-02
| AUMELL122U-WEB-example.com                         | RUNNING | 10.14.0.96 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| AUMELL123U-example.com                             | RUNNING | 10.1.0.101 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| AUMELL125U-example.com                             | RUNNING | 10.14.0.212 (eth0)   |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-custapiportal-01-uat-01-example.com           | RUNNING | 10.14.0.22 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-ecatblah1bob-01-uat-01-example.com            | RUNNING | 10.1.0.125 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-ecatuatau-01-uat-01-example.com               | RUNNING | 10.1.0.132 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-ecatblah2-01-uat-01-example.com               | RUNNING | 10.1.0.127 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-ecatblah3-01-uat-01-example.com              | RUNNING | 10.1.0.124 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-finex-web-01-demo-01-example.com              | RUNNING | 10.1.0.19 (eth0)     |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-ftp-01-uat-01-example.com                     | RUNNING | 10.1.0.68 (eth0)     |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-iframe-hidden-uat-01-example.com              | RUNNING | 10.14.0.26 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-sftp-02-uat-01-example.com                    | RUNNING | 10.1.0.71 (eth0)     |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-versioning-01-uat-01-example.com              | RUNNING | 10.1.0.128 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| mcon-wordpress-02-uat-01-example.com               | RUNNING | 10.1.0.133 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| phoenix-uat-lb-example.com                         | RUNNING | 10.14.0.216 (eth0:0) |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |
| proxy-02-uat-01-example.com                        | RUNNING | 10.14.0.95 (eth0)    |      | CONTAINER | 0         | lxdvm-dmz-02.uat-01.example.com        |

root@lxdvm-dmz-02.uat-01.example.com:/home/jonathan# for container in `lxc list|grep dmz-02|col2` ; do echo $container ; lxc info $container|grep -i Mem ; done
AUMELL122U-WEB-example.com
  Memory usage:
    Memory (current): 201.35MB
AUMELL123U-example.com
  Memory usage:
    Memory (current): 271.00MB
AUMELL125U-example.com
  Memory usage:
    Memory (current): 118.09MB
mcon-custapiportal-01-uat-01-example.com
  Memory usage:
    Memory (current): 49.13MB
mcon-ecatblah1bob-01-uat-01-example.com
  Memory usage:
    Memory (current): 104.63MB
mcon-ecatuatau-01-uat-01-example.com
  Memory usage:
    Memory (current): 36.29MB
mcon-ecatblah2-01-uat-01-example.com
  Memory usage:
    Memory (current): 55.52MB
mcon-ecatblah3-01-uat-01-example.com
  Memory usage:
    Memory (current): 45.14MB
mcon-finex-web-01-demo-01-example.com
  Memory usage:
    Memory (current): 81.96MB
mcon-ftp-01-uat-01-example.com
  Memory usage:
    Memory (current): 99.69MB
mcon-iframe-officeworks-01-uat-01-example.com
  Memory usage:
    Memory (current): 48.20MB
mcon-sftp-02-uat-01-example.com
  Memory usage:
    Memory (current): 153.90MB
mcon-versioning-01-uat-01-example.com
  Memory usage:
    Memory (current): 39.17MB
mcon-wordpress-02-uat-01-example.com
  Memory usage:
    Memory (current): 46.85MB
phoenix-uat-lb-example.com
  Memory usage:
    Memory (current): 132.25MB
proxy-02-uat-01-example.com
  Memory usage:
    Memory (current): 135.08MB

systemd-cgtop shows:

Control Group                                                                                                                                                                                                          Tasks   %CPU   Memory  Input/s Output/s
/                                                                                                                                                                                                                       2088  174.3    31.4G        -        -
lxc.payload.AUMELL123U-example.com                                                                                                                                                                                          175   59.8   220.3M        -        -
lxc.payload.proxy-02-uat-01-example.com                                                                                                                                                                                     115    0.9   210.8M        -        -
lxc.payload.AUMELL125U-example.com                                                                                                                                                                                           75    0.2   201.5M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice                                                                                                                                                                        114    0.9   196.8M        -        -
system.slice                                                                                                                                                                                                              89      -   179.2M        -        -
lxc.payload.mcon-ecatblah1bob-01-uat-01-example.com                                                                                                                                                                         111   26.8   156.8M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com                                                                                                                                                                                 111    8.5   156.1M        -        -
lxc.payload.phoenix-uat-lb-example.com                                                                                                                                                                                      143    5.6   143.4M        -        -
lxc.payload.phoenix-uat-lb-example.com/system.slice                                                                                                                                                                         128    5.6   126.2M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com                                                                                                                                                                            58    0.0   106.4M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com                                                                                                                                                                                   55    7.0    97.4M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com/system.slice                                                                                                                                                               57      -    94.9M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/system.slice                                                                                                                                                                     50      -    86.6M        -        -
lxc.payload.AUMELL122U-WEB-example.com                                                                                                                                                                                       50    0.3    85.4M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com/system.slice                                                                                                                                                                      47      -    77.9M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice/metricbeat.service                                                                                                                                                      14    0.0    66.4M        -        -
lxc.payload.mcon-ecatblah2-01-uat-01-example.com                                                                                                                                                                             91    0.2    55.7M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/user.slice                                                                                                                                                                       60      -    51.8M        -        -
lxc.payload.mcon-ecatblah3-01-uat-01-example.com                                                                                                                                                                            89    0.2    49.6M        -        -
lxc.payload.phoenix-uat-lb-example.com/system.slice/haproxy.service                                                                                                                                                           2    4.9    49.6M        -        -
lxc.payload.mcon-custapiportal-01-uat-01-example.com                                                                                                                                                                         21    0.0    47.9M        -        -
lxc.payload.mcon-wordpress-02-uat-01-example.com                                                                                                                                                                            204    0.6    47.3M        -        -
lxc.payload.mcon-iframe-officeworks-01-uat-01-example.com                                                                                                                                                                    70    0.0    46.4M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice/packetbeat.service                                                                                                                                                      11    0.1    45.6M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com/system.slice/system-postfix.slice                                                                                                                                          34      -    39.2M        -        -
lxc.payload.mcon-versioning-01-uat-01-example.com                                                                                                                                                                            62    0.2    37.8M        -        -
lxc.payload.mcon-custapiportal-01-uat-01-example.com/system.slice                                                                                                                                                            20      -    37.4M        -        -
lxc.payload.mcon-iframe-officeworks-01-uat-01-example.com/system.slice                                                                                                                                                       69      -    36.6M        -        -
lxc.payload.mcon-ecatuatau-01-uat-01-example.com                                                                                                                                                                             52    0.3    36.0M        -        -
system.slice/systemd-udevd.service                                                                                                                                                                                        27      -    35.7M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/system.slice/systemd-journald.service                                                                                                                                             1      -    29.6M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice/wazuh-agent.service                                                                                                                                                     45    0.5    28.5M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com/system.slice/proftpd.service                                                                                                                                                      12      -    26.1M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com/system.slice/systemd-udevd.service                                                                                                                                                20      -    23.4M        -        -
user.slice                                                                                                                                                                                                                 8      -    21.2M        -        -
user.slice/user-1005.slice                                                                                                                                                                                                 8      -    21.0M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice/filebeat.service                                                                                                                                                        13    0.1    19.2M        -        -
lxc.payload.phoenix-uat-lb-example.com/system.slice/wazuh-agent.service                                                                                                                                                      31    0.2    18.3M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/system.slice/systemd-udevd.service                                                                                                                                               15      -    17.9M        -        -
user.slice/user-1005.slice/session-1284796.scope                                                                                                                                                                           6      -    17.8M        -        -
system.slice/snapd.service                                                                                                                                                                                                19      -    17.1M        -        -
lxc.payload.phoenix-uat-lb-example.com/system.slice/osqueryd.service                                                                                                                                                         14    0.1    16.9M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/system.slice/ssh.service                                                                                                                                                          2      -    16.1M        -        -
system.slice/nagios-nrpe-server.service                                                                                                                                                                                    1      -    16.0M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com/system.slice/cron.service                                                                                                                                                   4      -    14.6M        -        -
system.slice/multipathd.service                                                                                                                                                                                            7      -    13.2M        -        -
system.slice/open-vm-tools.service                                                                                                                                                                                         3      -    11.7M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com/system.slice/systemd-journald.service                                                                                                                                       1      -    11.6M        -        -
lxc.payload.mcon-finex-web-01-demo-01-example.com/system.slice/apache2.service                                                                                                                                                6      -    10.4M        -        -
lxc.payload.mcon-iframe-officeworks-01-uat-01-example.com/system.slice/apache2.service                                                                                                                                       55      -    10.2M        -        -
lxc.payload.mcon-custapiportal-01-uat-01-example.com/system.slice/apache2.service                                                                                                                                             6      -     9.9M        -        -
lxc.payload.phoenix-uat-lb-example.com/system.slice/squid.service                                                                                                                                                             4    0.0     9.2M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/user.slice/user-1092.slice                                                                                                                                                       15      -     9.1M        -        -
system.slice/system-postfix.slice                                                                                                                                                                                          3      -     8.9M        -        -
system.slice/system-postfix.slice/postfix@-.service                                                                                                                                                                        3      -     8.9M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com/user.slice                                                                                                                                                                         7      -     8.8M        -        -
lxc.payload.mcon-ftp-01-uat-01-example.com/user.slice/user-1005.slice                                                                                                                                                         7      -     8.8M        -        -
lxc.payload.proxy-02-uat-01-example.com/system.slice/haproxy.service                                                                                                                                                          2    0.1     8.0M        -        -
lxc.payload.mcon-sftp-02-uat-01-example.com/user.slice/user-2356.slice                                                                                                                                                        9      -     7.8M        -        -
system.slice/cron.service                                                                                                                                                                                                  1      -     7.7M        -        -
system.slice/systemd-journald.service                                                                                                                                                                                      1      -     7.6M        -        -
system.slice/ssh.service                                                                                                                                                                                                   1      -     6.9M        -        -
lxc.payload.mcon-custapiportal-01-uat-01-example.com/system.slice/cron.service                                                                                                                                                1      -     6.8M        -        -

Any ideas, is there a known memory leak with lxc or lxfs with this version?

Thank you in advance.

Regards,
Jonathan

tomp · May 19, 2022, 8:30am

What is the memory usage of processes on the host system, rather than of the containers?

jono · May 19, 2022, 9:48am

smem -twk
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 30.9G 743.8M 30.2G
userspace memory 575.7M 130.3M 445.4M
free memory 149.4M 149.4M 0

                          31.6G    1023.5M      30.6G

But I am not able to pinpoint exactly where the memory is going…

cat /proc/slabinfo | awk ‘{print $2,$1}’ |sort -n
0 arc_buf_hdr_t_full_crypt
0 arc_buf_hdr_t_l2only
0 btrfs_delayed_node
0 btrfs_free_space_bitmap
0 btrfs_inode
0 btrfs_ordered_extent
0 ddt_entry_cache
0 dma-kmalloc-128
0 dma-kmalloc-16
0 dma-kmalloc-192
0 dma-kmalloc-1k
0 dma-kmalloc-256
0 dma-kmalloc-2k
0 dma-kmalloc-32
0 dma-kmalloc-4k
0 dma-kmalloc-64
0 dma-kmalloc-8
0 dma-kmalloc-8k
0 dma-kmalloc-96
0 dm_uevent
0 ecryptfs_auth_tok_list_item
0 ecryptfs_file_cache
0 ecryptfs_inode_cache
0 ecryptfs_key_record_cache
0 fat_cache
0 fat_inode_cache
0 fsverity_info
0 iint_cache
0 ip4-frags
0 ip6-frags
0 kcf_areq_cache
0 kcf_context_cache
0 kcf_sreq_cache
0 kcopyd_job
0 kmalloc-rcl-16
0 kmalloc-rcl-1k
0 kmalloc-rcl-2k
0 kmalloc-rcl-32
0 kmalloc-rcl-4k
0 kmalloc-rcl-512
0 kmalloc-rcl-8
0 kmalloc-rcl-8k
0 nfs_direct_cache
0 userfaultfd_ctx_cache
name #

slabinfo
15 dmaengine-unmap-256
16 lz4_cache
28 zio_data_buf_16384
30 dmaengine-unmap-128
32 zio_data_buf_10240
32 zio_data_buf_12288
32 zio_data_buf_14336
34 zio_buf_10240
36 zio_buf_12288
44 zio_buf_14336
48 zio_data_buf_7168
60 request_queue
60 zio_buf_7168
60 zio_data_buf_6144
64 dma-kmalloc-512
64 zio_buf_8192
68 zio_data_buf_5120
68 zio_data_buf_8192
69 zio_buf_6144
80 zio_buf_5120
84 dax_cache
90 net_namespace
120 zio_data_buf_4096
144 biovec-128
170 mod_hash_entries
173 zio_buf_4096
180 zio_buf_2048
184 names_cache
186 numa_policy
198 zio_buf_3584
212 zio_buf_3072
221 biovec-max
256 biovec-64
256 dnotify_struct
270 zio_data_buf_3072
279 zio_data_buf_3584
286 PINGv6
296 uts_namespace
305 user_namespace
312 bdev_cache
312 pid_namespace
325 zio_buf_2560
348 zio_data_buf_2560
368 xfrm_state
368 zfs_znode_hold_cache
376 taskstats
392 ext4_groupinfo_4k
424 request_sock_TCPv6
424 zil_zcw_cache
432 zio_data_buf_2048
442 mqueue_inode_cache
504 khugepaged_mm_slot
510 ext4_system_zone
512 dquot
512 ext4_allocation_context
512 ext4_io_end
512 spl_vn_cache
512 spl_vn_file_cache
530 request_sock_TCP
546 zio_buf_1536
560 fuse_request
576 kmalloc-rcl-256
609 zio_data_buf_1536
657 mbcache
680 jbd2_journal_handle
920 rpc_inode_cache
1024 ext4_pending_reservation
1024 jbd2_revoke_table_s
1040 zio_cache
1161 zil_lwb_cache
1275 task_group
1364 zio_data_buf_1024
1404 nfs_read_data
1418 skbuff_fclone_cache
1443 file_lock_cache
1512 skbuff_ext_cache
1768 jbd2_journal_head
1792 scsi_sense_cache
1806 squashfs_inode_cache
1932 trace_event_file
1959 zio_buf_1024
2046 tw_sock_TCPv6
2048 fsnotify_mark_connector
2080 TCPv6
2535 Acpi-ParseExt
2795 zio_data_buf_512
3264 hugetlbfs_inode_cache
3436 zio_buf_16384
3582 nf_conntrack
3792 UDPv6
3822 zio_buf_512
4212 RAWv6
4796 pool_workqueue
4828 kmalloc-8k
5081 nfs_inode_cache
5659 arc_buf_t
6444 task_struct
6468 proc_dir_entry
6561 nsproxy
6724 fuse_inode
7140 zio_link_cache
7344 ext4_extent_status
7344 mnt_cache
7350 kmalloc-rcl-192
7584 RAW
7657 kmem_cache
7902 ext4_inode_cache
8304 sio_cache_2
9169 sio_cache_1
9967 zfs_znode_cache
10575 sighand_cache
11309 kmem_cache_node
11396 sa_cache
11520 sio_cache_0
11592 Acpi-Operand
11721 vmap_area
12408 tw_sock_TCP
13082 UDP
13260 ftrace_event_field
14177 TCP
14752 kmalloc-128
16250 uid_cache
18139 signal_cache
18350 mm_struct
19187 kmalloc-2k
20066 abd_t
22111 shmem_inode_cache
25788 kmalloc-256
25956 files_cache
26628 PING
29824 kmalloc-rcl-128
34020 kmalloc-1k
36708 lsm_file_cache
38879 arc_buf_hdr_t_full
41412 pde_opener
42270 kmalloc-96
43267 dmu_buf_impl_t
44574 buffer_head
44968 eventpoll_pwq
46064 dnode_t
47022 task_delay_info
48800 kmalloc-192
52164 kmalloc-rcl-96
76068 skbuff_head_cache
76321 cred_jar
79152 sock_inode_cache
87264 pid
89903 proc_inode_cache
107594 inode_cache
119430 kmalloc-rcl-64
171109 kmalloc-8
174689 range_seg_cache
223505 anon_vma
238616 filp
260283 kmalloc-512
266845 dentry
316897 vm_area_struct
338997 radix_tree_node
364613 kmalloc-16
377524 anon_vma_chain
982224 kmalloc-4k
984759 Acpi-State
1152280 kmalloc-64
1302412 kernfs_node_cache
2243770 kmalloc-32

tomp · May 19, 2022, 9:51am

Well you can rule out LXD itself by reloading the LXD process using:

sudo systemctl reload snap.lxd.daemon

This will restart LXD without restarting your instances or LXCFS.

jono · May 20, 2022, 8:21am

Thank you Thomas, yep tried that but not freeing any memory. Still not sure where the memory is being consumed.

sandersbud4 · May 21, 2022, 11:39am

I have also tried mentioned methods here but none of them is working for me.

jono · May 25, 2022, 8:09am

Hi Thomas,

I have noticed this on the affected VM:

udev goes crazy when I start one particular container, if I stop the container the issue goes away?
Any idea?

udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[14568.511975] remove /kernel/slab/:A-0001024/cgroup/PING(177612:session-24369.scope) (cgroup)
KERNEL[14568.512020] remove /kernel/slab/sock_inode_cache/cgroup/sock_inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.512030] remove /kernel/slab/skbuff_head_cache/cgroup/skbuff_head_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.512038] remove /kernel/slab/:A-0000080/cgroup/task_delay_info(177612:session-24369.scope) (cgroup)
KERNEL[14568.512046] remove /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.512055] remove /kernel/slab/shmem_inode_cache/cgroup/shmem_inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513293] remove /kernel/slab/:A-0000256/cgroup/filp(177612:session-24369.scope) (cgroup)
UDEV [14568.513323] remove /kernel/slab/skbuff_head_cache/cgroup/skbuff_head_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513334] remove /kernel/slab/inode_cache/cgroup/inode_cache(177612:session-24369.scope) (cgroup)
UDEV [14568.513346] remove /kernel/slab/:A-0000256/cgroup/filp(177612:session-24369.scope) (cgroup)
KERNEL[14568.513355] remove /kernel/slab/:A-0000208/cgroup/vm_area_struct(177612:session-24369.scope) (cgroup)
UDEV [14568.513365] remove /kernel/slab/inode_cache/cgroup/inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513374] remove /kernel/slab/mm_struct/cgroup/mm_struct(177612:session-24369.scope) (cgroup)
UDEV [14568.513383] remove /kernel/slab/proc_inode_cache/cgroup/proc_inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513392] remove /kernel/slab/:A-0000704/cgroup/files_cache(177612:session-24369.scope) (cgroup)
UDEV [14568.513419] remove /kernel/slab/mm_struct/cgroup/mm_struct(177612:session-24369.scope) (cgroup)
KERNEL[14568.513428] remove /kernel/slab/:A-0001088/cgroup/signal_cache(177612:session-24369.scope) (cgroup)
UDEV [14568.513448] remove /kernel/slab/sock_inode_cache/cgroup/sock_inode_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513456] remove /kernel/slab/sighand_cache/cgroup/sighand_cache(177612:session-24369.scope) (cgroup)
UDEV [14568.513466] remove /kernel/slab/:A-0000704/cgroup/files_cache(177612:session-24369.scope) (cgroup)
KERNEL[14568.513483] remove /kernel/slab/task_struct/cgroup/task_struct(177612:session-24369.scope) (cgroup)
UDEV [14568.513495] remove /kernel/slab/:A-0001024/cgroup/PING(177612:session-24369.scope) (cgroup)
KERNEL[14568.513506] remove /kernel/slab/:A-0000064/cgroup/anon_vma_chain(177612:session-24369.scope) (cgroup)
UDEV [14568.513519] remove /kernel/slab/:A-0000208/cgroup/vm_area_struct(177612:session-24369.scope) (cgroup)
KERNEL[14568.513527] remove /kernel/slab/anon_vma/cgroup/anon_vma(177612:session-24369.scope) (cgroup)
UDEV [14568.513538] remove /kernel/slab/:A-0000080/cgroup/task_delay_info(177612:session-24369.scope) (cgroup)

jono · May 25, 2022, 8:47am

The container is running and sftp server and using mysecureshell, somehow users logging in triggers udev on the HOST/VM to go crazy? when I stop the ssh service monitor calms down?

jono · May 27, 2022, 12:05am

I found the issue, maybe this helps someone out in the future:

Wonderful DBUS on the container everytime someone logged into the sftp server triggered the KERNEL/UDEV on the vm (observed with udevadm monitor), I cloned the container and started check
all services. As soon as I stopped dbus all the problems gone. dbus on a server? disabled it, rebooted and memory issue solved.

Uninstall dbus will uninstall netplan.io and some other packages…really annoying. I installed netplan.io again and left all the other packages. rebooted, woohooo 3 days of sold checking paid off.

sandersbud4 · May 31, 2022, 12:56pm

Thanks, man! Indeed it has helped me a lot.

Where did my memory go?

smem -twk Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 30.9G 743.8M 30.2G userspace memory 575.7M 130.3M 445.4M free memory 149.4M 149.4M 0

smem -twk
Area Used Cache Noncache
firmware/hardware 0 0 0
kernel image 0 0 0
kernel dynamic memory 30.9G 743.8M 30.2G
userspace memory 575.7M 130.3M 445.4M
free memory 149.4M 149.4M 0