Problems to run podman containers inside LXD container

So, I understand the scenario I’m going to describe is a bit convoluted, but for reasons regarding my CI/CD infrastructure, I’m currently in a situation where I want to build and run podman images inside LXD containers.
I’m not trying to use podman on rootless mode, so every podman command is ran as sudo.

But when running a fresh LXD container using the base image images:centos/8-Stream, I install podman and try pulling the quay.io/centos/centos:stream8 podman image, and this pull fails with a pretty generic error message.

I am aware of this fix/workaround:

  • Editing the file at /etc/containers/storage.conf and uncommenting this line:
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

But at this point, I just wanted to understand a little better why this is happening, since I’m not trying to use podman in rootless mode, which is one of the main reasons I believe should cause such issue.

On top of that, I am able to pull and work just fine with some newer podman images, such as quay.io/centos/centos:stream9 and registry.fedoraproject.org/fedora:35.

I understand from this post that from kernel 5.13 onwards podman is getting rootless overlay support, and since I still have to support some legacy servers running CentOS Stream 8 I am pseudo-locked to kernel 4.18. But even then, shouldn’t this kernel 5.13 support be related only to podman on rootless mode? Why am I getting such problems even running it with sudo? Can anyone shed some light on the situation for me?

Also, please let me know if any more info is needed.

I really apreciate any help!

Steps to reproduce

  1. Boot up a new machine or vm running CentOS Stream 8
  2. Launch a fresh LXD container using the images:centos/8-Stream base image:
lxc launch images:centos/8-Stream testcontainer
  1. From inside the LXD container, simply install podman and try pulling the quay.io/centos/centos:stream8 image:
lxc exec testcontainer -- bash
dnf update -y && dnf install podman -y
podman pull quay.io/centos/centos:stream8
  1. You should get an output like this:
Trying to pull quay.io/centos/centos:stream8...
Getting image source signatures
Copying blob a0b8f3931ffa skipped: already exists  
Copying blob 04f0eb705bff done  
Copying blob 4a7e61ebcfec done  
Copying blob 1ac891d08dc2 done  
Error: writing blob: adding layer with blob "sha256:04f0eb705bffc1db22f04bd42987ebb9d5e40c08c0253d0c2a56881c75bc6af8": Error processing tar file(exit status 1): operation not permitted

Useful Information

  • Distribution: CentOS
  • Distribution version: Stream 8
  • Kernel version: 4.18.0-383.el8.x86_64
  • LXD version: 5.1-4ae3604
  • Host Partition File System: ext4

Useful outputs

lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICGzCCAaGgAwIBAgIQd+pUOTDr2A5+YEbHyKR4xjAKBggqhkjOPQQDAzA8MRww
    GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRwwGgYDVQQDDBNyb290QHBvZG1h
    bi10ZXN0aW5nMB4XDTIyMDUxMTA5MTUyNFoXDTMyMDUwODA5MTUyNFowPDEcMBoG
    A1UEChMTbGludXhjb250YWluZXJzLm9yZzEcMBoGA1UEAwwTcm9vdEBwb2RtYW4t
    dGVzdGluZzB2MBAGByqGSM49AgEGBSuBBAAiA2IABFqXQbRJx2Lmnnbahbyy+w/f
    dU2j43Xvj8z2VLm9n+sNKjaxzsmtarYYtcSERh2Yxr7CJ08QtiT+E/3WmMlnMSnV
    5mfU+x4uXinXcoCxEDlcEsV258IfsIjTPiov6zGaYaNoMGYwDgYDVR0PAQH/BAQD
    AgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwMQYDVR0RBCow
    KIIOcG9kbWFuLXRlc3RpbmeHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZI
    zj0EAwMDaAAwZQIxAOrmwS7Qx/Zs0mjVkLFk/rW6K556yHePbM8o9Q0gFwZF3JWD
    eaUbdnTkib/9T8iOEgIwBvpwRKOQv+USyaYEHejMCOOSo6s2VswHIuVWAgmnjRcy
    C9tb9eJFSMUlGVXLk/o3
    -----END CERTIFICATE-----
  certificate_fingerprint: 61a2d8d6e4b2165ca9917ce25fd5239a24c94415bc359c02e49d5ed130f51906
  driver: lxc
  driver_version: 4.0.12
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "false"
    netnsid_getifaddrs: "true"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 4.18.0-383.el8.x86_64
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: CentOS Stream
  os_version: "8"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: podman-testing
  server_pid: 5591
  server_version: "5.1"
  storage: lvm
  storage_version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.43.0
  storage_supported_drivers:
  - name: btrfs
    version: 5.4.1
    remote: false
  - name: cephfs
    version: 15.2.14
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.43.0
    remote: false
  - name: ceph
    version: 15.2.14
    remote: true
  
lxc storage ls
+---------+--------+--------------------------------------------+-------------+---------+---------+
|  NAME   | DRIVER |                   SOURCE                   | DESCRIPTION | USED BY |  STATE  |
+---------+--------+--------------------------------------------+-------------+---------+---------+
| default | lvm    | /var/snap/lxd/common/lxd/disks/default.img |             | 3       | CREATED |
+---------+--------+--------------------------------------------+-------------+---------+---------+
lxc storage show default
config:
  lvm.thinpool_name: LXDThinPool
  lvm.vg_name: default
  size: 7GB
  source: /var/snap/lxd/common/lxd/disks/default.img
description: ""
name: default
driver: lvm
used_by:
- /1.0/images/ed4a6eb25898b301c93dafa6db5b081d207ca2db53a1a6a6a061baa8f78e11c9
- /1.0/instances/testcontainer
- /1.0/profiles/default
status: Created
locations:
- none

dmesg
[    5.890595] Console: switching to colour frame buffer device 128x48
[    5.897174] virtio_gpu virtio0: [drm] fb0: virtio_gpudrmfb frame buffer device
[    8.042443] virtio_net virtio1 eth0: renamed from enp1s0
[    8.257117] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    9.651824] EXT4-fs (sda1): resizing filesystem from 878848 to 9983483 blocks
[    9.755856] EXT4-fs (sda1): resized filesystem to 9983483
[  154.925329] SELinux:  Converting 347 SID table entries...
[  154.926800] SELinux:  policy capability network_peer_controls=1
[  154.927347] SELinux:  policy capability open_perms=1
[  154.927800] SELinux:  policy capability extended_socket_class=1
[  154.928332] SELinux:  policy capability always_check_network=0
[  154.928854] SELinux:  policy capability cgroup_seclabel=1
[  154.929338] SELinux:  policy capability nnp_nosuid_transition=1
[  233.230875] SELinux:  Converting 361 SID table entries...
[  233.233687] SELinux:  policy capability network_peer_controls=1
[  233.234693] SELinux:  policy capability open_perms=1
[  233.235547] SELinux:  policy capability extended_socket_class=1
[  233.236530] SELinux:  policy capability always_check_network=0
[  233.237498] SELinux:  policy capability cgroup_seclabel=1
[  233.238421] SELinux:  policy capability nnp_nosuid_transition=1
[  249.611083] loop: module loaded
[  249.615297] loop0: detected capacity change from 0 to 4096
[  249.638837] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[  255.193719] loop0: detected capacity change from 0 to 4096
[  258.022818] loop0: detected capacity change from 0 to 46845952
[  259.519094] loop1: detected capacity change from 0 to 4096
[  264.699867] loop1: detected capacity change from 0 to 64909312
[  267.866600] loop2: detected capacity change from 0 to 84246528
[  333.095362] new mount options do not match the existing superblock, will be ignored
[  333.123335] fuse: init (API version 7.33)
[  334.164721] NET: Registered protocol family 40
[  335.685472] device-mapper: uevent: version 1.0.3
[  335.687910] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com
[  379.802312] loop3: detected capacity change from 0 to 7000000000
[  381.449243] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[  381.478021] IPv6: ADDRCONF(NETDEV_UP): lxdbr0: link is not ready
[  422.183322] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: discard
[  440.881989] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: discard
[  441.545705] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: discard
[  441.875666] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: discard
[  444.458185] IPv6: ADDRCONF(NETDEV_UP): veth300231c0: link is not ready
[  444.522649] lxdbr0: port 1(veth300231c0) entered blocking state
[  444.523758] lxdbr0: port 1(veth300231c0) entered disabled state
[  444.524947] device veth300231c0 entered promiscuous mode
[  444.826585] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[  444.847307] physG0ixXo: renamed from veth02a33f28
[  444.853987] eth0: renamed from physG0ixXo
[  444.859131] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  444.861664] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  444.864082] lxdbr0: port 1(veth300231c0) entered blocking state
[  444.865964] lxdbr0: port 1(veth300231c0) entered forwarding state
[  444.868368] IPv6: ADDRCONF(NETDEV_CHANGE): lxdbr0: link becomes ready
[  491.336615] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
[  491.462665] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

lxc info testcontainer --show-log
Name: testcontainer
Status: RUNNING
Type: container
Architecture: x86_64
PID: 6016
Created: 2022/05/11 09:16 UTC
Last Used: 2022/05/11 09:17 UTC

Resources:
  Processes: 12
  Disk usage:
    root: 1.13GiB
  CPU usage:
    CPU usage (in seconds): 53
  Memory usage:
    Memory (current): 493.17MiB
    Memory (peak): 723.67MiB
  Network usage:
    eth0:
      Type: broadcast
      State: UP
      Host interface: veth300231c0
      MAC address: 00:16:3e:80:67:05
      MTU: 1500
      Bytes received: 425.48MB
      Bytes sent: 968.14kB
      Packets received: 35778
      Packets sent: 13318
      IP addresses:
        inet:  10.41.51.38/24 (global)
        inet6: fe80::216:3eff:fe80:6705/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

Log:

lxc testcontainer 20220511091714.843 WARN     conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc testcontainer 20220511091714.844 WARN     conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc testcontainer 20220511091714.844 WARN     conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc testcontainer 20220511091714.844 WARN     conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc testcontainer 20220511091720.849 WARN     attach - attach.c:get_attach_context:477 - No security context received
lxc testcontainer 20220511093245.762 WARN     attach - attach.c:get_attach_context:477 - No security context received

lxc config show testcontainer --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Centos 8-Stream amd64 (20220511_07:08)
  image.os: Centos
  image.release: 8-Stream
  image.serial: "20220511_07:08"
  image.type: squashfs
  image.variant: default
  volatile.base_image: ed4a6eb25898b301c93dafa6db5b081d207ca2db53a1a6a6a061baa8f78e11c9
  volatile.cloud-init.instance-id: 06a6a788-02e8-4963-9795-2dbc489b8ab1
  volatile.eth0.host_name: veth300231c0
  volatile.eth0.hwaddr: 00:16:3e:80:67:05
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 1b8b906d-1595-4d06-8bb8-c5fcbe092ca6
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

/var/snap/lxd/common/lxd/logs/lxd.log
time="2022-05-11T09:15:24Z" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
time="2022-05-11T09:15:24Z" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
time="2022-05-11T09:15:24Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2022-05-11T09:15:24Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
time="2022-05-11T09:15:25Z" level=warning msg="Failed to initialize fanotify, falling back on fsnotify" err="Failed to initialize fanotify: invalid argument"

@sdeziel you said it was working for you, presumably that was 5.1 snap too but on Ubuntu rather than CentOS8?

Correct, Ubuntu 20.04 + 5.13 kernel + LXD snap 5.1. I had to specify the fuse thing in the mount_program option otherwise it was complaining about zfs not supporting overlay something (don’t remember the exact message).

Yeah, in the case of a zfs filesystem I understand the need of that configuration and the limitation is clearly stated in the error message, but I’m still trying to completely understand why that “fix” is needed to an image not using rootless mode on a ext4 file system.

And just to update on the main post, I just fired up a new VM running CentOS Stream 9, which runs kernel 5.14.0 and could replicate the exact same error (LXD snap 5.1-1f6f485). From this I want to draw the conclusion that the problem is not the aforementioned kernel 5.13+ support

@lucasbaile this is my config for running kubernetes inside LXD so you may not need all of this, but even if it is overkill for your requirement, it should work. So perhaps some or all of the following may be helpful … try using this 'k8s-weavenet" profile to create your centos/8-stream LXD container (of course edit some parts to match your system). Also, you can probably get away with 2 cpu and 4 Gb but I recommend 4 cpu / 8 Gb if you have it.

[orabuntu@f34sv1 ~]$ lxc profile list
+--------------+----------------------------------------+---------+
|     NAME     |              DESCRIPTION               | USED BY |
+--------------+----------------------------------------+---------+
| default      | Default LXD profile                    | 0       |
+--------------+----------------------------------------+---------+
| k8s-weavenet | Kubernetes LXD WeaveNet                | 4       |
+--------------+----------------------------------------+---------+
| olxc_sw1a    | Orabuntu-LXD OpenvSwitch Clone profile | 4       |
+--------------+----------------------------------------+---------+
| olxc_sx1a    | Orabuntu-LXD OpenvSwitch Seeds profile | 2       |
+--------------+----------------------------------------+---------+
[orabuntu@f34sv1 ~]$ lxc profile show k8s-weavenet

config:
limits.cpu: “4”
limits.memory: 8GB
limits.memory.swap: “false”
linux.kernel_modules: ip_tables,ip6_tables,nf_nat,overlay,br_netfilter
raw.lxc: “lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
sys:rw\nlxc.mount.entry = /dev/kmsg dev/kmsg none defaults,bind,create=file”
security.nesting: “true”
security.privileged: “true”
description: Kubernetes LXD WeaveNet
devices:
eth0:
name: eth0
nictype: bridged
parent: sw1a
type: nic
root:
path: /
pool: local
type: disk
name: k8s-weavenet
used_by: []


[orabuntu@f34sv1 ~]$ lxc list
+----------+---------+------------------------+------+-----------+-----------+----------+
|   NAME   |  STATE  |          IPV4          | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+----------+---------+------------------------+------+-----------+-----------+----------+
| maestro  | RUNNING | 10.244.0.1 (cni0)      |      | CONTAINER | 0         | f34sv1   |
|          |         | 10.244.0.0 (flannel.1) |      |           |           |          |
|          |         | 10.209.53.4 (eth0)     |      |           |           |          |

Note that you may also have to run the following command regarding kmsg:

lxc config device add “ContainerName” “kmsg” unix-char source="/dev/kmsg" path="/dev/kmsg"’

Once that LXD container is built, then install podman inside the container in the usual way:

dnf -y install podman skopeo buildah

(I always install all 3 together)


Now you should be able to pull your required image:

[root@maestro ~]# which podman
/usr/bin/podman
[root@maestro ~]# podman pull quay.io/centos/centos:stream8
Trying to pull quay.io/centos/centos:stream8...
Getting image source signatures
Copying blob a0b8f3931ffa done  
Copying blob 1ac891d08dc2 done  
Copying blob 04f0eb705bff done  
Copying blob 4a7e61ebcfec done  
Copying config 67daa3fbd6 done  
Writing manifest to image destination
Storing signatures
67daa3fbd687e011241a14ec37dc733e19cccc35faf944a0455dc78dbddb9caf

details of LXD container:

[root@maestro ~]# uname -a
Linux maestro 5.17.6-100.fc34.x86_64 #1 SMP PREEMPT Mon May 9 14:41:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@maestro ~]# cat /etc/centos-release 
CentOS Stream release 8
[root@maestro ~]#

note the LXD container is built on a ZFS pool:

[orabuntu@f34sv1 lxd-provisioning]$ lxc storage list
+------------+--------+------------------+---------+---------+
|    NAME    | DRIVER |   DESCRIPTION    | USED BY |  STATE  |
+------------+--------+------------------+---------+---------+
| containerd | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| docker     | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| kubelet    | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| local      | zfs    | f34sv1-olxd-f5bf | 22      | CREATED |
+------------+--------+------------------+---------+---------+
[orabuntu@f34sv1 lxd-provisioning]$
1 Like

Thank you very much, @Gilbert_Standen!

I did some testing here and what did the trick was specifically the security.privileged: “true” part, which is much less annoying in my current flow than the previous trick with the mount_program config