Unable to start containers after restoring a linode image

I had set up lxd containers on ubuntu 18.04 LTS, Recently the subscription expired on linode and the server was shut down. However before it was shutdown an image was created. The subscription has now been renewed on linode. However after deploying the image i am unable to start the containers it is complaining that it can not find the ZFS module even though it is installed.

if i run lxc start tomcat --debug i get the following response

root@localhost:~# lxc start tomcat --debug
DBUG[02-13|10:37:52] Connecting to a local LXD over a Unix socket
DBUG[02-13|10:37:52] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
DBUG[02-13|10:37:52] Got response struct from LXD
DBUG[02-13|10:37:52]
{
“config”: {},
“api_extensions”: [
“storage_zfs_remove_snapshots”,
“container_host_shutdown_timeout”,
“container_stop_priority”,
“container_syscall_filtering”,
“auth_pki”,
“container_last_used_at”,
“etag”,
“patch”,
“usb_devices”,
“https_allowed_credentials”,
“image_compression_algorithm”,
“directory_manipulation”,
“container_cpu_time”,
“storage_zfs_use_refquota”,
“storage_lvm_mount_options”,
“network”,
“profile_usedby”,
“container_push”,
“container_exec_recording”,
“certificate_update”,
“container_exec_signal_handling”,
“gpu_devices”,
“container_image_properties”,
“migration_progress”,
“id_map”,
“network_firewall_filtering”,
“network_routes”,
“storage”,
“file_delete”,
“file_append”,
“network_dhcp_expiry”,
“storage_lvm_vg_rename”,
“storage_lvm_thinpool_rename”,
“network_vlan”,
“image_create_aliases”,
“container_stateless_copy”,
“container_only_migration”,
“storage_zfs_clone_copy”,
“unix_device_rename”,
“storage_lvm_use_thinpool”,
“storage_rsync_bwlimit”,
“network_vxlan_interface”,
“storage_btrfs_mount_options”,
“entity_description”,
“image_force_refresh”,
“storage_lvm_lv_resizing”,
“id_map_base”,
“file_symlinks”,
“container_push_target”,
“network_vlan_physical”,
“storage_images_delete”,
“container_edit_metadata”,
“container_snapshot_stateful_migration”,
“storage_driver_ceph”,
“storage_ceph_user_name”,
“resource_limits”,
“storage_volatile_initial_source”,
“storage_ceph_force_osd_reuse”,
“storage_block_filesystem_btrfs”,
“resources”,
“kernel_limits”,
“storage_api_volume_rename”,
“macaroon_authentication”,
“network_sriov”,
“console”,
“restrict_devlxd”,
“migration_pre_copy”,
“infiniband”,
“maas_network”,
“devlxd_events”,
“proxy”,
“network_dhcp_gateway”,
“file_get_symlink”,
“network_leases”,
“unix_device_hotplug”,
“storage_api_local_volume_handling”,
“operation_description”,
“clustering”,
“event_lifecycle”,
“storage_api_remote_volume_handling”,
“nvidia_runtime”,
“candid_authentication”,
“candid_config”,
“candid_config_key”,
“usb_optional_vendorid”
],
“api_status”: “stable”,
“api_version”: “1.0”,
“auth”: “trusted”,
“public”: false,
“auth_methods”: [
“tls”
],
“environment”: {
“addresses”: [],
“architectures”: [
“x86_64”,
“i686”
],
“certificate”: “-----BEGIN CERTIFICATE-----\nMIIFXDCCA0SgAwIBAgIRALVYUkUYpqOEhZOZYkM1CDIwDQYJKoZIhvcNAQELBQAw\nNzEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEXMBUGA1UEAwwOcm9vdEBs\nb2NhbGhvc3QwHhcNMjAwMTI4MTYxOTM3WhcNMzAwMTI1MTYxOTM3WjA3MRwwGgYD\nVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGxvY2FsaG9z\ndDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALY0K3Bv+XRfv9PpMe6I\n6Tlv8JSBxg1mJopre2/GsLjmn1MfsW8v4XO8OfagOaGxhhWockTszWrn5av5SdRm\n7THmExCW0nj9eoSzlPOc71ZGNBctRMru7mESeGUstKC3uMSStQj3PO9uKrQ2O/uP\nBql7GgD7AMkPXiYRuNkr1rGAO9EXi1dRGEHXeWrUPwW/ArE+Uskx3wKRYJkdRZYw\nBAXiJnQ6aU5v6MV2Ob0CXAhE6W7Co0YexIuGe1pjCVo8arOxY7CAQXLwSJ/nDBHq\niUSUABdmEJvMHoRQS3px5xzMYINHH3jjin7cu0+DMr5MrK8Slr3hy133ThKXMli/\nE8jsB5P6TkHQFGDNEi8RfFPx3G89J3CFI6YTnkdXGCxpgHtY/CPZz7bJoB2cbOw6\n2rFe/mobhLFlP7fMfTtHjzXSSwP62Ky7eN/1Xox59G5i9fuZRrwGRy3TM14moHp/\nOmz7b5UjVyTaCIYfVFdFFPwLC0byjOaNegbsFefLlV7VdpGj7I48q4X4DM6Q4YXk\nCRpcOxkBmt2nInm6ABksq4hRIF2C/mNZjes3vHtOKd0HlMF6ZsaAlgwoNpVNqF4G\nZQKLh+9GA/yp915CfIkxNu0tRYDloTPsHk4O7RFxpbkMfsQwO8zBPeL/iOLajddT\n38jFoD2j+B037ATlBOgFSyv3AgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIFoDATBgNV\nHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCwGA1UdEQQlMCOCCWxvY2Fs\naG9zdIcE1TSCmocQKgF+AAAAAADwPJL//iy6ATANBgkqhkiG9w0BAQsFAAOCAgEA\nQ4kBW7CIBBtZr7m/JD/Zz2WtVKFR4PfDSv+oXefuH2VBzigJ4DnKE2xjLA3IW0mU\nQOthBj2jlX4C3xxPGth7v+9dtOBz1JVE7jPBnzQkvibGG5c9fCfjJHCvNuohljJp\n7AkyQqOQkMjUSMsYuNhKLchcwggHn8G7FFVjjWtJLDJoKNHQoZ0e+R6jSqZ8d78z\n0Q9SCV6OFD9/4FzOSlXEkNe0LHP2HnXk0ckvimVzBQ1b1Q9b2Wu44ONPRV16L8Wq\nRRLigpKtzu9tBl6QNkkwfYxx9+YbVwT/5z156tsWgU10YLNv4FLrA3jhkYq8yp0g\n7W8z/w/L9otlRO2Ux6jlsCX0v0iYYL0hPVyfZiEUsbjWc/VaqdvLOOgOqm86VCI0\nLYQoLg1tADNGSkTypBgx6efLTi4KDBbPGtsoX34yO7lT6iBgPYIyUtqf6Gnr/LGx\nGeTsVI3DujRguso+WUK9nz1+kOoCBFHI/bZ1SNgXqiUvxhk9lhsr8D9o4H2rgUA0\nmyr9K5lXN0iKdKcdpAbfhgCLgbAw5whv8+K273uOyfZOVjXr5L1t8NHPFoQ8c6e1\ngA77Z33f1fSsCiV+kcto8IoFtAmxSWsHFz2wTK7XoaA5WYODrvmSZ+4/2+7H5p7w\nVb3w0DaULxOQpB62uq8nMWMTx/31O47vHy1osq/zfwM=\n-----END CERTIFICATE-----\n”,
“certificate_fingerprint”: “4ad9742858702a359701eaafe93ee2d76046be84c3a15bd899ed1ea579c44225”,
“driver”: “lxc”,
“driver_version”: “3.0.3”,
“kernel”: “Linux”,
“kernel_architecture”: “x86_64”,
“kernel_version”: “5.16.3-x86_64-linode152”,
“server”: “lxd”,
“server_pid”: 18635,
“server_version”: “3.0.3”,
“storage”: “”,
“storage_version”: “”,
“server_clustered”: false,
“server_name”: “localhost”,
“project”: “”
}
}
DBUG[02-13|10:37:52] Sending request to LXD method=GET url=http://unix.socket/1.0/containers/tomcat etag=
DBUG[02-13|10:37:52] Got response struct from LXD
DBUG[02-13|10:37:52]
{
“architecture”: “x86_64”,
“config”: {
“image.architecture”: “amd64”,
“image.description”: “ubuntu 18.04 LTS amd64 (release) (20200107)”,
“image.label”: “release”,
“image.os”: “ubuntu”,
“image.release”: “bionic”,
“image.serial”: “20200107”,
“image.version”: “18.04”,
“volatile.base_image”: “979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185”,
“volatile.eth0.hwaddr”: “00:16:3e:e1:6c:ed”,
“volatile.idmap.base”: “0”,
“volatile.idmap.next”: “[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]”,
“volatile.last_state.idmap”: “[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]”
},
“devices”: {
“eth0”: {
“ipv4.address”: “10.173.181.151”,
“name”: “eth0”,
“nictype”: “bridged”,
“parent”: “lxdbr0”,
“type”: “nic”
},
“root”: {
“path”: “/”,
“pool”: “default”,
“size”: “50GB”,
“type”: “disk”
}
},
“ephemeral”: false,
“profiles”: [
“default”
],
“stateful”: false,
“description”: “”,
“created_at”: “2020-01-28T16:30:49Z”,
“expanded_config”: {
“image.architecture”: “amd64”,
“image.description”: “ubuntu 18.04 LTS amd64 (release) (20200107)”,
“image.label”: “release”,
“image.os”: “ubuntu”,
“image.release”: “bionic”,
“image.serial”: “20200107”,
“image.version”: “18.04”,
“volatile.base_image”: “979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185”,
“volatile.eth0.hwaddr”: “00:16:3e:e1:6c:ed”,
“volatile.idmap.base”: “0”,
“volatile.idmap.next”: “[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]”,
“volatile.last_state.idmap”: “[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]”
},
“expanded_devices”: {
“eth0”: {
“ipv4.address”: “10.173.181.151”,
“name”: “eth0”,
“nictype”: “bridged”,
“parent”: “lxdbr0”,
“type”: “nic”
},
“root”: {
“path”: “/”,
“pool”: “default”,
“size”: “50GB”,
“type”: “disk”
}
},
“name”: “tomcat”,
“status”: “Stopped”,
“status_code”: 102,
“last_used_at”: “2021-09-29T11:46:41.64855725Z”,
“location”: “”
}
DBUG[02-13|10:37:52] Connected to the websocket
DBUG[02-13|10:37:52] Sending request to LXD method=PUT url=http://unix.socket/1.0/containers/tomcat/state etag=
DBUG[02-13|10:37:52]
{
“action”: “start”,
“timeout”: 0,
“force”: false,
“stateful”: false
}
DBUG[02-13|10:37:52] Got operation from LXD
DBUG[02-13|10:37:52]
{
“id”: “f7eceb41-7eb9-4ddd-b4ea-5e0860f48b83”,
“class”: “task”,
“description”: “Starting container”,
“created_at”: “2022-02-13T10:37:52.013266117Z”,
“updated_at”: “2022-02-13T10:37:52.013266117Z”,
“status”: “Running”,
“status_code”: 103,
“resources”: {
“containers”: [
“/1.0/containers/tomcat”
]
},
“metadata”: null,
“may_cancel”: false,
“err”: “”
}
DBUG[02-13|10:37:52] Sending request to LXD method=GET url=http://unix.socket/1.0/operations/f7eceb41-7eb9-4ddd-b4ea-5e0860f48b83 etag=
DBUG[02-13|10:37:52] Got response struct from LXD
DBUG[02-13|10:37:52]
{
“id”: “f7eceb41-7eb9-4ddd-b4ea-5e0860f48b83”,
“class”: “task”,
“description”: “Starting container”,
“created_at”: “2022-02-13T10:37:52.013266117Z”,
“updated_at”: “2022-02-13T10:37:52.013266117Z”,
“status”: “Running”,
“status_code”: 103,
“resources”: {
“containers”: [
“/1.0/containers/tomcat”
]
},
“metadata”: null,
“may_cancel”: false,
“err”: “”
}
Error: Common start logic: Set volatile.last_state.idmap config key on container “tomcat” (id 2): Initialize storage: Could not determine ZFS module version
Try lxc info --show-log tomcat for more info

if i run lxc info --show-log tomcat i get the following response

root@localhost:~# lxc info --show-log tomcat

Name: tomcat

Remote: unix://

Architecture: x86_64

Created: 2020/01/28 16:30 UTC

Status: Stopped

Type: persistent

Profiles: default

Log:

Please help

 “kernel_version”: “5.16.3-x86_64-linode152”,

I suspect you’re running a kernel which does not support ZFS so your ZFS pool can’t be mounted. Switch to a kernel with ZFS support or install zfs-dkms to get ZFS working.

@stgraber thanks a lot let me try what you have suggested

@stgraber, i have tried upgrading the kernel_version 5.7 but still more not working, moved on to version 5.10 its still showing the same error

root@localhost:~# lxc list
±---------±--------±-----±-----±-----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±---------±--------±-----±-----±-----------±----------+
| monitor | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
| postgres | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
| proxy | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
| proxy2 | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
| tomcat | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
| tomcat2 | STOPPED | | | PERSISTENT | 0 |
±---------±--------±-----±-----±-----------±----------+
root@localhost:~# lxc start tomcat
Error: Common start logic: Set volatile.last_state.idmap config key on container “tomcat” (id 2): Initialize storage: Could not determine ZFS module version
Try lxc info --show-log tomcat for more info
root@localhost:~#

is there a specific kernel version that you would recommend

You’ll need to use an Ubuntu kernel that has the ZFS module in it, not a vanilla linux kernel.