IncusOS failed to verify PE binaries

IncusOS
1

I’m currently playing around with IncusOS and have installed version 202601100100 on a Futro S740 a with optimal boot security few days ago. Installation went smoothly and I’m really impressed with what IncusOS has to offer.

After IncusOS recently updated to version 202601172317 the boot failed with the early error message IncusOS failed to verify PE binaries: ERROR: failes to verify any PE binary from TPM event log. Selecting the old version from the boot menu still worked as expected.

Today I did a (unrelated) fresh install of version 202601172317 and got the same message on first boot after the installation. I’ve noticed some recent changes to secureboot in the commit logs but don’t really know how to proceed from here. Happy to help with any debugging though - I’m not yet running anything important.

2

Fun, probably another UEFI implementation bug. :slight_smile:

If you could grab version 202601152002 from Index of /os/202601152002/x86_64/ and install it on that machine, it should boot and, more importantly, it will have a debug endpoint that can be used to fetch the TPM event log so we can figure out what’s going wrong. You’ll have to use the flasher tool to configure the install image once you’ve downloaded it (use the --image option so it doesn’t try to grab the latest one automatically).

Once you’ve got version 202601152002 installed, if you could run incus query /os/1.0/debug/secureboot/event-log and share the results that will let us further troubleshoot this.

3

Not sure if I did the flasher procedure correctly. I’ve used flasher-tool -i IncusOS_202601152002.img, selected the Incus application and provided my cert as Incus seed as shown in the documentation you’ve linked. Booting the seeded image from a flash drive, I’ve first hit this issue. After manually wiping the disk, the boot hangs for quite some time at IncusOS is starting… and fails.

I still have the working IncusOS_202601100100.img but as I understand, it does not have the debug entpoint enabled?

Screenshot From 2026-01-19 05-11-53
Screenshot From 2026-01-19 05-11-531722×1136 172 KB

Screenshot From 2026-01-19 05-12-13
Screenshot From 2026-01-19 05-12-131722×1136 526 KB

Screenshot From 2026-01-19 05-13-00
Screenshot From 2026-01-19 05-13-001722×1136 357 KB

4

Sorry about the noise, apparently I was just missing install mode. Booting the installed version 202601152002, the system updated immediately to the latest version and failed to boot again. Here is the output after manually booting 202601152002 afterwards:

{
	"event_log": [
		{
			"Data": "Yd/ki8qT0hGqDQDgmAMrjAoAAAAAAAAAAQAAAAAAAABTAGUAYwB1AHIAZQBCAG8AbwB0AAE=",
			"Digest": "zPxLsyiIo0W8iuraulUrYn2ZNIx2doGrMUH1sB5ApA4=",
			"Index": 7,
			"Type": 2147483649
		},
		{
			"Data": "Yd/ki8qT0hGqDQDgmAMrjAIAAAAAAAAAwAIAAAAAAABQAEsAoVnApeSUp0qHtasVXCvwcsACAAAAAAAApAIAAOB18BIHLT1JgRoAkgpywEwwggKQMIICNqADAgECAhRFvsPK5YIzBo9E6D2AAskyMIPxwzAKBggqhkjOPQQDAjA/MSIwIAYDVQQDDBlJbmN1cyBPUyAtIFNlY3VyZSBCb290IEUxMRkwFwYDVQQKDBBMaW51eCBDb250YWluZXJzMB4XDTI1MDYyNjA4MTA1NFoXDTQ1MDYyMTA4MTA1NFowQjElMCMGA1UEAwwcSW5jdXMgT1MgLSBTZWN1cmUgQm9vdCBQSyBSMTEZMBcGA1UECgwQTGludXggQ29udGFpbmVyczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJv8wjmYfnZwVNpaa8m84A4z2UZcorFnPjoa1JHLUcdUppD2774E6sUQLXQ6RqsEM3MLfe6uZfFcBHY1k359KNW9h6CzCs47K1dAxJ0rJdb2Ug8XscKIpe04u41B/HZqZ3eJnbBBpQY6s2BKpqJKViwgC1rC0CC+gACJ96ry+3tNGHB8lE7Pu9A9DqBql2riElIiGLhlHQGkMrNg8d8uKRMhzFKp44ceA0nhWmuTmCl0NzQFFq9cZMcCsBIxQk2c8I3P+1DSNwct/wgQ5fz0BZ6HvDDdBrilGcwXNFAyllE/un+S2unW+6NgdntdQ8aovMbHF/pKGRrfp6daiqqUq2sCAwEAAaNCMEAwHQYDVR0OBBYEFFQzZh6IxoKMSuDIaEwXE8xukYuDMB8GA1UdIwQYMBaAFPsANmMlgONBcSTZKY8UB3lhbc4zMAoGCCqGSM49BAMCA0gAMEUCIQDcS6w8s+Iuwxzitn0OReX7ONYqfRWCQVISu7afAm/ywAIgZHKuMVvGELwIY5rZQ51Gc1LITg//vXzoFyRgU0u7PdU=",
			"Digest": "N+wKgzjpHpFb+qwEt7fmDlxNKskvLiED+1yugUR5gA0=",
			"Index": 7,
			"Type": 2147483649
		},
		{
			"Data": "Yd/ki8qT0hGqDQDgmAMrjAMAAAAAAAAAwQIAAAAAAABLAEUASwChWcCl5JSnSoe1qxVcK/BywQIAAAAAAAClAgAA4HXwEgctPUmBGgCSCnLATDCCApEwggI3oAMCAQICFDyhnQXU3x6piMaE9lvwX7DF1mGWMAoGCCqGSM49BAMCMD8xIjAgBgNVBAMMGUluY3VzIE9TIC0gU2VjdXJlIEJvb3QgRTExGTAXBgNVBAoMEExpbnV4IENvbnRhaW5lcnMwHhcNMjUwNjI2MDgxMDU1WhcNNDUwNjIxMDgxMDU1WjBDMSYwJAYDVQQDDB1JbmN1cyBPUyAtIFNlY3VyZSBCb290IEtFSyBSMTEZMBcGA1UECgwQTGludXggQ29udGFpbmVyczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALpNIUAKuOYdBi2xgmbcHeENJ/OzonmWeXX4PJq2+tWr/a9bYOMF15vRABLM2lw2UZ8f+z2xVTVuXAcghMOfsiflwFqe44QaMNHdJgxdNh2PFOMk0YYnkhH2I6RoJRo2+dxXuVclaZX6rN0fJPDsag0YRljPbXfzVKPrQiJaCH7NFhzOQKsnnH1tc5uwdo1qk3x8OWXmf4OFaXRpW8qOTozKVRNV9VxrBOpaEgh/mmtyyqVkwxNvLWx2JsA2sImZ4x5WtwkvcyNkFw4IJoAGtPjFP1qmk2biUX6X3hex8M+/bVCBTnf2c2469QgHgbVkPmgp3MRnsj2wRuMXLgPFWJMCAwEAAaNCMEAwHQYDVR0OBBYEFOYhSKveXrUpS/v+RlvEn4SbmL74MB8GA1UdIwQYMBaAFPsANmMlgONBcSTZKY8UB3lhbc4zMAoGCCqGSM49BAMCA0gAMEUCIGXOh5gIDs+9FDXqmL21Wx4nPjgLpcSa8prCeI2cxXOPAiEAoVTK9JoluUY68G9drFv2Cr1Sg1g2lcyR2+WPY8+FDHA=",
			"Digest": "Qd66mTE/GXzwEPapgCUE/X0YhEQguj/3Ag07qxUHQkQ=",
			"Index": 7,
			"Type": 2147483649
		},
		{
			"Data": "y7IZ1zo9lkWjvNrQDmdlbwIAAAAAAAAAhQUAAAAAAABkAGIAoVnApeSUp0qHtasVXCvwcsMCAAAAAAAApwIAAOB18BIHLT1JgRoAkgpywEwwggKTMIICOKADAgECAhQShXbSTBtalLXh/9io8jKBAwn2qzAKBggqhkjOPQQDAjA/MSIwIAYDVQQDDBlJbmN1cyBPUyAtIFNlY3VyZSBCb290IEUxMRkwFwYDVQQKDBBMaW51eCBDb250YWluZXJzMB4XDTI1MDYyNjA4MTA1NVoXDTI3MDYyNjA4MTA1NVowRDEnMCUGA1UEAwweSW5jdXMgT1MgLSBTZWN1cmUgQm9vdCAyMDI1IFIxMRkwFwYDVQQKDBBMaW51eCBDb250YWluZXJzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzeUqrRbFMpgblqqf3Su7AXVwZIci4W6haW5GLnwv0mIADhYnqnShpAAvuv2tStjt4WJ+M6WO9QlCoainIsEojWxJxrONztl01c7FSFudTlvtDXjprTTcA1uaz2jMDiaHU4RXibZwyD9OALDXd/54tFWECVc7o88TMWg3+PWyoYM4j1D5lXXvNx5Ro+2aN9Gbt3U4yhP91u9ZgfTUa2PsiTYnkM44ZYBkbKDTb3n8r+3tEE51LsbmZtpkWlkhEQY6Aen0ewgaXN9kBgnhmfVEEQMLDjUGHVMDQNaDv045yJzEpC4H8IipIMio+MxX054INAf3N78drjh8wS2NCNfczwIDAQABo0IwQDAdBgNVHQ4EFgQUbNyIDF3zGxgXbdqjUoOUqgN5H5EwHwYDVR0jBBgwFoAU+wA2YyWA40FxJNkpjxQHeWFtzjMwCgYIKoZIzj0EAwIDSQAwRgIhANv+tkzSdPnpMVczNdAIKaNmtRWlMLnNsCyhOu1tYq7IAiEA2xFJS4kqapJosRdFKVIVUFXcL43cPLXW0fXgIlg+94ehWcCl5JSnSoe1qxVcK/BywgIAAAAAAACmAgAA4HXwEgctPUmBGgCSCnLATDCCApIwggI4oAMCAQICFHIG73F4OWvvCQB0xUhthn8piim3MAoGCCqGSM49BAMCMD8xIjAgBgNVBAMMGUluY3VzIE9TIC0gU2VjdXJlIEJvb3QgRTExGTAXBgNVBAoMEExpbnV4IENvbnRhaW5lcnMwHhcNMjUxMDIwMTgwNzI0WhcNMjcxMDIwMTgwNzI0WjBEMScwJQYDVQQDDB5JbmN1cyBPUyAtIFNlY3VyZSBCb290IDIwMjYgUjExGTAXBgNVBAoMEExpbnV4IENvbnRhaW5lcnMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCcD5l3IhGYALxuNqtK0lnXsmLuBe7etYqGxckpTfCSEMUaq8TC42cdiBN54/NipLXPSwlVg3Hw1mlAygwaDP/zTQQf6gF19z0dH4z8r8+A9x3GM2tpvr0+3qgiJmIWxjguGX+FRGhwGpS5Sgy8KkYTQzp5GHj4DfKTigFOgeJpWslH4yRA9Jv0QO4PYDBx6eMvLVYfRd6yw9wK/Lr/+SuzNcWtlPXa9G6MEMhPOg52wWu5O1FE83GBWof1cctbmkgFKYS2tnMTv3Uq4+A1zdgbo4+L0zB7VZRE3z4lLEfw5TYeNNN2foapjll5PtlqOkOvGzLfq6WdrB9h22LSasRtAgMBAAGjQjBAMB0GA1UdDgQWBBRl/bRelauow21Y+QYQ9J7uvA3UUTAfBgNVHSMEGDAWgBT7ADZjJYDjQXEk2SmPFAd5YW3OMzAKBggqhkjOPQQDAgNIADBFAiAyXN+yUhWx88jQYicnqnrGRYfC4XDsRKcwJ7U4zLG0qAIhAJYadsNZtIiGCKFtrAg2luRYueQIUrxeB7JTAvdAdXNC",
			"Digest": "Ojwj8bmIb8PeRpNgNwPrDiDtL7c7gTNyABGtUvFAOs4=",
			"Index": 7,
			"Type": 2147483649
		},
		{
			"Data": "y7IZ1zo9lkWjvNrQDmdlbwMAAAAAAAAAAAAAAAAAAABkAGIAeAA=",
			"Digest": "n3W2gjv/avECSk4gNnGc3VSNPLwr8d6OfvTQ7QH5S/k=",
			"Index": 7,
			"Type": 2147483649
		},
		{
			"Data": "AAAAAA==",
			"Digest": "3z9hmASpL9tAVxktxD3XSOp3itxSvEmM6AUkwBS4ERk=",
			"Index": 7,
			"Type": 4
		},
		{
			"Data": "AAAAAA==",
			"Digest": "3z9hmASpL9tAVxktxD3XSOp3itxSvEmM6AUkwBS4ERk=",
			"Index": 4,
			"Type": 4
		},
		{
			"Data": "y7IZ1zo9lkWjvNrQDmdlbwIAAAAAAAAApgIAAAAAAABkAGIA4HXwEgctPUmBGgCSCnLATDCCApIwggI4oAMCAQICFHIG73F4OWvvCQB0xUhthn8piim3MAoGCCqGSM49BAMCMD8xIjAgBgNVBAMMGUluY3VzIE9TIC0gU2VjdXJlIEJvb3QgRTExGTAXBgNVBAoMEExpbnV4IENvbnRhaW5lcnMwHhcNMjUxMDIwMTgwNzI0WhcNMjcxMDIwMTgwNzI0WjBEMScwJQYDVQQDDB5JbmN1cyBPUyAtIFNlY3VyZSBCb290IDIwMjYgUjExGTAXBgNVBAoMEExpbnV4IENvbnRhaW5lcnMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCcD5l3IhGYALxuNqtK0lnXsmLuBe7etYqGxckpTfCSEMUaq8TC42cdiBN54/NipLXPSwlVg3Hw1mlAygwaDP/zTQQf6gF19z0dH4z8r8+A9x3GM2tpvr0+3qgiJmIWxjguGX+FRGhwGpS5Sgy8KkYTQzp5GHj4DfKTigFOgeJpWslH4yRA9Jv0QO4PYDBx6eMvLVYfRd6yw9wK/Lr/+SuzNcWtlPXa9G6MEMhPOg52wWu5O1FE83GBWof1cctbmkgFKYS2tnMTv3Uq4+A1zdgbo4+L0zB7VZRE3z4lLEfw5TYeNNN2foapjll5PtlqOkOvGzLfq6WdrB9h22LSasRtAgMBAAGjQjBAMB0GA1UdDgQWBBRl/bRelauow21Y+QYQ9J7uvA3UUTAfBgNVHSMEGDAWgBT7ADZjJYDjQXEk2SmPFAd5YW3OMzAKBggqhkjOPQQDAgNIADBFAiAyXN+yUhWx88jQYicnqnrGRYfC4XDsRKcwJ7U4zLG0qAIhAJYadsNZtIiGCKFtrAg2luRYueQIUrxeB7JTAvdAdXNC",
			"Digest": "bgHcU984O0mHt+5voMA8ZGrBiLmPh3Kn6vYpIX+Q9lk=",
			"Index": 7,
			"Type": 2147483872
		},
		{
			"Data": "GPBRbgAAAAC45wEAAAAAAAAAMAEBAAAAkAAAAAAAAAACAQwA0EEDCgAAAAABAQYAABIDEgoAAQD//wAABAEqAAEAAAAACAAAAAAAAAAAQAAAAAAAmyd3aF9CR0WKInfjCGf2kgICBARGAFwARQBGAEkAXABTAFkAUwBUAEUATQBEAFwAUwBZAFMAVABFAE0ARAAtAEIATwBPAFQAWAA2ADQALgBFAEYASQAAAH//BAA=",
			"Digest": "R8LUV0+hL+OrZnW1OPbGVp0xsjdIad1dlI1x5r/jJBc=",
			"Index": 4,
			"Type": 2147483651
		},
		{
			"Data": "GMBPZQAAAAC4M7AFAAAAAAAA+U0BAAAAlgAAAAAAAAACAQwA0EEDCgAAAAABAQYAABIDEgoAAQD//wAABAEqAAEAAAAACAAAAAAAAAAAQAAAAAAAmyd3aF9CR0WKInfjCGf2kgICBARMAFwARQBGAEkAXABMAGkAbgB1AHgAXABJAG4AYwB1AHMATwBTAF8AMgAwADIANgAwADEAMQA1ADIAMAAwADIALgBlAGYAaQAAAH//BAA=",
			"Digest": "X4D53Ts8m9wmzcCXGsCS2DhAJPLLcWpHUdClsxp6Pag=",
			"Index": 4,
			"Type": 2147483651
		},
		{
			"Data": "LgBsAGkAbgB1AHgAAAA=",
			"Digest": "DaKT43rVURxZvkeZN2mqy5GyQ/fQECiOEY3JDpWq71o=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBsAGkAbgB1AHgAAAA=",
			"Digest": "b5xnYTn8vKMZk6nWcLXw6B52FM3ehkhj1iOFflj4e7I=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBvAHMAcgBlAGwAAAA=",
			"Digest": "P7nk48yBDUMmtcE87xiu4fnfjF9Pf1uWZlck+juEbgg=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBvAHMAcgBlAGwAAAA=",
			"Digest": "hAGXVgshc8iR0jb3COtp8nN/E9IdeQFSGgoUmH3wvU0=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBjAG0AZABsAGkAbgBlAAAA",
			"Digest": "RhIDqJ8j42w6TcgX+QWwBITSz359k3bxPfkcQdhKvkY=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBjAG0AZABsAGkAbgBlAAAA",
			"Digest": "dMTJZFO2s5NQ3zxuPFFa+yb2vMbboz1DKDCQ4xYFcrQ=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBpAG4AaQB0AHIAZAAAAA==",
			"Digest": "Fe43518ejUIIDpH9u9JWB4CRjIH+NoeubRXEcrvarHU=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBpAG4AaQB0AHIAZAAAAA==",
			"Digest": "XTezGB8fVmYktsEfxn4u1bHXILnXEOTxeJbywCekmMo=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgB1AGMAbwBkAGUAAAA=",
			"Digest": "RUwEagQ0IJklhGobioSiNMQy6n3fhqH17+zP6hLTNO0=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgB1AGMAbwBkAGUAAAA=",
			"Digest": "+XUNv3gq/VtVdQSj6ZZkN/uTVziuY3xfcso6dNHttJ8=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgB1AG4AYQBtAGUAAAA=",
			"Digest": "2nptlByqnSi4o2ZcSGXBQ9uPmUAKyI2IM3CuMCFjbDA=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgB1AG4AYQBtAGUAAAA=",
			"Digest": "w4qfjBuRsSwfpMQ5moFzEApZ3yHUZ1PaQer2aP6bMD4=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBzAGIAYQB0AAAA",
			"Digest": "/1Uv0lW+GKPWHA2oiXb8cVWdE6rRLR3+FwjPlQzEt0w=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBzAGIAYQB0AAAA",
			"Digest": "7nJoRL4mPMA9jkQgm8Qr09SB8yw3YKuh6eGqZbtGhH4=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBwAGMAcgBwAGsAZQB5AAAA",
			"Digest": "krE1H3J5/IhcJONAniP+0/hL3vS7kL62GKzRRXY6KT8=",
			"Index": 11,
			"Type": 13
		},
		{
			"Data": "LgBwAGMAcgBwAGsAZQB5AAAA",
			"Digest": "mjb2KR2ky4UVZ0r58L5qlwicdh20ij45iudKuepufTQ=",
			"Index": 11,
			"Type": 13
		}
	],
	"pcr4": "c5d9ceb7ad16c730ae95b08784de628626df0ee031f3528896a7b968cb6c8e31",
	"pcr7": "0d04afe0738bb9b96e26f22ef84b852ad9d5f7393ba46bd02dd2e4110d4d85c9"
}
5

Thanks – very useful!

It turns out your UEFI implementation isn’t properly recording the EFIAction (a known, handled bug), plus it’s CAPITALIZING the entire PE binary path, so the simple string equality test we’re using is failing. Since the ESP partition is vfat, capitalization doesn’t matter in filesystem actions, and it won’t be too hard to make the equality test ignore case sensitivity which should fix the problem you’re seeing.

6

Should be fixed with More PE validation fixes by gibmat · Pull Request #824 · lxc/incus-os · GitHub. Once that lands in a new IncusOS build you’ll be good to go.

7

Awesome, I wasn’t even expecting much due to the age of the machine. Got a few of them refurbished for 33€ a pop. Quite the steal for x86 with TPM 2.0 “support”. Amazing value – especially with IncusOS on them.

Thanks for lowering the entry barriers to self hosting with secure boot and TPM usage, guys. :+1:

8

FYI, at some point this year we’ll be bumping the instruction set baseline to x86_64-v3, which roughly speaking should be supported by most CPUs manufactured in the past decade (a notable exception are some of Intel’s lower-end Atom CPUs). The documentation does mention x86_64-v3 as a requirement, but we haven’t enabled builds of the kernel and other packages to actually take advantage of this yet. That being said, we do want to support running IncusOS on as many systems as possible, and we’re happy to attempt to fix/work around issues people might encounter that can be addressed in software. :slight_smile:

9

Just updated to 202601220238. The error is gone as expected. Thanks again.