It looks like you’re probably missing an install.yaml file in your local setup. The presence of that file, even if empty, is what triggers the install process. So before creating your seed.tar you’ll want to touch seed/install.yaml. That will likely get you to the same result as you saw when using the online image customizer.
(Without that install.yaml, IncusOS is trying to run in “live” mode from the install media, which must be at least 50GiB in size; if not, you’ll get the error you report.)
Because there’s more than one possible drive present that you could install IncusOS to, you’ll need to provide a target as part of the install seed. You likely want the nvme one, which you can easily select by providing the following install.yaml in your seed archive: