I believe this is a common issue you run into as soon as you have a few cameras configured.
The workaround I use is a bit tricky as you need to edit the main “config.json” file located at the instance root volume. To do this you need to mount the root volume in one cmd window and edit the file in a second cmd window. Alternative try the following, assume “frigate” is your instance name and your storage pool is located at default location “/var/lib/incus”:
incus start frigate && vi /var/lib/incus/storage-pools/default/containers/frigate/config.json
It tries to start frigate, mounts the root volume and vim opens the “config.json”. This holds a lock on the mount until you finish editing the file. Search for “/dev/shm” and update the size to your needs:
{
"destination": "/dev/shm",
"type": "tmpfs",
"source": "shm",
"options": [
"nosuid",
"noexec",
"nodev",
"mode=1777",
"size=524288k"
]
},
I upped my to 512MB (as you can see, have enough of RAM) which is plenty properly 256MB or 128MB will do to. After changing the shm just save the file which will unmount the root volume. Now you can start frigate and will start with the configured amount of shared ram.
The change is persistent until you update / rebuild the instance. There might be a better way to do this. May be @stgraber can chime in and explain the proper way 
Enjoy…