I was able to workaroud this by passing the serial device to the VM via.
config:
raw.qemu.conf: |-
[chardev "serial"]
id = "cma"
backend = "serial"
path = "/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2669785-if00"
[device "serial"]
driver = "usb-serial"
chardev = "cma"