Have a look at the storage.backups_volume setting on the LXD server:
https://linuxcontainers.org/lxd/docs/master/server
You can create a custom storage volume, using a dir pool at a specific directory, or any of the driver types we support, and then specify that custom volume be used for backups.