So after days of pain i finally figured it out. I came across this tool https://github.com/ddimick/proxmox-lxc-idmapper
For multiple users and the shared group I needed:
$ cat /etc/subuid
root:1000:1
root:1001:1
root:1002:1
root:1003:1
root:100000:65536
lxd:100000:65536
root:1000:1
root:1001:1
root:1002:1
root:1003:1
root:1004:1
root:100000:65536
lxd:100000:65536
Then in /etc/lxc/default.conf I needed:
lxc.idmap = u 0 100000 1000
lxc.idmap = g 0 100000 1000
lxc.idmap = u 1000 1000 1
lxc.idmap = g 1000 1000 1
lxc.idmap = u 1001 101001 0
lxc.idmap = g 1001 101001 0
lxc.idmap = u 1001 1001 1
lxc.idmap = g 1001 1001 1
lxc.idmap = u 1002 101002 0
lxc.idmap = g 1002 101002 0
lxc.idmap = u 1002 1002 1
lxc.idmap = g 1002 1002 1
lxc.idmap = u 1003 101003 0
lxc.idmap = g 1003 101003 0
lxc.idmap = u 1003 1003 1
lxc.idmap = g 1003 1003 1
lxc.idmap = u 1004 101004 64532
lxc.idmap = g 1004 1004 1
lxc.idmap = g 1005 101005 64532
And finally:
printf "both 1003 1003\ngid 1004 1004\n" | sudo lxc config set c1 raw.idmap -