What would be the equivalent lxd syntax for these two lxc.mount.entry directives used in the lxc config file ?
lxc.mount.entry = /dev/lxc_luns /var/lib/lxc/ora84c10/rootfs/dev/lxc_luns none defaults,bind,create=dir 0 0
lxc.mount.entry = shm dev/shm tmpfs size=3500m,nosuid,nodev,noexec,create=dir 0 0
Thanks
The first would be:
The second cannot be done from outside the container with LXD. You should instead do it with a /etc/fstab entry inside the container.
Still working on this … wondering if some tweak is needed because this is LXD SNAP.
Not working:
[ubuntu@o83sv2 ~]$ lxc config device add ora84d10 asm disk source=/dev/lxc_luns path=/dev/lxc_luns
Error: Failed to add device “asm”: Missing source path “/dev/lxc_luns” for disk “asm”
[ubuntu@o83sv2 ~]$
Shouldn’t be, the snap environment is supposed to have the same view of /dev as the rest of the system.
What does ls -lh /dev/lxc_luns show you?
[ubuntu@o78sv2 ~]$ sudo fdisk -l /dev/lxc_luns/asm_data_201_00
Disk /dev/lxc_luns/asm_data_201_00: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes[ubuntu@o78sv2 ~]$ ls -lh /dev/lxc_luns/
total 0
brw-rw----. 1 grid asmadmin 252, 9 Aug 1 17:36 asm_data_201_00
brw-rw----. 1 grid asmadmin 252, 10 Aug 1 17:36 asm_mgmt_201_00
brw-rw----. 1 grid asmadmin 252, 11 Aug 1 17:36 asm_sysd_201_00
And then here is the UDEV rule that creates these devices. This UDEV rule was created so that these devices could be introduced into LXC containers without having to deal with the default soft link system that is now the default standard. This naming has worked perfectly with LXC containers, but now trying to get them to work with LXD.
Container-Friendly Multipath Naming (from the /etc/udev/rules.d/99-grid.rules file):
KERNEL==“dm-*”, SUBSYSTEM==“block”, PROGRAM==“/lib/udev/scsi_id --page=0x83 --whitelisted --device=/dev/%k”, RESULT==“2353163346332372d”, RUN+="/bin/sh -c '/bin/mkdir -p /dev/lxc_luns; /bin/mknod /dev/lxc_luns/asm_data_201_00 b $major $minor; chown grid:asmadmin /dev/lxc_luns/asm_data_201_00; chmod 0660 /dev/lxc_luns/asm_data_201_00
So we actually offer three different device naming options on the LXC/D hosts:
“Default”:
[root@o78sv2 rules.d]# ls -lh /dev/mapper/asm*
lrwxrwxrwx. 1 root root 7 Aug 1 17:36 /dev/mapper/asm_data_201_00 → …/dm-9
lrwxrwxrwx. 1 root root 8 Aug 1 17:36 /dev/mapper/asm_mgmt_201_00 → …/dm-10
lrwxrwxrwx. 1 root root 8 Aug 1 17:36 /dev/mapper/asm_sysd_201_00 → …/dm-11
Two “custom” options (custom soft-link, and non-linked devices (“LXC friendly” option)
[root@o78sv2 rules.d]# ls -lh /dev/asm
total 0
lrwxrwxrwx. 1 root root 7 Aug 1 17:36 asm_data_201_00 → …/dm-9
lrwxrwxrwx. 1 root root 8 Aug 1 17:36 asm_mgmt_201_00 → …/dm-10
lrwxrwxrwx. 1 root root 8 Aug 1 17:36 asm_sysd_201_00 → …/dm-11
[root@o78sv2 rules.d]# ls -lh /dev/lxc_luns/
total 0
brw-rw----. 1 grid asmadmin 252, 9 Aug 1 17:36 asm_data_201_00
brw-rw----. 1 grid asmadmin 252, 10 Aug 1 17:36 asm_mgmt_201_00
brw-rw----. 1 grid asmadmin 252, 11 Aug 1 17:36 asm_sysd_201_00
Ok, that’s odd… Can you check what’s in ls -lh /proc/$(cat /var/snap/lxd/common/lxd.pid)/root/dev/ ?
Thanks.
[ubuntu@o78sv2 ~]$ sudo ls -lh /proc/$(cat /var/snap/lxd/common/lxd.pid)/root/dev/
total 0
drwxr-xr-x. 2 root root 100 Aug 2 21:27 asm
crw-r–r–. 1 root root 10, 235 Aug 2 21:27 autofs
drwxr-xr-x. 2 root root 660 Aug 2 21:27 block
drwxr-xr-x. 2 root root 300 Aug 2 21:27 bsg
crw-------. 1 root root 10, 234 Aug 2 21:27 btrfs-control
drwxr-xr-x. 3 root root 60 Aug 2 21:27 bus
lrwxrwxrwx. 1 root root 3 Aug 2 21:27 cdrom → sr0
drwxr-xr-x. 2 root root 5.9K Aug 2 21:27 char
crw-------. 1 root root 5, 1 Aug 2 21:27 console
lrwxrwxrwx. 1 root root 11 Aug 2 21:27 core → /proc/kcore
drwxr-xr-x. 3 root root 60 Aug 2 21:27 cpu
crw-------. 1 root root 10, 62 Aug 2 21:27 cpu_dma_latency
crw-------. 1 root root 10, 203 Aug 2 21:27 cuse
drwxr-xr-x. 8 root root 160 Aug 2 21:27 disk
crw-------. 1 root root 10, 61 Aug 2 21:27 dlm-control
crw-------. 1 root root 10, 60 Aug 2 21:27 dlm-monitor
crw-------. 1 root root 10, 59 Aug 2 21:27 dlm_plock
brw-rw----. 1 root disk 252, 0 Aug 2 21:27 dm-0
brw-rw----. 1 root disk 252, 1 Aug 2 21:27 dm-1
brw-rw----. 1 grid asmadmin 252, 10 Aug 2 21:27 dm-10
brw-rw----. 1 grid asmadmin 252, 11 Aug 2 21:27 dm-11
brw-rw----. 1 root disk 252, 2 Aug 2 21:27 dm-2
brw-rw----. 1 root disk 252, 3 Aug 2 21:27 dm-3
brw-rw----. 1 root disk 252, 4 Aug 2 21:27 dm-4
brw-rw----. 1 root disk 252, 5 Aug 2 21:27 dm-5
brw-rw----. 1 root disk 252, 6 Aug 2 21:27 dm-6
brw-rw----. 1 root disk 252, 7 Aug 2 21:27 dm-7
brw-rw----. 1 root disk 252, 8 Aug 2 21:27 dm-8
brw-rw----. 1 grid asmadmin 252, 9 Aug 2 21:27 dm-9
drwxr-xr-x. 2 root root 80 Aug 2 21:27 dri
crw-rw----. 1 root video 29, 0 Aug 2 21:27 fb0
lrwxrwxrwx. 1 root root 13 Aug 2 21:27 fd → /proc/self/fd
crw-rw-rw-. 1 root root 1, 7 Aug 2 21:27 full
crw-rw-rw-. 1 root root 10, 229 Aug 2 21:28 fuse
crw-------. 1 root root 10, 228 Aug 2 21:27 hpet
drwxr-xr-x. 3 root root 0 Aug 2 21:27 hugepages
crw-------. 1 root root 10, 183 Aug 2 21:27 hwrng
lrwxrwxrwx. 1 root root 25 Aug 2 21:27 initctl → /run/systemd/initctl/fifo
drwxr-xr-x. 3 root root 220 Aug 2 21:27 input
crw-------. 1 root root 239, 0 Aug 2 21:27 iscsi-scst-ctl
drwxr-xr-x. 2 root root 2.6K Aug 2 21:27 isert
crw-------. 1 root root 238, 0 Aug 2 21:27 isert_scst
crw-r–r–. 1 root root 1, 11 Aug 2 21:27 kmsg
srw-rw-rw-. 1 root root 0 Aug 2 21:27 log
brw-rw----. 1 root disk 7, 0 Aug 2 21:27 loop0
brw-rw----. 1 root disk 7, 1 Aug 2 21:27 loop1
brw-rw----. 1 root disk 7, 2 Aug 2 21:27 loop2
brw-rw----. 1 root disk 7, 3 Aug 2 21:27 loop3
crw-rw----. 1 root disk 10, 237 Aug 2 21:27 loop-control
drwxr-xr-x. 2 root root 100 Aug 2 21:27 lxc_luns
drwxr-xr-x. 2 root root 300 Aug 2 21:27 mapper
crw-------. 1 root root 10, 227 Aug 2 21:27 mcelog
crw-r-----. 1 root kmem 1, 1 Aug 2 21:27 mem
drwxrwxrwt. 2 root root 40 Aug 2 21:27 mqueue
drwxr-xr-x. 2 root root 60 Aug 2 21:27 net
crw-rw-rw-. 1 root root 1, 3 Aug 2 21:27 null
crw-------. 1 root root 10, 144 Aug 2 21:27 nvram
drwxr-xr-x. 2 root root 100 Aug 2 21:27 ol_o76sv1
crw-r-----. 1 root kmem 1, 4 Aug 2 21:27 port
crw-------. 1 root root 108, 0 Aug 2 21:27 ppp
crw-rw-rw-. 1 root root 5, 2 Aug 2 21:27 ptmx
drwxr-xr-x. 2 root root 0 Aug 2 21:27 pts
crw-rw-rw-. 1 root root 1, 8 Aug 2 21:27 random
drwxr-xr-x. 2 root root 60 Aug 2 21:27 raw
crw-rw-r–. 1 root root 10, 242 Aug 2 21:27 rfkill
lrwxrwxrwx. 1 root root 4 Aug 2 21:27 rtc → rtc0
crw-------. 1 root root 251, 0 Aug 2 21:27 rtc0
crw-------. 1 root root 240, 0 Aug 2 21:27 scst_event
brw-rw----. 1 root disk 8, 0 Aug 2 21:27 sda
brw-rw----. 1 root disk 8, 1 Aug 2 21:27 sda1
brw-rw----. 1 root disk 8, 2 Aug 2 21:27 sda2
brw-rw----. 1 root disk 8, 16 Aug 2 21:27 sdb
brw-rw----. 1 root disk 8, 32 Aug 2 21:27 sdc
brw-rw----. 1 root disk 8, 48 Aug 2 21:27 sdd
brw-rw----. 1 root disk 8, 64 Aug 2 21:27 sde
brw-rw----. 1 root disk 8, 80 Aug 2 21:27 sdf
brw-rw----. 1 root disk 8, 96 Aug 2 21:27 sdg
brw-rw----. 1 root disk 8, 112 Aug 2 21:27 sdh
brw-rw----. 1 root disk 8, 128 Aug 2 21:27 sdi
brw-rw----. 1 root disk 8, 144 Aug 2 21:27 sdj
brw-rw----. 1 root disk 8, 160 Aug 2 21:27 sdk
brw-rw----. 1 root disk 8, 176 Aug 2 21:27 sdl
crw-rw----. 1 root cdrom 21, 0 Aug 2 21:27 sg0
crw-rw----. 1 root disk 21, 1 Aug 2 21:27 sg1
crw-rw----. 1 root disk 21, 10 Aug 2 21:27 sg10
crw-rw----. 1 root disk 21, 11 Aug 2 21:27 sg11
crw-rw----. 1 root disk 21, 12 Aug 2 21:27 sg12
crw-rw----. 1 root disk 21, 2 Aug 2 21:27 sg2
crw-rw----. 1 root disk 21, 3 Aug 2 21:27 sg3
crw-rw----. 1 root disk 21, 4 Aug 2 21:27 sg4
crw-rw----. 1 root disk 21, 5 Aug 2 21:27 sg5
crw-rw----. 1 root disk 21, 6 Aug 2 21:27 sg6
crw-rw----. 1 root disk 21, 7 Aug 2 21:27 sg7
crw-rw----. 1 root disk 21, 8 Aug 2 21:27 sg8
crw-rw----. 1 root disk 21, 9 Aug 2 21:27 sg9
drwxrwxrwt. 2 root root 40 Aug 2 21:27 shm
crw-------. 1 root root 10, 231 Aug 2 21:27 snapshot
drwxr-xr-x. 3 root root 180 Aug 2 21:27 snd
brw-rw----. 1 root cdrom 11, 0 Aug 2 21:27 sr0
lrwxrwxrwx. 1 root root 15 Aug 2 21:27 stderr → /proc/self/fd/2
lrwxrwxrwx. 1 root root 15 Aug 2 21:27 stdin → /proc/self/fd/0
lrwxrwxrwx. 1 root root 15 Aug 2 21:27 stdout → /proc/self/fd/1
crw-rw-rw-. 1 root tty 5, 0 Aug 2 21:27 tty
crw–w----. 1 root tty 4, 0 Aug 2 21:27 tty0
crw–w----. 1 root tty 4, 1 Aug 2 21:27 tty1
crw–w----. 1 root tty 4, 10 Aug 2 21:27 tty10
crw–w----. 1 root tty 4, 11 Aug 2 21:27 tty11
crw–w----. 1 root tty 4, 12 Aug 2 21:27 tty12
crw–w----. 1 root tty 4, 13 Aug 2 21:27 tty13
crw–w----. 1 root tty 4, 14 Aug 2 21:27 tty14
crw–w----. 1 root tty 4, 15 Aug 2 21:27 tty15
crw–w----. 1 root tty 4, 16 Aug 2 21:27 tty16
crw–w----. 1 root tty 4, 17 Aug 2 21:27 tty17
crw–w----. 1 root tty 4, 18 Aug 2 21:27 tty18
crw–w----. 1 root tty 4, 19 Aug 2 21:27 tty19
crw–w----. 1 root tty 4, 2 Aug 2 21:27 tty2
crw–w----. 1 root tty 4, 20 Aug 2 21:27 tty20
crw–w----. 1 root tty 4, 21 Aug 2 21:27 tty21
crw–w----. 1 root tty 4, 22 Aug 2 21:27 tty22
crw–w----. 1 root tty 4, 23 Aug 2 21:27 tty23
crw–w----. 1 root tty 4, 24 Aug 2 21:27 tty24
crw–w----. 1 root tty 4, 25 Aug 2 21:27 tty25
crw–w----. 1 root tty 4, 26 Aug 2 21:27 tty26
crw–w----. 1 root tty 4, 27 Aug 2 21:27 tty27
crw–w----. 1 root tty 4, 28 Aug 2 21:27 tty28
crw–w----. 1 root tty 4, 29 Aug 2 21:27 tty29
crw–w----. 1 root tty 4, 3 Aug 2 21:27 tty3
crw–w----. 1 root tty 4, 30 Aug 2 21:27 tty30
crw–w----. 1 root tty 4, 31 Aug 2 21:27 tty31
crw–w----. 1 root tty 4, 32 Aug 2 21:27 tty32
crw–w----. 1 root tty 4, 33 Aug 2 21:27 tty33
crw–w----. 1 root tty 4, 34 Aug 2 21:27 tty34
crw–w----. 1 root tty 4, 35 Aug 2 21:27 tty35
crw–w----. 1 root tty 4, 36 Aug 2 21:27 tty36
crw–w----. 1 root tty 4, 37 Aug 2 21:27 tty37
crw–w----. 1 root tty 4, 38 Aug 2 21:27 tty38
crw–w----. 1 root tty 4, 39 Aug 2 21:27 tty39
crw–w----. 1 root tty 4, 4 Aug 2 21:27 tty4
crw–w----. 1 root tty 4, 40 Aug 2 21:27 tty40
crw–w----. 1 root tty 4, 41 Aug 2 21:27 tty41
crw–w----. 1 root tty 4, 42 Aug 2 21:27 tty42
crw–w----. 1 root tty 4, 43 Aug 2 21:27 tty43
crw–w----. 1 root tty 4, 44 Aug 2 21:27 tty44
crw–w----. 1 root tty 4, 45 Aug 2 21:27 tty45
crw–w----. 1 root tty 4, 46 Aug 2 21:27 tty46
crw–w----. 1 root tty 4, 47 Aug 2 21:27 tty47
crw–w----. 1 root tty 4, 48 Aug 2 21:27 tty48
crw–w----. 1 root tty 4, 49 Aug 2 21:27 tty49
crw–w----. 1 root tty 4, 5 Aug 2 21:27 tty5
crw–w----. 1 root tty 4, 50 Aug 2 21:27 tty50
crw–w----. 1 root tty 4, 51 Aug 2 21:27 tty51
crw–w----. 1 root tty 4, 52 Aug 2 21:27 tty52
crw–w----. 1 root tty 4, 53 Aug 2 21:27 tty53
crw–w----. 1 root tty 4, 54 Aug 2 21:27 tty54
crw–w----. 1 root tty 4, 55 Aug 2 21:27 tty55
crw–w----. 1 root tty 4, 56 Aug 2 21:27 tty56
crw–w----. 1 root tty 4, 57 Aug 2 21:27 tty57
crw–w----. 1 root tty 4, 58 Aug 2 21:27 tty58
crw–w----. 1 root tty 4, 59 Aug 2 21:27 tty59
crw–w----. 1 root tty 4, 6 Aug 2 21:27 tty6
crw–w----. 1 root tty 4, 60 Aug 2 21:27 tty60
crw–w----. 1 root tty 4, 61 Aug 2 21:27 tty61
crw–w----. 1 root tty 4, 62 Aug 2 21:27 tty62
crw–w----. 1 root tty 4, 63 Aug 2 21:27 tty63
crw–w----. 1 root tty 4, 7 Aug 2 21:27 tty7
crw–w----. 1 root tty 4, 8 Aug 2 21:27 tty8
crw–w----. 1 root tty 4, 9 Aug 2 21:27 tty9
crw-rw----. 1 root dialout 4, 64 Aug 2 21:27 ttyS0
crw-rw----. 1 root dialout 4, 65 Aug 2 21:27 ttyS1
crw-rw----. 1 root dialout 4, 66 Aug 2 21:27 ttyS2
crw-rw----. 1 root dialout 4, 67 Aug 2 21:27 ttyS3
crw-------. 1 root root 10, 239 Aug 2 21:27 uhid
crw-------. 1 root root 10, 223 Aug 2 21:27 uinput
crw-rw-rw-. 1 root root 1, 9 Aug 2 21:27 urandom
crw-------. 1 root root 244, 0 Aug 2 21:27 usbmon0
crw-------. 1 root root 244, 1 Aug 2 21:27 usbmon1
crw-rw----. 1 root tty 7, 0 Aug 2 21:27 vcs
crw-rw----. 1 root tty 7, 1 Aug 2 21:27 vcs1
crw-rw----. 1 root tty 7, 2 Aug 2 21:27 vcs2
crw-rw----. 1 root tty 7, 3 Aug 2 21:27 vcs3
crw-rw----. 1 root tty 7, 4 Aug 2 21:27 vcs4
crw-rw----. 1 root tty 7, 5 Aug 2 21:27 vcs5
crw-rw----. 1 root tty 7, 6 Aug 2 21:27 vcs6
crw-rw----. 1 root tty 7, 128 Aug 2 21:27 vcsa
crw-rw----. 1 root tty 7, 129 Aug 2 21:27 vcsa1
crw-rw----. 1 root tty 7, 130 Aug 2 21:27 vcsa2
crw-rw----. 1 root tty 7, 131 Aug 2 21:27 vcsa3
crw-rw----. 1 root tty 7, 132 Aug 2 21:27 vcsa4
crw-rw----. 1 root tty 7, 133 Aug 2 21:27 vcsa5
crw-rw----. 1 root tty 7, 134 Aug 2 21:27 vcsa6
crw-rw----. 1 root tty 7, 64 Aug 2 21:27 vcsu
crw-rw----. 1 root tty 7, 65 Aug 2 21:27 vcsu1
crw-rw----. 1 root tty 7, 66 Aug 2 21:27 vcsu2
crw-rw----. 1 root tty 7, 67 Aug 2 21:27 vcsu3
crw-rw----. 1 root tty 7, 68 Aug 2 21:27 vcsu4
crw-rw----. 1 root tty 7, 69 Aug 2 21:27 vcsu5
crw-rw----. 1 root tty 7, 70 Aug 2 21:27 vcsu6
drwxr-xr-x. 2 root root 60 Aug 2 21:27 vfio
crw-------. 1 root root 10, 63 Aug 2 21:27 vga_arbiter
crw-------. 1 root root 10, 137 Aug 2 21:27 vhci
crw-------. 1 root root 10, 238 Aug 2 21:27 vhost-net
crw-------. 1 root root 10, 241 Aug 2 21:27 vhost-vsock
crw-rw-rw-. 1 root root 1, 5 Aug 2 21:27 zero
crw-rw-rw-. 1 root root 10, 249 Aug 2 21:27 zfs
[ubuntu@o78sv2 ~]$
Okay, so it is actually there 
Does it have the right content too?
Stephane, never mind I figured it out and got it to work.
This is an LXD cluster running on Oracle Linux 7. It has two physical nodes in the LXD cluster (o78sv1 and o78sv2). The directory being added resides on o78sv2. The containers I was testing out the command on happen to reside atm on o78sv1. When the command is run on o78sv2 to add the directory to LXD containers also currently residing on o78sv2 everything works fine, of course.
[ubuntu@o78sv2 lxc_luns]$ lxc list
±---------±--------±--------------------±-----±----------±----------±---------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
±---------±--------±--------------------±-----±----------±----------±---------+
| oel79d10 | STOPPED | | | CONTAINER | 0 | o78sv1 |
±---------±--------±--------------------±-----±----------±----------±---------+
| oel79d11 | STOPPED | | | CONTAINER | 0 | o78sv2 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d10 | RUNNING | 10.209.53.18 (eth0) | | CONTAINER | 0 | o78sv1 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d11 | RUNNING | 10.209.53.21 (eth0) | | CONTAINER | 0 | o78sv1 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d12 | RUNNING | 10.209.53.19 (eth0) | | CONTAINER | 0 | o78sv1 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d13 | RUNNING | 10.209.53.20 (eth0) | | CONTAINER | 0 | o78sv1 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d14 | RUNNING | 10.209.53.14 (eth0) | | CONTAINER | 0 | o78sv2 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d15 | RUNNING | 10.209.53.15 (eth0) | | CONTAINER | 0 | o78sv2 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d16 | RUNNING | 10.209.53.16 (eth0) | | CONTAINER | 0 | o78sv2 |
±---------±--------±--------------------±-----±----------±----------±---------+
| ora79d17 | RUNNING | 10.209.53.17 (eth0) | | CONTAINER | 0 | o78sv2 |
±---------±--------±--------------------±-----±----------±----------±---------+
[ubuntu@o78sv2 lxc_luns]$
I was running the command to add the directory to LXD containers running on o78sv1, but the LUNs are on o78sv2. When I try the command on an LXD container running on o78sv2 it works:
[ubuntu@o78sv2 lxc_luns]$ lxc config device add ora79d17 asm disk source=/dev/lxc_luns path=/dev/lxc_luns
Device asm added to ora79d17
[ubuntu@o78sv2 lxc_luns]$ lxc exec ora79d17 bash
[root@ora79d17 ~]# cd /dev/lxc_luns
[root@ora79d17 lxc_luns]# ls -lrt
total 0
brw-rw----. 1 65534 65534 252, 9 Aug 3 02:27 asm_data_201_00
brw-rw----. 1 65534 65534 252, 10 Aug 3 02:27 asm_sysd_201_00
brw-rw----. 1 65534 65534 252, 11 Aug 3 02:27 asm_mgmt_201_00
The next step will be to test out connecting to the LUNs in the usual way over iscsi-initiator from within the LXD containers, rather than presentation of the actual directory.