LXD on Oracle free tier service

Ricardo_Passos · January 16, 2023, 2:47pm

Dear,

I’m using Google Translate, as I’m not good at English. I created an Oracle Cloud account to use the free tier services. I’m running an ubuntu 22.04 instance and I’ve configured the vcn to allow all traffic on port 8443.

I have the lxdware service running on a vps in Contabo. I did a new installation of lxd via snap in the oracle instance, ran the lxd init command, configured the network as a bridge and did all the settings to access oracle’s lxd through lxdware in the account. So far, everything has worked out.

However, in lxdware I request that it download an image for the ubuntu container and, soon after, I create a container. This container does not receive ipv4 but receives ipv6. I had already done all of this at Contabo, with no problems. But in Oracle I’m having this problem.

I read several posts here on the forum about similar problems (containers not receiving ipv4). I configured the firewall in several ways, as well as configured the oracle’s vcn in several ways, but, so far, I still haven’t been able to make a container receive ipv4 in the instance that I’m running in oracle.

stgraber · January 16, 2023, 2:53pm

Oracle Cloud enables pretty strict firewalling by default (ufw).
You’ll most likely need to reconfigure ufw to have things work.

https://linuxcontainers.org/lxd/docs/latest/howto/network_bridge_firewalld/#ufw-add-rules-for-the-bridge

Ricardo_Passos · January 16, 2023, 3:31pm

Hello Stgraber.

Yes, I imagine that Oracle Cloud must have some contribution to this problem. I’m using a new container installation and I had already made these settings in ufw for the lxdbr0 bridge interface (mine is configured like this).

But it didn’t work, the problem persists.

kriszos · January 17, 2023, 10:15am

@Ricardo_Passos please post your container and network configuration, if you are using cloud images they sometimes don’t use dhcp by default unless you specify it in cloud-init configuration

lxc config show <ct-name> --expanded
lxc network show <network-name>

Ricardo_Passos · January 17, 2023, 4:39pm

Thanks, @kriszos

$ lxc config show U1 --expanded
architecture: x86_64
config:
boot.autostart: “true”
image.architecture: amd64
image.description: ubuntu 22.04 LTS amd64 (release) (20230107)
image.label: release
image.os: ubuntu
image.release: jammy
image.serial: “20230107”
image.type: squashfs
image.version: “22.04”
volatile.base_image: ed7509d7e83f29104ff6caa207140619a8b235f66b5997f1ed6c5e462617fb71
volatile.cloud-init.instance-id: 5281aeed-856c-45fd-8983-423f0d54537c
volatile.eth0.host_name: veth623836a1
volatile.eth0.hwaddr: 00:16:3e:64:38:aa
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[]’
volatile.last_state.power: RUNNING
volatile.last_state.ready: “false”
volatile.uuid: 0ffdc9a7-2a2d-41d6-8b88-460e7e5c0c48
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: pool1
type: disk
ephemeral: false
profiles:

default
stateful: false
description: “”

$ lxc network show lxdbr0
config:
ipv4.address: 10.224.230.1/24
ipv4.firewall: “false”
ipv4.nat: “true”
ipv6.address: fd42:459e:bdd8:8c38::1/64
ipv6.firewall: “false”
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

/1.0/instances/U1
/1.0/profiles/default
managed: true
status: Created
locations:
none

Ricardo_Passos · January 17, 2023, 4:41pm

The network lxdbr0 defined in lxd receives ipv4. But the container created (U1) does not receive ipv4

tomp · January 17, 2023, 4:43pm

Please show ip a and ip r on the host.

Ricardo_Passos · January 17, 2023, 4:53pm

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:17:03:69:ba brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 10.0.0.159/24 metric 100 brd 10.0.0.255 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::17ff:fe03:69ba/64 scope link
valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:83:c0:48 brd ff:ff:ff:ff:ff:ff
inet 10.224.230.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:459e:bdd8:8c38::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe83:c048/64 scope link
valid_lft forever preferred_lft forever
21: veth623836a1@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 4e:98:df:89:75:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0

$ ip r
default via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.159 metric 100
10.0.0.0/24 dev ens3 proto kernel scope link src 10.0.0.159 metric 100
10.0.0.1 dev ens3 proto dhcp scope link src 10.0.0.159 metric 100
10.224.230.0/24 dev lxdbr0 proto kernel scope link src 10.224.230.1
169.254.169.254 via 10.0.0.1 dev ens3 proto dhcp src 10.0.0.159 metric 100

Ricardo_Passos · January 20, 2023, 12:31pm

@tomp Can you help me?

tomp · January 20, 2023, 1:01pm

Please show sudo iptables-save and sudo nft list ruleset (if available).

Also please show output of lxc info | grep firewall:

Ricardo_Passos · January 21, 2023, 10:46am

$ sudo iptables-save

Generated by iptables-save v1.8.7 on Sat Jan 21 07:41:10 2023

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [580599:435840972]
:InstanceServices - [0:0]
-A INPUT -p tcp -m tcp --dport 8443 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.4.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.5.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 123 -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -m comment --comment “See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule” -j REJECT --reject-with icmp-port-unreachable
COMMIT

Completed on Sat Jan 21 07:41:10 2023

$ sudo nft list ruleset
table ip filter {
chain INPUT {
type filter hook input priority filter; policy accept;
meta l4proto tcp tcp dport 8443 counter packets 61161 bytes 9095699 accept
ct state related,established counter packets 709488 bytes 849973254 accept
meta l4proto icmp counter packets 1 bytes 80 accept
iifname “lo” counter packets 23284 bytes 2020709 accept
meta l4proto udp udp sport 123 counter packets 0 bytes 0 accept
meta l4proto tcp ct state new tcp dport 22 counter packets 26619 bytes 1567650 accept
counter packets 8566 bytes 2804598 reject with icmp type host-prohibited
}

chain FORWARD {
	type filter hook forward priority filter; policy accept;
	counter packets 0 bytes 0 reject with icmp type host-prohibited
}

chain OUTPUT {
	type filter hook output priority filter; policy accept;
	ip daddr 169.254.0.0/16 counter packets 67870 bytes 5224050 jump InstanceServices
}

chain InstanceServices {
	meta l4proto tcp ip daddr 169.254.0.2 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.2.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.4.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.5.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.0.2 tcp dport 80  counter packets 0 bytes 0 accept
	meta l4proto udp ip daddr 169.254.169.254 udp dport 53  counter packets 18649 bytes 1727793 accept
	meta l4proto tcp ip daddr 169.254.169.254 tcp dport 53  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.0.3 skuid 0 tcp dport 80  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.0.4 tcp dport 80  counter packets 0 bytes 0 accept
	meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 49005 bytes 3477762 accept
	meta l4proto udp ip daddr 169.254.169.254 udp dport 67  counter packets 9 bytes 2763 accept
	meta l4proto udp ip daddr 169.254.169.254 udp dport 69  counter packets 0 bytes 0 accept
	meta l4proto udp ip daddr 169.254.169.254 udp dport 123  counter packets 207 bytes 15732 accept
	meta l4proto tcp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject with tcp reset
	meta l4proto udp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject
}

}
table ip6 filter {
chain INPUT {
type filter hook input priority filter; policy accept;
}

chain FORWARD {
	type filter hook forward priority filter; policy accept;
}

chain OUTPUT {
	type filter hook output priority filter; policy accept;
}

}
table inet lxd {
chain pstrt.lxdbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 10.224.230.0/24 ip daddr != 10.224.230.0/24 masquerade
ip6 saddr fd42:459e:bdd8:8c38::/64 ip6 daddr != fd42:459e:bdd8:8c38::/64 masquerade
}
}

Ricardo_Passos · January 21, 2023, 10:59am

@tomp
You didn’t give me the ‘ready and finished’ solution but you gave me the way!! I released the udp/tcp ports on Iptables for the dhcp service and it worked!!

Ricardo_Passos · March 8, 2023, 2:41pm

Dear,

I found a workable solution and ended up not posting such a solution here on the forum. So I will post it now.

In Oracle’s administration page there are already pre-defined rules for accessing ports (tcp, udp, etc). You can change these rules as per your needs.

However, on the virtual machine itself, I simply disabled the firewall, thereby transferring the security trust to the rules described in the OCI control panel.

To disable the firewall, I used:

sudo iptables -F

sudo netfilter-persistent save

mcc4075 · November 14, 2024, 2:04am

Instead of just disabling the firewall alltogether, consider this instead.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [64483:155340056]
:InstanceServices - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp -m udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp --dport 67:68 -j ACCEPT###
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -i incusbr0 -j ACCEPT###
-A FORWARD -o incusbr0 -j ACCEPT###
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -j ACCEPT
-A InstanceServices -d 169.254.4.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -j ACCEPT
-A InstanceServices -d 169.254.5.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 123 -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -j REJECT --reject-with icmp-port-unreachable
COMMIT

Notice the 3 rules annotated with ###. Those allow DHCP traffic so the container will get an IP address, and the 2 below it allow forward traffic in and out of the bridge. You may have to change the interface name if yours are different.