So I’ve started migration from LXD to incus on our terraform modules GitHub - upmaru/terraform-aws-instellar: Terraform module for bootstrapping LXD cluster for using with https://instellar.app. I’m trying to setup 2 machines clustered using the bridge network. I booted up 2 containers (alpine 3.18) and tried to get them to ping each other and I can’t seem to get it to work. I’ve read the documentation about ufw and ran all the commands and nothing seems to be working. When I used to use LXD it came with fan networking that used to work out of the box. Let me know where i’m going wrong here.
I’m setting this up on AWS ubuntu 22.04 VMs
$ sudo nft list ruleset
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 10.199.238.0/24 ip daddr != 10.199.238.0/24 masquerade
ip6 saddr fd42:6115:687d:79b7::/64 ip6 daddr != fd42:6115:687d:79b7::/64 masquerade
}
chain fwd.incusbr0 {
type filter hook forward priority filter; policy accept;
ip6 version 6 oifname "incusbr0" accept
ip6 version 6 iifname "incusbr0" accept
}
chain in.incusbr0 {
type filter hook input priority filter; policy accept;
iifname "incusbr0" tcp dport 53 accept
iifname "incusbr0" udp dport 53 accept
iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
iifname "incusbr0" udp dport 547 accept
}
chain out.incusbr0 {
type filter hook output priority filter; policy accept;
oifname "incusbr0" tcp sport 53 accept
oifname "incusbr0" udp sport 53 accept
oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
oifname "incusbr0" udp sport 547 accept
}
}
table ip filter {
}
table ip6 filter {
}
I’ve also tried configuring resolvectl based on the documentation
resolvectl status incusbr0
Link 3 (incusbr0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.199.238.1
DNS Servers: 10.199.238.1
DNS Domain: \047var\047lib\047incus
config:
ipv4.address: 10.199.238.1/24
ipv4.nat: "true"
ipv6.address: fd42:6115:687d:79b7::1/64
ipv6.nat: "true"
description: ""
name: incusbr0
type: bridge
used_by:
- /1.0/instances/test-01
- /1.0/instances/test-02
- /1.0/profiles/default
managed: true
status: Created
locations:
- ip-172-31-26-149
- incus-exp-02
I have also made sure to enable all traffic between the 2 nodes for the security group in AWS. If the containers are on the same host, they can see / ping each other.