List leases is out of date

zacksiri · December 29, 2022, 3:11am

I’m having trouble with the lxc network list-leases lxdfan0 command

In my container list I have the following

+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
|        NAME         |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |    LOCATION     |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
| rdio-staging-web-01 | RUNNING | 240.4.0.96 (eth0)  |      | CONTAINER | 0         | upmaru-stage-01 |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
| uplink-stage-01     | RUNNING | 240.4.0.252 (eth0) |      | CONTAINER | 0         | upmaru-stage-01 |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
| uplink-stage-02     | RUNNING | 240.2.0.137 (eth0) |      | CONTAINER | 0         | upmaru-stage-02 |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
| uplink-stage-03     | RUNNING | 240.3.0.11 (eth0)  |      | CONTAINER | 0         | upmaru-stage-03 |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+
| xcaddy-test         | RUNNING | 240.2.0.78 (eth0)  |      | CONTAINER | 0         | upmaru-stage-02 |
+---------------------+---------+--------------------+------+-----------+-----------+-----------------+

however when i run the list-leases i get the following:

+-----------------+-------------------+-------------+---------+-----------------+
|    HOSTNAME     |    MAC ADDRESS    | IP ADDRESS  |  TYPE   |    LOCATION     |
+-----------------+-------------------+-------------+---------+-----------------+
| uplink-stage-01 | 00:16:3e:59:dc:32 | 240.4.0.252 | DYNAMIC | upmaru-stage-01 |
+-----------------+-------------------+-------------+---------+-----------------+
| uplink-stage-02 | 00:16:3e:88:64:e5 | 240.2.0.137 | DYNAMIC | upmaru-stage-02 |
+-----------------+-------------------+-------------+---------+-----------------+
| uplink-stage-03 | 00:16:3e:4e:4f:3d | 240.3.0.11  | DYNAMIC | upmaru-stage-03 |
+-----------------+-------------------+-------------+---------+-----------------+
| xcaddy-test     | 00:16:3e:f7:16:01 | 240.2.0.78  | DYNAMIC | upmaru-stage-02 |
+-----------------+-------------------+-------------+---------+-----------------+

one container is missing

zacksiri · December 29, 2022, 3:13am

An update on this. When I restarted the container, it showed up in the leases

tomp · January 3, 2023, 11:37am

Are you able to reproduce the issue? It maybe that dnsmasq hadn’t flushed the lease to the leases file that LXD reads from.

zacksiri · January 6, 2023, 2:03pm

This happens from time to time. I think I’m using the API a lot, I’ve noticed it happens when many operations are done together. But a restart of the container generally solves the problem.

tomp · January 10, 2023, 3:10pm

Are you able to produce a reproducer script we can use to diagnose the problem?

zacksiri · January 28, 2023, 2:46am

@tomp I have the code on a github repository but it’s written in elixir.

github.com

upmaru/uplink/blob/develop/lib/uplink/packages/instance/bootstrap.ex#L108

      
        
              ],
              "packages" => [
                %{
                  "slug" => package.slug
                }
              ]
            }
            
            
formation_instance = Formation.new_lxd_instance(formation_instance_params)
            
            
LXD.client()
            |> Formation.lxd_create(node_name, instance_params)
            |> Formation.lxd_start(name)
            |> Formation.setup_lxd_instance(formation_instance)
            |> case do
              {:ok, _message} ->
                %{
                  formation_instance: formation_instance_params,
                  install_id: install.id
                }
                |> Instance.Install.new()

In this routine I do the following

Create the instance
Start the instance
Setup the instance

Which executes the following

github.com

upmaru/formation/blob/develop/lib/formation/lxd/instance.ex#L40

      
        
                packages: Enum.map(packages, &Package.new/1)
              }
            end
            
            
def new(params) when is_map(params) do
              params
              |> atomize_keys()
              |> new()
            end
            
            
def setup(%Tesla.Client{} = client, %__MODULE__{} = instance) do
              with {:ok, _add_public_key_result} <-
                     Alpine.add_repository_public_key(client, instance),
                   {:ok, _append_result} <-
                     Alpine.append_repository(client, instance),
                   {:ok, :repository_verified = output} <-
                     Alpine.verify_repository(client, instance) do
                {:ok, output}
              else
                {:error, failed_output} ->
                  {:error, failed_output}

And then it goes on to execute this

github.com

upmaru/formation/blob/develop/lib/formation/lxd/instance.ex#L54

      
        
                     Alpine.append_repository(client, instance),
                   {:ok, :repository_verified = output} <-
                     Alpine.verify_repository(client, instance) do
                {:ok, output}
              else
                {:error, failed_output} ->
                  {:error, failed_output}
              end
            end
            
            
def add_package_and_restart(%Tesla.Client{} = client, %__MODULE__{} = instance) do
              lxd = Lxd.impl()
            
            
  with {:ok, add_package_output} <-
                     Alpine.add_package(client, instance),
                   {:ok, %{body: restart_operation}} <-
                     lxd.restart_instance(client, instance.slug),
                   {:ok, _restart_result} <-
                     lxd.wait_for_operation(client, restart_operation["id"], query: [timeout: Lxd.timeout()]) do
                {:ok, add_package_output}
              else

I’ve noticed from time to time when things go abit too fast sometimes there is no network available. So I wrote it to retry after sometime. Then it works.

tomp · February 2, 2023, 11:09am

Thanks.

I’m not familiar with Elixir I’m afraid.

Do you see it with using lxc commands directly?

zacksiri · February 13, 2023, 1:14pm

If I come across it with lxd commands will let you know. As of now I haven’t run into it yet.