No available dqlite leader server found

sarlacpit · August 23, 2019, 11:26am

Hi Guys,

I installed lxd with snap this morning.
I have two LXD clustered servers [lxd01 & lxd02] and a ceph 3 node storage. [ clustered on lxd01, lxd02 (the ceph mds is on this node) & ceph1]
I have a test container on lxd01 called test.

I had to reboot lxd02
When lxd02 returned, I can no longer do anything on the cluster…

lxc list
Error: failed to begin transaction: failed to create dqlite connection: no available dqlite leader server found

I don’t suspect my Ceph cluster ( a new technology for me)

  cluster:
    id:     a0f57ea2-7318-43de-aed4-bde27b075637
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum lxd01,lxd02,ceph1
    mgr: ceph1(active), standbys: lxd01
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 32 pgs
    objects: 325  objects, 1.2 GiB
    usage:   6.6 GiB used, 2.0 TiB / 2.0 TiB avail
    pgs:     32 active+clean

I have obviously made an error and suspect rebooting lxd02 was a mistake. My question is why? and what do i do to return lxd to its former glory?

Many thanks

freeekanayaka · August 23, 2019, 11:38am

First of all please consider adding a third node two your cluster. We don’t recommend a two node cluster since it is worse than having no cluster at all: if you lose one of the two nodes, also the other won’t function. See also https://github.com/lxc/lxd/blob/master/doc/clustering.md.

Please attach the log of both LXD nodes, you can find them in /var/snap/lxd/common/lxd/logs/lxd.log.

sarlacpit · August 23, 2019, 11:47am

Thanks for the quick response and the advice… I’ll consider it a must.
Looking at the logs I think I may have lost my DB.

sarlacpit · August 27, 2019, 1:58pm

As this was a test lab, I re-initialised my config and started from scratch. No real answers found but lessons learnt. I am now running a three node cluster.

As such I’ll close.

Thanks

sarlacpit · August 28, 2019, 1:00pm

Actually, further to this I have managed to recreate this on a three node cluster.
I have the same servers clustered (now including ceph1).

lxc cluster list
+-------+----------------------------+----------+--------+-------------------+
| NAME  |            URL             | DATABASE | STATE  |      MESSAGE      |
+-------+----------------------------+----------+--------+-------------------+
| ceph1 | https://100.65.20.231:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+
| lxd01 | https://100.65.20.225:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+
| lxd02 | https://100.65.20.230:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+

If i stop the LXD daemon on lxd02 the cluster completely fails.
lxd01 & ceph1 get the same message when trying to issue commands:

Error: failed to begin transaction: failed to create dqlite connection: no available dqlite leader server found

To recover I had to restart the daemon on lxd02.

This allowed me temporarily to get some info about the cluster from ceph1

user@ceph1:~$ lxc cluster list
+-------+----------------------------+----------+---------+----------------------------------+
| NAME  |            URL             | DATABASE |  STATE  |             MESSAGE              |
+-------+----------------------------+----------+---------+----------------------------------+
| ceph1 | https://100.65.20.231:8443 | YES      | OFFLINE | no heartbeat since 42.858418628s |
+-------+----------------------------+----------+---------+----------------------------------+
| lxd01 | https://100.65.20.225:8443 | YES      | OFFLINE | no heartbeat since 42.858860457s |
+-------+----------------------------+----------+---------+----------------------------------+
| lxd02 | https://100.65.20.230:8443 | YES      | OFFLINE | no heartbeat since 42.858597847s |
+-------+----------------------------+----------+---------+----------------------------------+
user@ceph1:~$ lxc cluster list
+-------+----------------------------+----------+--------+-------------------+
| NAME  |            URL             | DATABASE | STATE  |      MESSAGE      |
+-------+----------------------------+----------+--------+-------------------+
| ceph1 | https://100.65.20.231:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+
| lxd01 | https://100.65.20.225:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+
| lxd02 | https://100.65.20.230:8443 | YES      | ONLINE | fully operational |
+-------+----------------------------+----------+--------+-------------------+
user@ceph1:~$ lxc cluster list
Error: failed to begin transaction: failed to create dqlite connection: no available dqlite leader server found

Lxd01 now gives the following error:

lxc cluster list
Error: cannot fetch node config from database: driver: bad connection

Lxd02 now gives the following error:

t=2019-08-28T12:56:14+0000 lvl=warn msg="Failed to get current cluster nodes: Failed to fetch nodes: driver: bad connection"

In order to bring the cluster back I then had to restart the LXD service on Ceph1. Is there some leadership issue? I’m not sure this is how it should behave.

stgraber · August 28, 2019, 6:12pm

@freeekanayaka

freeekanayaka · August 29, 2019, 7:28am

Are all these nodes on the same local network? Is network connectivity working reliably?

sarlacpit · August 29, 2019, 7:32am

Yes, they are on the same network and connected to the same switch. I am not seeing any packet loss between them.

--- 100.65.20.230 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 72ms
rtt min/avg/max/mdev = 0.574/0.687/1.065/0.066 ms, ipg/ewma 0.729/0.669 ms


--- 100.65.20.225 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 64ms
rtt min/avg/max/mdev = 0.523/0.617/1.228/0.089 ms, ipg/ewma 0.656/0.636 ms

freeekanayaka · August 29, 2019, 7:49am

Please can you turn on debug logging on all three nodes with:

snap set lxd daemon.debug=true
systemctl reload snap.lxd.daemon
systemctl restart snap.lxd.daemon

then run lxc command until you see the failure and finally attach here the three /var/snap/lxd/common/lxd/logs/lxd.log files? Hopefully they’ll contain some more info.

sarlacpit · August 29, 2019, 8:30am

Order of actions.
Applied debug on all three nodes
Reloaded & restarted lxd on lxd01, ceph1 then lxd02.
lxd02 hung for a while and I manage to do an lxc list on ceph1
I tried rerunning the command a few seconds later when I got the error.

Have hopefully pasted the relevant debug below…

lxd01 logs:

t=2019-08-29T08:12:37+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=8" 
t=2019-08-29T08:12:37+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.230:8443 attempt=8" 
t=2019-08-29T08:12:37+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.231:8443 attempt=8" 
t=2019-08-29T08:12:37+0000 lvl=dbug msg="Dqlite: connection failed err=no available dqlite leader server found attempt=8" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:100.65.20.225:8443} {ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443}]" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Partial node list heartbeat received, skipping full update" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=9" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Replace current raft nodes with [{ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443} {ID:1 Address:100.65.20.225:8443}]" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.230:8443 attempt=9" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Dqlite: connected address=100.65.20.231:8443 attempt=9" 
t=2019-08-29T08:12:38+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:12:39+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:12:39+0000 lvl=dbug msg="Listening for events on node 100.65.20.230:8443" 
t=2019-08-29T08:12:39+0000 lvl=dbug msg="Found cert" name=0

lxd02 logs:

t=2019-08-29T08:13:16+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:13:16+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:13:17+0000 lvl=dbug msg="Success for task operation: b910a266-8b46-49d5-a29c-b8b7fdec05cb" 
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:100.65.20.225:8443} {ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443}]" 
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Cluster node is up-to-date" 
t=2019-08-29T08:13:19+0000 lvl=warn msg="Excluding offline node from refresh: {ID:2 Address:100.65.20.230:8443 RaftID:2 Raft:true LastHeartbeat:2019-08-29 08:12:38.900060709 +0000 UTC Online:false updated:false}" 
t=2019-08-29T08:13:20+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: Failed to fetch nodes: driver: bad connection" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (Failed to fetch nodes: driver: bad connection)"

ceph1 logs:

t=2019-08-29T08:12:24+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:24+0000 lvl=dbug msg="Replace current raft nodes with [{ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443} {ID:1 Address:100.65.20.225:8443}]" 
t=2019-08-29T08:12:24+0000 lvl=dbug msg="Cluster node is up-to-date" 
t=2019-08-29T08:12:30+0000 lvl=warn msg="Error during dqlite proxy copy: tls: use of closed connection" 
t=2019-08-29T08:12:30+0000 lvl=dbug msg="Database error: failed to begin transaction: failed to receive response: failed to receive header: failed to receive header: EOF" 
t=2019-08-29T08:12:30+0000 lvl=warn msg="Failed to get current cluster nodes: failed to begin transaction: failed to receive response: failed to receive header: failed to receive header: EOF" 
t=2019-08-29T08:12:30+0000 lvl=dbug msg="Disconnected event listener: c2f33fff-ca86-4c4b-9452-d26056ddecd0" 
t=2019-08-29T08:12:30+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:30+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:30+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:12:31+0000 lvl=dbug msg="Dqlite: closing client" 
t=2019-08-29T08:12:31+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=0" 
t=2019-08-29T08:12:31+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.230:8443 attempt=0" 
t=2019-08-29T08:12:31+0000 lvl=dbug msg="Found cert" name=0

Does this help?

freeekanayaka · August 29, 2019, 9:47am

Please post the full log of the last run (the one you started after enabling debugging), for each of the three servers.

sarlacpit · August 29, 2019, 10:42am

Sorry, I hope this captures what you are looking for…
Many thanks.

lxd01

t=2019-08-29T08:11:36+0000 lvl=info msg="LXD 3.16 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-08-29T08:11:36+0000 lvl=info msg="Kernel uid/gid map:" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - u 0 0 4294967295" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - g 0 0 4294967295" 
t=2019-08-29T08:11:36+0000 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - u 0 1000000 1000000000" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - g 0 1000000 1000000000" 
t=2019-08-29T08:11:36+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-08-29T08:11:36+0000 lvl=info msg="Kernel features:" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - uevent injection: no" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - seccomp listener: no" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2019-08-29T08:11:36+0000 lvl=info msg=" - shiftfs support: no" 
t=2019-08-29T08:11:36+0000 lvl=info msg="Initializing local database" 
t=2019-08-29T08:11:36+0000 lvl=dbug msg="Initializing database gateway" 
t=2019-08-29T08:11:36+0000 lvl=dbug msg="Start database node" address=100.65.20.225:8443 id=1
t=2019-08-29T08:11:37+0000 lvl=info msg="Starting /dev/lxd handler:" 
t=2019-08-29T08:11:37+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-08-29T08:11:37+0000 lvl=info msg="REST API daemon:" 
t=2019-08-29T08:11:37+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-08-29T08:11:37+0000 lvl=info msg=" - binding TCP socket" socket=100.65.20.225:8443
t=2019-08-29T08:11:37+0000 lvl=info msg="Initializing global database" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=0" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Dqlite: connected address=100.65.20.230:8443 attempt=0" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Notify node 100.65.20.230:8443 of state changes" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Notify node 100.65.20.231:8443 of state changes" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Initializing storage pools" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Initializing and checking storage pool \"remote\"" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Checking CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Checked CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Initializing networks" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:37+0000 lvl=dbug msg="New task operation: 6e3eeade-1196-4328-b0fd-636ccb272340" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Pruning leftover image files" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Started task operation: 6e3eeade-1196-4328-b0fd-636ccb272340" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Done pruning leftover image files" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Loading daemon configuration" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Initialized inotify with file descriptor 28" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Success for task operation: 6e3eeade-1196-4328-b0fd-636ccb272340" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Skipping image synchronization task since we're not leader" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:37+0000 lvl=dbug msg=Handling ip=100.65.20.231:59958 method=GET url="/1.0/events?project=%2A" user=
t=2019-08-29T08:11:37+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="New event listener: 6c053b8c-1189-4d00-b28a-7e955f3f0044" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="New task operation: c97692b7-d801-4c98-9298-1f2f51232795" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.225:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd01\",\n\t\t\t\t\"server_pid\": 911039,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:11:37+0000 lvl=dbug msg="\n\t{\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"error_code\": 503,\n\t\t\"type\": \"error\"\n\t}" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Pruning expired images" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Started task operation: c97692b7-d801-4c98-9298-1f2f51232795" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Done pruning expired images" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="New task operation: d36d2e4f-51f7-4b18-9c86-95820ef8bb72" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Success for task operation: c97692b7-d801-4c98-9298-1f2f51232795" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Pruning expired container backups" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Started task operation: d36d2e4f-51f7-4b18-9c86-95820ef8bb72" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Done pruning expired container backups" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Success for task operation: d36d2e4f-51f7-4b18-9c86-95820ef8bb72" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Listening for events on node 100.65.20.230:8443" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Scheduler: network: mac05af7f37 has been added: updating network priorities" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="New task operation: b931c9a6-7bf5-4f56-92d6-c9f01093a2b5" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Updating images" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Started task operation: b931c9a6-7bf5-4f56-92d6-c9f01093a2b5" 
t=2019-08-29T08:11:37+0000 lvl=info msg="Done updating images" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:11:37+0000 lvl=dbug msg="Listening for events on node 100.65.20.231:8443" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="New task operation: 15af8455-d357-4c73-9eb4-e4fc820f28ca" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Expiring log files" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Started task operation: 15af8455-d357-4c73-9eb4-e4fc820f28ca" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="New task operation: b17020d2-cd88-4706-b97f-d48caf40a05e" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Done expiring log files" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Success for task operation: 15af8455-d357-4c73-9eb4-e4fc820f28ca" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Updating instance types" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Started task operation: b17020d2-cd88-4706-b97f-d48caf40a05e" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Done updating instance types" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Processing image" alias=default fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Using SimpleStreams cache entry" expiry=2019-08-29T09:11:23+0000 server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Connecting to a remote simplestreams server" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Image already exists in the db" image=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Image already exists on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Already up to date" fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Success for task operation: b931c9a6-7bf5-4f56-92d6-c9f01093a2b5" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=100.65.20.230:56066 method=GET url="/1.0/events?project=%2A" user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="New event listener: c881d562-95cc-41cc-867a-b30054690e05" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Starting container" action=start created=2019-08-23T13:35:25+0000 ephemeral=false name=jump1 project=default stateful=false used=2019-08-28T12:31:20+0000
t=2019-08-29T08:11:38+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.225:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd01\",\n\t\t\t\t\"server_pid\": 911039,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="\n\t{\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"error_code\": 503,\n\t\t\"type\": \"error\"\n\t}" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/containers/1/onstart user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"jump1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Scheduler: container jump1 started: re-balancing" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Success for task operation: b17020d2-cd88-4706-b97f-d48caf40a05e" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Started container" action=start created=2019-08-23T13:35:25+0000 ephemeral=false name=jump1 project=default stateful=false used=2019-08-28T12:31:20+0000
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Scheduler: network: mac43e70585 has been added: updating network priorities" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.225:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd01\",\n\t\t\t\t\"server_pid\": 911039,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="\n\t{\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"error_code\": 503,\n\t\t\"type\": \"error\"\n\t}" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=info msg="Starting container" action=start created=2019-08-28T07:24:11+0000 ephemeral=false name=rad1 project=default stateful=false used=2019-08-28T12:31:30+0000
t=2019-08-29T08:11:38+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/containers/5/onstart user=
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounting RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Mounted RBD storage volume for container \"rad1\" on storage pool \"remote\"" 
t=2019-08-29T08:11:38+0000 lvl=dbug msg="Scheduler: container rad1 started: re-balancing" 
t=2019-08-29T08:11:39+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:11:39+0000 lvl=info msg="Started container" action=start created=2019-08-28T07:24:11+0000 ephemeral=false name=rad1 project=default stateful=false used=2019-08-28T12:31:30+0000
t=2019-08-29T08:11:39+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:11:39+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.225:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.225:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd01\",\n\t\t\t\t\"server_pid\": 911039,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:11:39+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:11:39+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:11:40+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:40+0000 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:100.65.20.225:8443} {ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443}]" 
t=2019-08-29T08:11:40+0000 lvl=dbug msg="Cluster node is up-to-date" 
t=2019-08-29T08:11:51+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:11:51+0000 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:100.65.20.225:8443} {ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443}]"

ceph1

t=2019-08-29T08:42:23+0000 lvl=info msg="LXD 3.16 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-08-29T08:42:23+0000 lvl=info msg="Kernel uid/gid map:" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - u 0 0 4294967295" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - g 0 0 4294967295" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - u 0 1000000 1000000000" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - g 0 1000000 1000000000" 
t=2019-08-29T08:42:23+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-08-29T08:42:23+0000 lvl=info msg="Kernel features:" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - uevent injection: no" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - seccomp listener: no" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - shiftfs support: no" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Initializing local database" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Initializing database gateway" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Start database node" address=100.65.20.231:8443 id=3
t=2019-08-29T08:42:23+0000 lvl=info msg="Starting /dev/lxd handler:" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-08-29T08:42:23+0000 lvl=info msg="REST API daemon:" 
t=2019-08-29T08:42:23+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-08-29T08:42:23+0000 lvl=info msg=" - binding TCP socket" socket=100.65.20.231:8443
t=2019-08-29T08:42:23+0000 lvl=info msg="Initializing global database" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=0" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Dqlite: connected address=100.65.20.230:8443 attempt=0" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"peer node 100.65.20.225:8443 is down\"}" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Could not notify all nodes of database upgrade: peer node 100.65.20.225:8443 is down" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Initializing storage pools" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Initializing and checking storage pool \"remote\"" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Checking CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Checked CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Initializing networks" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: 9488e34c-2449-4830-b997-f47973fd80b1" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Pruning leftover image files" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: 9488e34c-2449-4830-b997-f47973fd80b1" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done pruning leftover image files" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Loading daemon configuration" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Initialized inotify with file descriptor 21" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Success for task operation: 9488e34c-2449-4830-b997-f47973fd80b1" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: d27ad533-abc0-4b77-802b-258829aad327" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Pruning expired images" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: d27ad533-abc0-4b77-802b-258829aad327" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.231:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.231:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.231:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"ceph1\",\n\t\t\t\t\"server_pid\": 45862,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done pruning expired images" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:42:23+0000 lvl=dbug msg="\n\t{\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"error_code\": 503,\n\t\t\"type\": \"error\"\n\t}" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: 756361e5-4008-488a-9c22-0ede9b79fe67" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Success for task operation: d27ad533-abc0-4b77-802b-258829aad327" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Pruning expired container backups" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: 756361e5-4008-488a-9c22-0ede9b79fe67" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done pruning expired container backups" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Success for task operation: 756361e5-4008-488a-9c22-0ede9b79fe67" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: 9e696cee-b040-4950-8978-318071e9fd6e" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: 46781a87-b999-42d7-bd08-3c58c6fc57ac" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Expiring log files" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: 9e696cee-b040-4950-8978-318071e9fd6e" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Updating images" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: 46781a87-b999-42d7-bd08-3c58c6fc57ac" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="New task operation: fed618e6-c367-42ed-b15f-25e7b5b746b5" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done expiring log files" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done updating images" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Success for task operation: 9e696cee-b040-4950-8978-318071e9fd6e" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Updating instance types" 
t=2019-08-29T08:42:23+0000 lvl=dbug msg="Started task operation: fed618e6-c367-42ed-b15f-25e7b5b746b5" 
t=2019-08-29T08:42:23+0000 lvl=info msg="Done updating instance types" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Processing image" alias=default fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Using SimpleStreams cache entry" expiry=2019-08-29T09:11:58+0000 server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Connecting to a remote simplestreams server" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Image already exists in the db" image=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Image already exists on storage pool \"remote\"" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Already up to date" fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Success for task operation: 46781a87-b999-42d7-bd08-3c58c6fc57ac" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:42:24+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.231:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.231:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.231:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"ceph1\",\n\t\t\t\t\"server_pid\": 45862,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:42:24+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Skipping image synchronization task since we're not leader" 
t=2019-08-29T08:42:24+0000 lvl=dbug msg="Success for task operation: fed618e6-c367-42ed-b15f-25e7b5b746b5" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Replace current raft nodes with [{ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443} {ID:1 Address:100.65.20.225:8443}]" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Cluster node is up-to-date" 
t=2019-08-29T08:42:33+0000 lvl=warn msg="Excluding offline node from refresh: {ID:3 Address:100.65.20.231:8443 RaftID:3 Raft:true LastHeartbeat:2019-08-29 08:13:29.01641155 +0000 UTC Online:false updated:false}" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:42:33+0000 lvl=dbug msg=Handling ip=100.65.20.230:44580 method=GET url="/1.0/events?project=%2A" user=
t=2019-08-29T08:42:33+0000 lvl=dbug msg="New event listener: cb619fa1-f6be-489d-9e4d-097045572c11" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:42:33+0000 lvl=dbug msg=Handling ip=100.65.20.225:42342 method=GET url="/1.0/events?project=%2A" user=
t=2019-08-29T08:42:33+0000 lvl=dbug msg="New event listener: 87f96349-8cd2-444e-80ef-d2d7b5a401f7" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Listening for events on node 100.65.20.225:8443" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:42:33+0000 lvl=dbug msg="Listening for events on node 100.65.20.230:8443"

lxd02

t=2019-08-29T08:13:14+0000 lvl=info msg="LXD 3.16 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-08-29T08:13:14+0000 lvl=info msg="Kernel uid/gid map:" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - u 0 0 4294967295" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - g 0 0 4294967295" 
t=2019-08-29T08:13:14+0000 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - u 0 1000000 1000000000" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - g 0 1000000 1000000000" 
t=2019-08-29T08:13:14+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-08-29T08:13:14+0000 lvl=info msg="Kernel features:" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - uevent injection: no" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - seccomp listener: no" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2019-08-29T08:13:14+0000 lvl=info msg=" - shiftfs support: no" 
t=2019-08-29T08:13:14+0000 lvl=info msg="Initializing local database" 
t=2019-08-29T08:13:14+0000 lvl=dbug msg="Initializing database gateway" 
t=2019-08-29T08:13:14+0000 lvl=dbug msg="Start database node" address=100.65.20.230:8443 id=2
t=2019-08-29T08:13:15+0000 lvl=info msg="Starting /dev/lxd handler:" 
t=2019-08-29T08:13:15+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-08-29T08:13:15+0000 lvl=info msg="REST API daemon:" 
t=2019-08-29T08:13:15+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-08-29T08:13:15+0000 lvl=info msg=" - binding TCP socket" socket=100.65.20.230:8443
t=2019-08-29T08:13:15+0000 lvl=info msg="Initializing global database" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.225:8443 attempt=0" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Dqlite: server connection failed err=failed to establish network connection: 503 Service Unavailable address=100.65.20.230:8443 attempt=0" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Dqlite: connected address=100.65.20.231:8443 attempt=0" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Database error: failed to update node version info: database is locked" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Retry failed db interaction (failed to update node version info: database is locked)" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Database error: failed to update node version info: database is locked" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Retry failed db interaction (failed to update node version info: database is locked)" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Notify node 100.65.20.225:8443 of state changes" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Notify node 100.65.20.231:8443 of state changes" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:13:15+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Initializing storage pools" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Initializing and checking storage pool \"remote\"" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Checking CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Checked CEPH storage pool \"remote\" (noop)" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Initializing networks" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: 63ab646a-c983-4a83-878d-550de2033c07" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Pruning leftover image files" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: 63ab646a-c983-4a83-878d-550de2033c07" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done pruning leftover image files" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Loading daemon configuration" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Initialized inotify with file descriptor 34" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Success for task operation: 63ab646a-c983-4a83-878d-550de2033c07" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Skipping image synchronization task since we're not leader" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: 8c78d868-1769-4a65-b3cc-a612f0bb25b5" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.230:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.230:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.230:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd02\",\n\t\t\t\t\"server_pid\": 12274,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Pruning expired images" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: 8c78d868-1769-4a65-b3cc-a612f0bb25b5" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:13:16+0000 lvl=dbug msg="\n\t{\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"error_code\": 503,\n\t\t\"type\": \"error\"\n\t}" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done pruning expired images" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Listening for events on node 100.65.20.225:8443" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Connecting to a remote LXD over HTTPs" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: 2c34abc0-99f8-4a1f-87dc-fc786982cdf1" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Success for task operation: 8c78d868-1769-4a65-b3cc-a612f0bb25b5" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Pruning expired container backups" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: 2c34abc0-99f8-4a1f-87dc-fc786982cdf1" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done pruning expired container backups" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Success for task operation: 2c34abc0-99f8-4a1f-87dc-fc786982cdf1" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: af3fb43f-bf28-4775-a658-26ebbb0880c3" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Expiring log files" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: af3fb43f-bf28-4775-a658-26ebbb0880c3" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Connected to the websocket" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Listening for events on node 100.65.20.231:8443" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: b910a266-8b46-49d5-a29c-b8b7fdec05cb" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done expiring log files" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Updating instance types" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: b910a266-8b46-49d5-a29c-b8b7fdec05cb" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Success for task operation: af3fb43f-bf28-4775-a658-26ebbb0880c3" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done updating instance types" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="New task operation: ed5ec27f-6ce8-42c2-9e66-2deb38b571c0" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Updating images" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Started task operation: ed5ec27f-6ce8-42c2-9e66-2deb38b571c0" 
t=2019-08-29T08:13:16+0000 lvl=info msg="Done updating images" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Processing image" alias=default fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Using SimpleStreams cache entry" expiry=2019-08-29T09:12:40+0000 server=https://cloud-images.ubuntu.com/releases
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Connecting to a remote simplestreams server" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Image already exists in the db" image=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Image already exists on storage pool \"remote\"" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Already up to date" fp=2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac
t=2019-08-29T08:13:16+0000 lvl=dbug msg="Success for task operation: ed5ec27f-6ce8-42c2-9e66-2deb38b571c0" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2019-08-29T08:13:16+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"cluster.https_address\": \"100.65.20.230:8443\",\n\t\t\t\t\"core.https_address\": \"100.65.20.230:8443\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"100.65.20.230:8443\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB7zCCAXWgAwIBAgIRAIytXS/bm9UJVUSczv4sL9IwCgYIKoZIzj0EAwMwMzEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBseGQw\\nMTAeFw0xOTA4MjMxMzAzMTNaFw0yOTA4MjAxMzAzMTNaMDMxHDAaBgNVBAoTE2xp\\nbnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAbHhkMDEwdjAQBgcqhkjO\\nPQIBBgUrgQQAIgNiAASfpo7gEaW7TQxFaCj2l4eOVI8+oiYp1eJs1jbj8hl4+lE1\\n4hagrS9m+6V0mIuM+oGL+Auo7dWVb16EOjh2gaADv1bO94HWG+A46b9xo34dEofZ\\nbyqXp88cAItD6ozJqBejTTBLMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr\\nBgEFBQcDATAMBgNVHRMBAf8EAjAAMBYGA1UdEQQPMA2CBWx4ZDAxhwRkQRThMAoG\\nCCqGSM49BAMDA2gAMGUCMQCiKHv9A7+i4DXcjrH+sNPOAqkRPJLBZeMrOzOAlNVL\\n+GCgOuQaFThnwYuuiqdKZWwCMDk6TkQ5VLdIV/OZCO4Hlb2IsjJuZPcCTX5f5eLN\\nL3VlybTZmy/zE+AuEluvnkV0Fg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6d81ad92cb9c69227fc64ed46f1d197be4d4b819ddd094dd279f92d15fef009b\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"3.2.1\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"false\",\n\t\t\t\t\t\"seccomp_listener\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"false\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"4.15.0-58-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": true,\n\t\t\t\t\"server_name\": \"lxd02\",\n\t\t\t\t\"server_pid\": 12274,\n\t\t\t\t\"server_version\": \"3.16\",\n\t\t\t\t\"storage\": \"ceph\",\n\t\t\t\t\"storage_version\": \"ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)\"\n\t\t\t}\n\t\t}\n\t}" 
t=2019-08-29T08:13:16+0000 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2019-08-29T08:13:16+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" 
t=2019-08-29T08:13:17+0000 lvl=dbug msg="Success for task operation: b910a266-8b46-49d5-a29c-b8b7fdec05cb" 
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Replace current raft nodes with [{ID:1 Address:100.65.20.225:8443} {ID:2 Address:100.65.20.230:8443} {ID:3 Address:100.65.20.231:8443}]" 
t=2019-08-29T08:13:19+0000 lvl=dbug msg="Cluster node is up-to-date" 
t=2019-08-29T08:13:19+0000 lvl=warn msg="Excluding offline node from refresh: {ID:2 Address:100.65.20.230:8443 RaftID:2 Raft:true LastHeartbeat:2019-08-29 08:12:38.900060709 +0000 UTC Online:false updated:false}" 
t=2019-08-29T08:13:20+0000 lvl=dbug msg="Found cert" name=0
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: Failed to fetch nodes: driver: bad connection" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (Failed to fetch nodes: driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: Failed to fetch nodes: driver: bad connection" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (Failed to fetch nodes: driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Retry failed db interaction (driver: bad connection)" 
t=2019-08-29T08:13:21+0000 lvl=warn msg="Failed to rollback transaction after error (driver: bad connection): cannot rollback - no transaction is active" 
t=2019-08-29T08:13:21+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"driver: bad connection\"}"

freeekanayaka · August 29, 2019, 12:09pm

The logs of lxd seem to start only at 8:42, while the other two cover an earlier time stamp. Please can you restart all three nodes about at the same time and then send me the logs after the restart, for all of them?

sarlacpit · September 3, 2019, 10:10am

First of all apologies for the delayed reply. I have only just sat back down to the lab.
I have just retested and have been unable to replicate thus far after several attempts.

I moved a few containers across from one cluster member to another the other day whilst demoing the lab to my manager.

This makes it really hard to debug now. I’ll give the containers a reshuffle (in accordance to the shell history) to see if I can get it misbehaving again.

roka · December 4, 2019, 4:50pm

Face the same issue 3.18 latest version, when the elected leader sudden shutdown the other 2 nodes cannot elect a leader and give the same error.

Error: failed to begin transaction: failed to create dqlite connection: no available dqlite leader server found

After restarting the lxd daemon on both the nodes it still gives the same error.