Unable to remove node from cluster

Hi All,

Thank you in advance for any guidance/help on this.

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxd sql local "select * from raft_nodes"
+----+--------------------+------+
| id |      address       | role |
+----+--------------------+------+
| 1  | 172.16.32.10:8443  | 1    |
| 2  | 172.16.32.11:8443  | 0    |
| 3  | 172.16.32.12:8443  | 1    |
| 4  | 172.16.32.13:8443  | 0    |
| 5  | 172.16.32.14:8443  | 0    |
| 6  | 172.16.32.15:8443  | 2    |
| 7  | 172.16.32.16:8443  | 2    |
| 8  | 172.16.32.250:8443 | 2    |
| 10 | 172.16.32.17:8443  | 2    |
| 11 | 172.16.32.18:8443  | 2    |
| 12 | 172.16.32.19:8443  | 2    |
| 13 | 172.16.32.20:8443  | 2    |
| 14 | 172.16.32.21:8443  | 2    |
+----+--------------------+------+

Trying to remove node 8

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxc cluster remove lxdvm-ibm-sec-01.uat-03.example.com
Error: No such object
root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxc storage list local
+-------+-------------+--------+---------+---------+
| NAME  | DESCRIPTION | DRIVER |  STATE  | USED BY |
+-------+-------------+--------+---------+---------+
| local |             | zfs    | CREATED | 2       |
+-------+-------------+--------+---------+---------+
root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxc storage show local
config: {}
description: ""
name: local
driver: zfs
used_by:
- /1.0/images/e04d2d6c2900b7529ecad27529ca86a93c205286e82463c6b057fcda610c0a6c
- /1.0/profiles/default
status: Created
locations:
- lxdvm-minio-01.uat-03.example.com
- lxdvm-dbreads-01.uat-03.example.com
- lxdvm-dbbackup-01.uat-03.example.com
- lxdvm-sec-01.uat-03.example.com
- lxdvm-sec-02.uat-03.example.com
- lxdvm-dmz-01.uat-03.example.com
- lxdvm-dmz-02.uat-03.example.com
- lxdvm-ibm-sec-01.uat-03.example.com
- lxdvm-legacy-sec-01.uat-03.example.com
- lxdvm-legacy-sec-02.uat-03.example.com
- lxdvm-legacy-dmz-01.uat-03.example.com
- lxdvm-legacy-dmz-02.uat-03.example.com
- lxdvm-dbmasters-01.uat-03.example.com
root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxd sql global "SELECT * FROM storage_volumes WHERE name='e04d2d6c2900b7529ecad27529ca86a93c205286e82463c6b057fcda610c0a6c';" 
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| id |                               name                               | storage_pool_id | node_id | type | description | project_id |
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| 52 | e04d2d6c2900b7529ecad27529ca86a93c205286e82463c6b057fcda610c0a6c | 1               | 8       | 1    |             | 1          |
+----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxc storage volume list local
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+
| TYPE  |                               NAME                               | DESCRIPTION | USED BY |           LOCATION           |
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+
| image | 88eb6d9e7832042f7b928b1ef034cb1a3ddd303fb94fd1410b23cf87e0660082 |             | 1       | lxdvm-dmz-01.uat-03.example.com |
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxd sql global "SELECT * FROM storage_volumes WHERE name='88eb6d9e7832042f7b928b1ef034cb1a3ddd303fb94fd1410b23cf87e0660082';" 
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| id  |                               name                               | storage_pool_id | node_id | type | description | project_id |
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| 104 | 88eb6d9e7832042f7b928b1ef034cb1a3ddd303fb94fd1410b23cf87e0660082 | 1               | 6       | 1    |             | 1          |
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+

I am not sure how to proceed to remove it, also, why is the volume_list not showing correctly?

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# snap list|grep lxd
lxd     4.0.1     14804  latest/stable/…  canonical*  -

if I do:

root@lxdvm-ibm-sec-01.uat-03.example.com:/home/jonathan# lxd sql global "SELECT * FROM storage_volumes;" 
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| id  |                               name                               | storage_pool_id | node_id | type | description | project_id |
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+
| 1   | mcon-dbbackup-01-uat-03-example.com                                 | 1               | 3       | 0    |             | 1          |
| 2   | 5fce6c808f3a3e470bb10978ffd8bf701b4f1bb4ca39c4f419c05cdce642af19 | 1               | 3       | 1    |             | 1          |
| 3   | mcon-nfs-01-uat-03-example.com                                      | 1               | 1       | 0    |             | 1          |
| 4   | 86a6549c9bdf47f3bbecb9f6be99626b2ef059e254c63c81997d625a15e45f82 | 1               | 1       | 1    |             | 1          |
| 5   | mcon-nfs-02-uat-03-example.com                                      | 1               | 1       | 0    |             | 1          |
| 6   | mcon-nfs-03-uat-03-example.com                                      | 1               | 1       | 0    |             | 1          |
| 7   | mcon-archive-01-uat-03-example.com                                  | 1               | 1       | 0    |             | 1          |
| 8   | 54cb9b9b938050f2177b11695d3a4b40645f2de5d8d15bf1acb4061f4215164e | 1               | 1       | 1    |             | 1          |
| 10  | 54cb9b9b938050f2177b11695d3a4b40645f2de5d8d15bf1acb4061f4215164e | 1               | 10      | 1    |             | 1          |
| 11  | DB-01-UAT-EU                                                     | 1               | 10      | 0    |             | 1          |
| 12  | MAP-01-UAT-EU                                                    | 1               | 11      | 0    |             | 1          |
| 13  | 54cb9b9b938050f2177b11695d3a4b40645f2de5d8d15bf1acb4061f4215164e | 1               | 11      | 1    |             | 1          |
| 14  | PM-01-UAT-EU                                                     | 1               | 10      | 0    |             | 1          |
| 15  | 2ecdb74f3277ad3c0385fe52a6f869b04d2eb98e46fa0084fbefdca45d95f08e | 1               | 10      | 1    |             | 1          |
| 16  | mcon-notify-01-uat                                               | 1               | 11      | 0    |             | 1          |
| 17  | 2ecdb74f3277ad3c0385fe52a6f869b04d2eb98e46fa0084fbefdca45d95f08e | 1               | 11      | 1    |             | 1          |
| 18  | MCON-APP-02-UAT-03                                               | 1               | 10      | 0    |             | 1          |
| 20  | mcon-dataprocessingservice-01-uat-03-example.com                    | 1               | 10      | 0    |             | 1          |
| 22  | 2ecdb74f3277ad3c0385fe52a6f869b04d2eb98e46fa0084fbefdca45d95f08e | 1               | 12      | 1    |             | 1          |
| 26  | WEB-01-UAT-EU                                                    | 1               | 12      | 0    |             | 1          |
| 29  | 59bd48b80edb25ea8886ebe7152bad491c0b80c8784b8b8ea8253a57ee64ab53 | 1               | 13      | 1    |             | 1          |
| 30  | COMMS-01-UAT-EU                                                  | 1               | 13      | 0    |             | 1          |
| 31  | FAX-01-UAT-EU                                                    | 1               | 13      | 0    |             | 1          |
| 32  | AS2-01-UAT-03                                                    | 1               | 12      | 0    |             | 1          |
| 33  | 59bd48b80edb25ea8886ebe7152bad491c0b80c8784b8b8ea8253a57ee64ab53 | 1               | 12      | 1    |             | 1          |
| 34  | ECAT-DYADEM-UAT-03                                               | 1               | 12      | 0    |             | 1          |
| 35  | OFTP2-01-UAT-03                                                  | 1               | 13      | 0    |             | 1          |
| 36  | mcon-ecatuatfr-01                                                | 1               | 12      | 0    |             | 1          |
| 37  | mcon-ecatuatus-01                                                | 1               | 12      | 0    |             | 1          |
| 38  | mcon-ecatyess-01                                                 | 1               | 13      | 0    |             | 1          |
| 39  | mcon-proxy-01-uat                                                | 1               | 13      | 0    |             | 1          |
| 40  | mcon-mail-02                                                     | 1               | 13      | 0    |             | 1          |
| 41  | WEB-02-UAT-EU                                                    | 1               | 12      | 0    |             | 1          |
| 42  | mcon-minionfs-01-uat-03-example.com                                 | 1               | 1       | 0    |             | 1          |
| 43  | 4216b0190243f3110354038970059581c3ba542b01a2e6800845cc37ff1a0feb | 1               | 1       | 1    |             | 1          |
| 44  | mcon-minionfs-02-uat-03-example.com                                 | 1               | 1       | 0    |             | 1          |
| 45  | mcon-minionfs-03-uat-03-example.com                                 | 1               | 1       | 0    |             | 1          |
| 46  | mcon-db-ocr-01-uat-03-example.com                                   | 1               | 14      | 0    |             | 1          |
| 47  | 86962d0ec09d4353002dbe72eefda0b5a0d251637030e739da8a054d9ab6132d | 1               | 14      | 1    |             | 1          |
| 48  | mcon-dbbackup-ocr-01-uat-03-example.com                             | 1               | 3       | 0    |             | 1          |
| 49  | 86962d0ec09d4353002dbe72eefda0b5a0d251637030e739da8a054d9ab6132d | 1               | 3       | 1    |             | 1          |
| 52  | e04d2d6c2900b7529ecad27529ca86a93c205286e82463c6b057fcda610c0a6c | 1               | 8       | 1    |             | 1          |
| 53  | mcon-ecat-ascometal-01-uat-03-example.com                           | 1               | 6       | 0    |             | 1          |
| 54  | e0df8095ab8021c46a0edbbd232693d622a7349b25a07a0cb327b3608e1cda54 | 1               | 6       | 1    |             | 1          |
| 55  | mcon-db-map-01-uat-03-example.com                                   | 1               | 14      | 0    |             | 1          |
| 56  | 68789bbf724bab46840d77049a415a3f077c65c3f6519ce42e2a94118b5f8cc8 | 1               | 14      | 1    |             | 1          |
| 57  | mcon-dbbackup-map-01-uat-03-example.com                             | 1               | 3       | 0    |             | 1          |
| 58  | 68789bbf724bab46840d77049a415a3f077c65c3f6519ce42e2a94118b5f8cc8 | 1               | 3       | 1    |             | 1          |
| 59  | mcon-minio-v2todataprocessing-01-uat-03-example.com                 | 1               | 1       | 0    |             | 1          |
| 60  | 565bb5d3279e982192ff3081983e7c9f8de7f9b939d02f5ca81bd29312024bd4 | 1               | 1       | 1    |             | 1          |
| 61  | mcon-phoenixapp-08-uat-03-example.com                               | 1               | 4       | 0    |             | 1          |
| 62  | c513df7ae070c1cb302290129d0eaf8ba424ee1fc51c1f8f74ae0815d42d7749 | 1               | 4       | 1    |             | 1          |
| 63  | mcon-phoenixcomms-03-uat-03-example.com                             | 1               | 6       | 0    |             | 1          |
| 64  | c513df7ae070c1cb302290129d0eaf8ba424ee1fc51c1f8f74ae0815d42d7749 | 1               | 6       | 1    |             | 1          |
| 65  | mcon-map-api-01-uat-03-example.com                                  | 1               | 5       | 0    |             | 1          |
| 66  | mcon-phoenix-webapi-01-uat-03-example.com                           | 1               | 7       | 0    |             | 1          |
| 67  | 1c999d7a49dc0be365aa523fbff832213351f90a1b0c001e99a45c914110f111 | 1               | 7       | 1    |             | 1          |
| 68  | mcon-web-slgbrands-01-uat-03-example.com                            | 1               | 7       | 0    |             | 1          |
| 69  | f672fa184ca6c52c2bc760f3584839890302d715f1cb2c8267892c9c4774d18f | 1               | 7       | 1    |             | 1          |
| 70  | mcon-db-web-01-uat-03-example.com                                   | 1               | 14      | 0    |             | 1          |
| 71  | f672fa184ca6c52c2bc760f3584839890302d715f1cb2c8267892c9c4774d18f | 1               | 14      | 1    |             | 1          |
| 72  | mcon-dbbackup-web-01-uat-03-example.com                             | 1               | 3       | 0    |             | 1          |
| 73  | f672fa184ca6c52c2bc760f3584839890302d715f1cb2c8267892c9c4774d18f | 1               | 3       | 1    |             | 1          |
| 74  | mcon-map-02-uat-03-example.com                                      | 1               | 4       | 0    |             | 1          |
| 75  | d5b2972fae2bb20ec1db4976b154301366db24aedc7fde9dad892b37329b3f7e | 1               | 4       | 1    |             | 1          |
| 78  | mcon-web-atkore-01-uat-03-example.com                               | 1               | 6       | 0    |             | 1          |
| 79  | b7d14fad64e6fb8ff60570430785fbaeb92e7363393e0ba5e48e8204422a0e33 | 1               | 6       | 1    |             | 1          |
| 80  | ubuntu20-test-chi                                                | 1               | 6       | 0    |             | 1          |
| 81  | 39a441e6ae3ba581669000982704a73c8a43ad70aa3ff1720e4c74fd142c5bc4 | 1               | 6       | 1    |             | 1          |
| 82  | mcon-phoenixapi-01-uat-03-example.com                               | 1               | 5       | 0    |             | 1          |
| 83  | 39a441e6ae3ba581669000982704a73c8a43ad70aa3ff1720e4c74fd142c5bc4 | 1               | 5       | 1    |             | 1          |
| 89  | 3ef43bb0edb0c5eeba5db15c00aca476b69787087e7e46635d398c903c1777c8 | 1               | 12      | 1    |             | 1          |
| 90  | eb71b35fc423267ff277cc29ced2fda194d70fc7819f12408e3d2c2aa22b2155 | 1               | 12      | 1    |             | 1          |
| 91  | mcon-as2-01-uat-03-example.com                                      | 1               | 12      | 0    |             | 1          |
| 92  | 4aefcf7e6d888306251df1035c99f9d1924a5c146462da640dbab1bff2f7d2eb | 1               | 12      | 1    |             | 1          |
| 94  | f5a952493510bfb5dfa54c4e658a031df7ed01ae5b62875984c9484149c470e8 | 1               | 6       | 1    |             | 1          |
| 97  | mcon-customercentrewebportal-01-uat-03-example.com                  | 1               | 6       | 0    |             | 1          |
| 98  | bac175b3494cd9c47a44ec211ec3f6f6ff9a122f867c9732ed9e2a38218bc533 | 1               | 6       | 1    |             | 1          |
| 100 | mcon-domibus-01-uat-03-example.com                                  | 1               | 6       | 0    |             | 1          |
| 101 | 84925ba34af021d1b08bb84f09a14192fb9075efb29a8c9cffe0404fd9e6292b | 1               | 6       | 1    |             | 1          |
| 102 | mcon-smp-01-uat-03-example.com                                      | 1               | 7       | 0    |             | 1          |
| 103 | 84925ba34af021d1b08bb84f09a14192fb9075efb29a8c9cffe0404fd9e6292b | 1               | 7       | 1    |             | 1          |
| 104 | 88eb6d9e7832042f7b928b1ef034cb1a3ddd303fb94fd1410b23cf87e0660082 | 1               | 6       | 1    |             | 1          |
+-----+------------------------------------------------------------------+-----------------+---------+------+-------------+------------+

but if I do:

root@lxdvm-ibm-sec-01.uat-03.b2be.com:/home/jonathan# lxc storage volume list local
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+
| TYPE  |                               NAME                               | DESCRIPTION | USED BY |           LOCATION           |
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+
| image | 88eb6d9e7832042f7b928b1ef034cb1a3ddd303fb94fd1410b23cf87e0660082 |             | 1       | lxdvm-dmz-01.uat-03.b2be.com |
+-------+------------------------------------------------------------------+-------------+---------+------------------------------+

I only see the one image?

Thank you in advance for the support.

Cheers
Jonathan

So that’s a problem ^

The current stable release for LXD 4.0 LTS is 4.0.8, you’re on something that’s almost 2 years out of date here.

Try running snap refresh lxd, if that doesn’t work, you’ll need to investigate potential firewall/proxy issues. Once you get it onto 4.0.8 we’ll be able to assist.

Hi Stéphane,

Ah thanks for pointing that out, we will update, should be able to do it without restarts?

Yeah, LXD itself will restart but the instances won’t be restarted and you shouldn’t need to restart the host systems.