Writing a tutorial on how to develop LXD


#1

I am planning to write a tutorial on how to start developing on LXD.
What I have in mind is,

  1. The general instructions are available at https://github.com/lxc/lxd
  2. For developing on the client (lxc executable), the easiest is to compile LXD on the desktop system and then invoke ~/go/bin/lxc to run the freshly compiled version. This client would use the existing live LXD installation of the desktop system. This would easily work if LXD comes from the repositories. It is likely to work on LXD as a snap package with the appropriate lxc remote command.
  3. For more involved developing, a KVM VM will be used. I plan to use multipass and make instructions so that the environment would be easy to replicate with a few commands.

The part that I am missing, is how to initialize LXD in the KVM VM. I manage to compile LXD, but when I run lxd init, it gets stuck. Specifically,

  1. I use the instructions at https://github.com/lxc/lxd on 16.04.3. The exception is that I get go from the snap, as the repositories only have go 1.6. That version is too old for LXD and gives a (cryptic) error message when compiling.
  2. I run lxd as follows.
    ubuntu@lxd:~$ sudo -E $GOPATH/bin/lxd --group sudo init
    WARN[01-30|18:31:44] CGroup memory swap accounting is disabled, swap limits will be ignored.
  3. The ps command shows the following that are stuck there. Should’t lxc profile complete?

    1323 pts/0 S+ 0:00 sudo -E /home/ubuntu/go/bin/lxd --group sudo init
    1324 pts/0 Sl+ 0:00 /home/ubuntu/go/bin/lxd --group sudo init
    1397 ? Sl 0:00 lxd-bridge-proxy --addr=[fe80::1%lxdbr0]:13128
    1413 ? Ss 0:00 /bin/sh -e /usr/lib/lxd/profile-config
    1417 ? Sl 0:00 lxc profile device get default eth0 nictype --force-local

  4. If I press Ctrl+C, lxd is killed but there are other commands that are spawned. It is difficult to get a clean system and you have to restart the VM.

How should these issues be dealt with, so ~/go/bin/lxd works in KVM?


(Stéphane Graber) #2

You’re likely conflicting with the pre-installed LXD in 16.04.

You should do:

apt remove --purge lxd lxd-client
apt install golang-1.9

That will get rid of the preinstalled LXD and will install Go 1.9 from the archive.
You’ll have to add /usr/lib/go-1.9/bin/ (from memory so may be a bit different) to your PATH, so that you end up using that version of the compiler rather than 1.6.


#3

Indeed, the processes getting stuck was due to the stock LXD that I forgot to remove.

What confused me, is that I could not get lxd init to run, if the command line was

sudo -E $GOPATH/bin/lxd --group sudo init

Specifically,

ubuntu@lxd:~$ sudo -E $GOPATH/bin/lxd --group sudo init
WARN[01-30|22:59:51] CGroup memory swap accounting is disabled, swap limits will be ignored. 
(stays here until I Ctrl+C)

However,

ubuntu@lxd:~$ sudo -E $GOPATH/bin/lxd init
!!! Running init for you!
error: Unable to talk to LXD: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: no such file or directory

The text !!!Running init for you! is a print statement I put in lxd/lxd/main_init.go:func cmdInit()

What happens, is the the parser for the command line options expects the subcommand (i.e. init) to come first, and then any flags such as --group sudo in this case. However, if init is first, then --group sudo is not evaluated soon and init runs without proper group permissions (and fails).

I filed a bug report on this,


#4

(in continuation from the github report)

If I freshly compile LXD, can I run init or do I have to set up the networking and storage manually?


(Stéphane Graber) #5

Fresh LXD will have the network and storage APIs, so you just need to run one instance of lxd --debug --group lxd and then in a separate window you can run lxd init which will do everything through the internal API.


(David Negreira) #6

Hi Simos,

I actually had the same idea and have a lot of notes regarding developing and contributing to LXD, I will pass the notes that I have at home and feel free to adapt and put it on your article.


#7

Here is the first part of the tutorial,
https://blog.simos.info/how-to-develop-the-lxd-hypervisor-part-1-client-side-lxc/

Here it focuses on the development of the client (lxc), reusing the existing snap LXD installation (the server). There is no talk yet on making commits.

Please report anything that can make the tutorial better.


(David Negreira) #8

Hi Simos,
The only thing that I have to comment is that the golang version should be at least 1.8 and not the default from ubuntu which is 1.6, you will run into problems when compiling LXD with 1.6.
I usually install golang 1.9, you can install golang 1.9 with apt install golang-1.9
Be aware that this go binary will be in /usr/lib/go-1.9/bin so you will also have to add this foler to your $PATH


#9

The lxd binary needs a more recent Go. However, the client compiles with Go 1.6 though I understand that this might not be the case for too long.
For the purposes of a simpler tutorial that compiles only the client (note that I do not compile the server), it should suffice for now.


(Stéphane Graber) #10

We keep compatibility with 1.6 in the stable-2.0 branch of LXD, unfortunately the same can’t be said of our dependencies, which is why it’s not currently buildable with a straight go get on 1.6…

To build LXD 2.0.x with 1.6, you’ll need to use older versions of some of the packages that LXD depends on, that’s effectively what Ubuntu itself does with the packaged version of those in the archive.


#11

Regarding Go, is there a minimum version that is recommended for use with LXD?

The choice of Go version and the installation are quite involved, and unless I find a page that explains it all, I would have to write a dedicated page.
Specifically,

  1. In Ubuntu 16.04, one needs to enable the backports repository in order to install go-1.9. Enabling the backports requires some explanation of the pros/cons. Fortunately, the Xenial image in multipass has the backports enabled by default. A desktop 16.04 does not have them enabled.
  2. Adding into the $PATH the location of the new Go binary.

Would there be any reason not to such to some users the Go snap for LXD?

The Go snap looks like nice, and defaults to version 1.9.x.

$ snap info go
name:      go
summary:   Go programming language compiler, linker, stdlib
publisher: mwhudson
contact:   michael.hudson@ubuntu.com
description: |
  This snap provides an assembler, compiler, linker, and compiled libraries for
  the Go programming language.
snap-id:          Md1HBASHzP4i0bniScAjXGnOII9cEK6e
channels:                              
  stable:         1.9.3         (1333) 55MB  classic
  candidate:      ↑                          
  beta:           1.10beta1     (1167) 64MB  classic
  edge:           devel-851e98f (1356) 133MB classic
  1.10/stable:    –                          
  1.10/candidate: 1.10rc1       (1340) 64MB  classic
  1.10/beta:      1.10beta1     (1167) 64MB  classic
  1.10/edge:      ↑                          
  1.6/stable:     1.6.4         (122)  49MB  classic
  1.6/candidate:  ↑                          
  1.6/beta:       ↑                          
  1.6/edge:       ↑                          
  1.7/stable:     1.7.6         (324)  48MB  classic
  1.7/candidate:  ↑                          
  1.7/beta:       ↑                          
  1.7/edge:       ↑                          
  1.8/stable:     1.8.5         (1018) 51MB  classic
  1.8/candidate:  ↑                          
  1.8/beta:       ↑                          
  1.8/edge:       ↑                          
  1.9/stable:     1.9.3         (1333) 55MB  classic
  1.9/candidate:  ↑                          
  1.9/beta:       ↑                          
  1.9/edge:       ↑

#12

So I wrote a post to deal with installing Go,
https://blog.simos.info/installing-the-go-programming-language-in-ubuntu/


(Stéphane Graber) #13

I’ve never actually used the Go snap before, it may be perfectly fine though.

I’m not sure what you mean by cons of having backports enabled, the only thing I can think of is the very slight extra download overhead when getting the package index files. Backports is scored such that it will never automatically update packages from it, only packages that you specifically grab from backports will be coming from it, making it safe to always enable everywhere (and why we try to do so in Ubuntu in general).

For Go version, the most reason the better really, but 1.7 or higher is fine right now and is what we test for.
As I said before, our own code is fine with 1.6 for our stable branch, but our upstreams have broken compatibility so you need some mangling to work around that…

Anyway, these days, if you have it, I’d say just go with 1.9.


#14

Here is Part #2,
https://blog.simos.info/how-to-develop-the-lxd-hypervisor-part-2-using-multipass-kvm-vm/

It talks about setting up multipass. I followed https://github.com/lxc/lxd and made sure to include only those packages that are missing from the Ubuntu image.
I show how to run lxd in one window, and lxd init/lxc in the other.
I close with using lxd-benchmark.
At the end I added a troubleshooting section with a summary of the issues I encountered.