Lxd-p2c 'admin password' error

(Dan Mac Donald) #1

I’m trying to get lxd-p2c to work but I have not been able to get past its prompt for an admin password, every time it tells me:

Error: not authorized

First off the bat, I must say that is a vague request on behalf of lxd-p2c followed with an equally vague error message. What is the ‘Admin password’? Admin of what, where? If p2c actually means the root password of the LXD server then that’s what it should ask for. I’ve tried that several times with no luck. I’m happy to open a bug report to request this is made clearer to the user.

I have run:

# lxc config set core.trust_password foopass

On my LXD server (as well as having run lxd init) but entering that password has also failed with the same “Error: not authorized” message.

If I run

# lxc config trust list

On my LXD server, I get an empty list so I presume I need to add my host to be containerised to this list before I can use p2c but I have been unable to find any docs on how to achieve this. My suspicion is that it might involve a command such as

# lxc config trust add ...

but that seems to require a cert file? What cert? Is that the cert fingerprint printed on the source machine when I run p2c?

I also tried running lxc monitor on my LXD server whilst I ran lxd-p2c on the source machine but it didn’t tell me anything more - no errors or warnings get printed.

There doesn’t seem to be any documentation for lxd-p2c apart from what is available via --help, right?

I am running lxd/lxc 3.0.2 from 16.04 backports (on a Ubuntu 16.04 / amd64 server) and I am testing lxd-p2c built from gh on a Ubuntu 18.04 amd64 machine.


(Stéphane Graber) #2

Nope, nothing to do with lxc config trust, you should only need to enter the trust password.

One thing to make sure though, since this is TLS based authentication is that both server’s time and date are correct, if one of them is way off, that’d explain the error.

(Dan Mac Donald) #3

I’m getting a bit further now, but still no luck:

sudo ./lxd-p2c https://server:8443 test /
Generating a temporary client certificate. This may take a minute…
Certificate fingerprint: 50c60f7057c222b9c70b1fd6d44952ab686f9d8722aa3ad24806a8e275bc70d7
ok (y/n)? y
Admin password for https://server:8443:
Error: Failed to rsync: exit status 12
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]

The LXD log:

ephemeral=false lvl=info msg=“Creating container” name=test t=2018-12-13T14:03:06+0000
ephemeral=false lvl=info msg=“Created container” name=test t=2018-12-13T14:03:06+0000
lvl=warn msg=“Unable to update backup.yaml at this time” name=test t=2018-12-13T14:03:06+0000
lvl=eror msg="Rsync receive failed: /var/lib/lxd/containers/test/: exit status 12: " t=2018-12-13T14:03:06+0000
err=“exit status 12” lvl=eror msg=“Error during migration sink” t=2018-12-13T14:03:06+0000
created=2018-12-13T14:03:05+0000 ephemeral=false lvl=info msg=“Deleting container” name=test t=2018-12-13T14:03:06+0000 used=1970-01-01T01:00:00+0100
created=2018-12-13T14:03:05+0000 ephemeral=false lvl=info msg=“Deleting container” name=test t=2018-12-13T14:03:06+0000 used=1970-01-01T01:00:00+0100
created=2018-12-13T14:03:05+0000 ephemeral=false lvl=info msg=“Deleted container” name=test t=2018-12-13T14:03:07+0000 used=1970-01-01T01:00:00+0100
err=“sql: no rows in result set” lvl=eror msg=“Failed deleting container entry” name=test t=2018-12-13T14:03:26+0000

LXD server 3.0.2 / 16.04, client is 18.04

(Stéphane Graber) #4

Did you build the 3.0.2 version of lxd-p2c? Otherwise there may be some rsync options mismatch.

(Dan Mac Donald) #5

Correct - I was using lxd-p2c built from git. I have had no luck in building lxd-p2c from the 3.0.2 source tarball.

3.0.2 doesn’t have a lxd-p2c makefile target. I tried adding it in from the latest git but it gave me several missing dependency errors, just like when I cd into its dir and run go build.

I have tried running make deps in the top level of the lxd source tree. After fixing up the makefile to include the library path for dqlite, I got the deps for 3.0.2 to build but lxd-p2c still complains about several missing deps. I’ve had a hell of time getting lxd-p2c to build and run, mainly because I’m unfamiliar with go.

~/src/lxd-3.0.2/lxd-p2c$ go build
transfer.go:12:2: cannot find package “github.com/gorilla/websocket” in any of:
/usr/lib/go-1.10/src/github.com/gorilla/websocket (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/gorilla/websocket (from $GOPATH)
utils.go:13:2: cannot find package “github.com/lxc/lxd/client” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/client (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/client (from $GOPATH)
main_migrate.go:14:2: cannot find package “github.com/lxc/lxd/lxc/utils” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/lxc/utils (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/lxc/utils (from $GOPATH)
transfer.go:15:2: cannot find package “github.com/lxc/lxd/lxd/migration” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/lxd/migration (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/lxd/migration (from $GOPATH)
transfer.go:16:2: cannot find package “github.com/lxc/lxd/shared” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/shared (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/shared (from $GOPATH)
main_migrate.go:15:2: cannot find package “github.com/lxc/lxd/shared/api” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/shared/api (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/shared/api (from $GOPATH)
main_netcat.go:12:2: cannot find package “github.com/lxc/lxd/shared/eagain” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/shared/eagain (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/shared/eagain (from $GOPATH)
main_migrate.go:16:2: cannot find package “github.com/lxc/lxd/shared/osarch” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/shared/osarch (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/shared/osarch (from $GOPATH)
main.go:8:2: cannot find package “github.com/lxc/lxd/shared/version” in any of:
/usr/lib/go-1.10/src/github.com/lxc/lxd/shared/version (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/lxc/lxd/shared/version (from $GOPATH)
transfer.go:13:2: cannot find package “github.com/pborman/uuid” in any of:
/usr/lib/go-1.10/src/github.com/pborman/uuid (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/pborman/uuid (from $GOPATH)
main.go:6:2: cannot find package “github.com/spf13/cobra” in any of:
/usr/lib/go-1.10/src/github.com/spf13/cobra (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/github.com/spf13/cobra (from $GOPATH)
utils.go:11:2: cannot find package “golang.org/x/crypto/ssh/terminal” in any of:
/usr/lib/go-1.10/src/golang.org/x/crypto/ssh/terminal (from $GOROOT)
/home/local/ISDADS/sgs548/go/src/golang.org/x/crypto/ssh/terminal (from $GOPATH)

Maybe I can build it with go get if I can specify a specific release? I’d prefer to stick with the 3.0.2 backports package but maybe I’ll have to upgrade the server to the latest git release if I want to use lxd-p2c?

(Stéphane Graber) #6

3.0.3 is on its way to Ubuntu 18.04, likely to be released next week and will have the newer rsync receiver code which should avoid this issue.

Building the 3.0.2 version is easiest done with:

stgraber@castiana:~$ go get github.com/lxc/lxd/lxd-p2c
stgraber@castiana:~$ cd go/src/github.com/lxc/lxd
stgraber@castiana:~/go/src/github.com/lxc/lxd (master)$ git checkout lxd-3.0.2
Note: checking out 'lxd-3.0.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2fae3c53 Release LXD 3.0.2
stgraber@castiana:~/go/src/github.com/lxc/lxd ((HEAD detached at lxd-3.0.2))$ go get github.com/lxc/lxd/lxd-p2c
stgraber@castiana:~/go/src/github.com/lxc/lxd ((HEAD detached at lxd-3.0.2))$ 

(Dan Mac Donald) #7

Thanks for your help Stephane, I can now build the correct version of lxd-p2c, it connects and it starts transferring the container but unfortunately it doesn’t stop!

As in my previous posts, for my test I am trying to transfer my 18.04 laptop onto our 16.04 server running LXD 3.0.2. I have stopped the logging services on my laptop using these commands:

$ sudo systemctl disable rsyslog
$ sudo systemctl stop rsyslog
$ sudo systemctl mask systemd-journald
$ sudo systemctl stop systemd-journald

I am not running any databases or servers on my laptop, I closed my browser and all other apps except X and my xterm and I didn’t do anything else on my laptop whilst lxd-p2c was running. I only have 18 GB of data on my laptop drive but lxd-p2c said it had transferred 120+ GB of data to the container image before I decided to stop it as something was clearly wrong by that point.

So my next bit of feedback / feature request is for lxd-p2c to tell you not only how much data has supposedly been transferred (and the data rate) but it should also tell you the total amount to be transferred. An ETA would be nice too but not as important as knowing the total to be transferred according to lxd-p2c.

Here’s the output of systemctl list-unit-files from my laptop:

UNIT FILE                                  STATE          
home-local-ISDADS-sgs548-SGS548.automount  generated      
proc-sys-fs-binfmt_misc.automount          static         
-.mount                                    generated      
boot-efi.mount                             generated      
dev-hugepages.mount                        static         
dev-mqueue.mount                           static         
home-local-ISDADS-sgs548-SGS548.mount      generated      
proc-sys-fs-binfmt_misc.mount              static         
snap-core-5742.mount                       enabled        
snap-core-5897.mount                       enabled        
snap-core-6034.mount                       enabled        
snap-gnome\x2d3\x2d26\x2d1604-70.mount     enabled        
snap-gnome\x2d3\x2d26\x2d1604-74.mount     enabled        
snap-gnome\x2dcalculator-180.mount         enabled        
snap-gnome\x2dcalculator-238.mount         enabled        
snap-gnome\x2dcalculator-260.mount         enabled        
snap-gnome\x2dcharacters-103.mount         enabled        
snap-gnome\x2dcharacters-139.mount         enabled        
snap-gnome\x2dlogs-37.mount                enabled        
snap-gnome\x2dlogs-45.mount                enabled        
snap-gnome\x2dsystem\x2dmonitor-51.mount   enabled        
snap-gnome\x2dsystem\x2dmonitor-57.mount   enabled        
snap-gtk\x2dcommon\x2dthemes-701.mount     enabled        
snap-gtk\x2dcommon\x2dthemes-808.mount     enabled        
snap-gtk\x2dcommon\x2dthemes-818.mount     enabled        
sys-fs-fuse-connections.mount              static         
sys-kernel-config.mount                    static         
sys-kernel-debug.mount                     static         
acpid.path                                 enabled        
apport-autoreport.path                     enabled        
cups.path                                  enabled        
systemd-ask-password-console.path          static         
systemd-ask-password-plymouth.path         static         
systemd-ask-password-wall.path             static         
session-c2.scope                           transient      
accounts-daemon.service                    enabled        
acpid.service                              disabled       
alsa-restore.service                       static         
alsa-state.service                         static         
alsa-utils.service                         masked         
anacron.service                            enabled        
apparmor.service                           enabled        
apport-autoreport.service                  static         
apport-forward@.service                    static         
apport.service                             generated      
apt-daily-upgrade.service                  static         
apt-daily.service                          static         
autovt@.service                            enabled        
avahi-daemon.service                       enabled        
bluetooth.service                          enabled        
bolt.service                               static         
bootlogd.service                           masked         
bootlogs.service                           masked         
bootmisc.service                           masked         
brltty-udev.service                        static         
brltty.service                             disabled       
checkfs.service                            masked         
checkroot-bootclean.service                masked         
checkroot.service                          masked         
clean-mount-point@.service                 static         
colord.service                             static         
configure-printer@.service                 static         
console-getty.service                      disabled       
console-setup.service                      enabled        
container-getty@.service                   static         
cpufrequtils.service                       generated      
cron.service                               enabled        
cryptdisks-early.service                   masked         
cryptdisks.service                         masked         
cups-browsed.service                       enabled        
cups.service                               enabled        
dbus-fi.w1.wpa_supplicant1.service         enabled        
dbus-org.bluez.service                     enabled        
dbus-org.freedesktop.Avahi.service         enabled        
dbus-org.freedesktop.hostname1.service     static         
dbus-org.freedesktop.locale1.service       static         
dbus-org.freedesktop.login1.service        static         
dbus-org.freedesktop.ModemManager1.service enabled        
dbus-org.freedesktop.nm-dispatcher.service enabled        
dbus-org.freedesktop.resolve1.service      enabled        
dbus-org.freedesktop.thermald.service      enabled        
dbus-org.freedesktop.timedate1.service     static         
dbus.service                               static         
debug-shell.service                        disabled       
display-manager.service                    indirect       
dns-clean.service                          enabled        
emergency.service                          static         
friendly-recovery.service                  static         
fstrim.service                             static         
fuse.service                               masked         
fwupd-offline-update.service               static         
fwupd.service                              static         
fwupdate-cleanup.service                   static         
gdm.service                                static         
gdm3.service                               static         
geoclue.service                            static         
getty-static.service                       static         
getty@.service                             enabled        
gpu-manager.service                        enabled        
grub-common.service                        generated      
halt.service                               masked         
hddtemp.service                            generated      
hostname.service                           masked         
hwclock.service                            masked         
ifup@.service                              static         
iio-sensor-proxy.service                   static         
initrd-cleanup.service                     static         
initrd-parse-etc.service                   static         
initrd-switch-root.service                 static         
initrd-udevadm-cleanup-db.service          static         
irqbalance.service                         enabled        
kerneloops.service                         enabled        
keyboard-setup.service                     enabled        
killprocs.service                          masked         
kmod-static-nodes.service                  static         
kmod.service                               static         
lightdm.service                            indirect       
lm-sensors.service                         enabled        
loadcpufreq.service                        generated      
lwsmd.service                              enabled        
mdadm-grow-continue@.service               static         
mdadm-last-resort@.service                 static         
mdadm-shutdown.service                     disabled       
mdadm-waitidle.service                     masked         
mdadm.service                              masked         
mdmon@.service                             static         
mdmonitor.service                          static         
ModemManager.service                       enabled        
module-init-tools.service                  static         
motd-news.service                          static         
motd.service                               masked         
mountall-bootclean.service                 masked         
mountall.service                           masked         
mountdevsubfs.service                      masked         
mountkernfs.service                        masked         
mountnfs-bootclean.service                 masked         
mountnfs.service                           masked         
netplan-wpa@.service                       static         
network-manager.service                    enabled        
networkd-dispatcher.service                enabled        
networking.service                         enabled        
NetworkManager-dispatcher.service          enabled        
NetworkManager-wait-online.service         enabled        
NetworkManager.service                     enabled        
ondemand.service                           enabled        
openvpn-client@.service                    disabled       
openvpn-server@.service                    disabled       
openvpn.service                            enabled        
openvpn@.service                           disabled       
packagekit-offline-update.service          static         
packagekit.service                         static         
plymouth-halt.service                      static         
plymouth-kexec.service                     static         
plymouth-log.service                       static         
plymouth-poweroff.service                  static         
plymouth-quit-wait.service                 static         
plymouth-quit.service                      static         
plymouth-read-write.service                static         
plymouth-reboot.service                    static         
plymouth-start.service                     static         
plymouth-switch-root.service               static         
plymouth.service                           static         
polkit.service                             static         
postfix.service                            enabled        
postfix@.service                           indirect       
pppd-dns.service                           enabled        
procps.service                             static         
quotaon.service                            static         
rc-local.service                           static         
rc.local.service                           static         
rc.service                                 masked         
rcS.service                                masked         
reboot.service                             masked         
rescue.service                             static         
rmnologin.service                          masked         
rsync.service                              enabled        
rsyslog.service                            disabled       
rtkit-daemon.service                       disabled       
saned.service                              masked         
saned@.service                             indirect       
sendsigs.service                           masked         
serial-getty@.service                      disabled       
setvtrgb.service                           enabled        
single.service                             masked         
smartd.service                             enabled        
smartmontools.service                      enabled        
snapd.autoimport.service                   enabled        
snapd.core-fixup.service                   enabled        
snapd.seeded.service                       enabled        
snapd.service                              enabled        
snapd.snap-repair.service                  static         
snapd.system-shutdown.service              enabled        
speech-dispatcher.service                  generated      
spice-vdagent.service                      enabled        
spice-vdagentd.service                     enabled        
ssh.service                                enabled        
ssh@.service                               static         
sshd.service                               enabled        
stop-bootlogd-single.service               masked         
stop-bootlogd.service                      masked         
sudo.service                               masked         
system-update-cleanup.service              static         
systemd-ask-password-console.service       static         
systemd-ask-password-plymouth.service      static         
systemd-ask-password-wall.service          static         
systemd-backlight@.service                 static         
systemd-binfmt.service                     static         
systemd-exit.service                       static         
systemd-fsck-root.service                  enabled-runtime
systemd-fsck@.service                      static         
systemd-fsckd.service                      static         
systemd-halt.service                       static         
systemd-hibernate-resume@.service          static         
systemd-hibernate.service                  static         
systemd-hostnamed.service                  static         
systemd-hwdb-update.service                static         
systemd-hybrid-sleep.service               static         
systemd-initctl.service                    static         
systemd-journal-flush.service              static         
systemd-journald.service                   masked         
systemd-kexec.service                      static         
systemd-localed.service                    static         
systemd-logind.service                     static         
systemd-machine-id-commit.service          static         
systemd-modules-load.service               static         
systemd-networkd-wait-online.service       disabled       
systemd-networkd.service                   disabled       
systemd-poweroff.service                   static         
systemd-quotacheck.service                 static         
systemd-random-seed.service                static         
systemd-reboot.service                     static         
systemd-remount-fs.service                 static         
systemd-resolved.service                   enabled        
systemd-rfkill.service                     static         
systemd-suspend-then-hibernate.service     static         
systemd-suspend.service                    static         
systemd-sysctl.service                     static         
systemd-timedated.service                  static         
systemd-timesyncd.service                  enabled        
systemd-tmpfiles-clean.service             static         
systemd-tmpfiles-setup-dev.service         static         
systemd-tmpfiles-setup.service             static         
systemd-udev-settle.service                static         
systemd-udev-trigger.service               static         
systemd-udevd.service                      static         
systemd-update-utmp-runlevel.service       static         
systemd-update-utmp.service                static         
systemd-user-sessions.service              static         
systemd-volatile-root.service              static         
thermald.service                           enabled        
udev.service                               static         
udisks2.service                            enabled        
ufw.service                                enabled        
umountfs.service                           masked         
umountnfs.service                          masked         
umountroot.service                         masked         
unattended-upgrades.service                enabled        
upower.service                             disabled       
urandom.service                            static         
ureadahead-stop.service                    static         
ureadahead.service                         enabled        
usb_modeswitch@.service                    static         
usbmuxd.service                            static         
user@.service                              static         
uuidd.service                              indirect       
wacom-inputattach@.service                 static         
whoopsie.service                           enabled        
wpa_supplicant-wired@.service              disabled       
wpa_supplicant.service                     enabled        
wpa_supplicant@.service                    disabled       
x11-common.service                         masked         
zed.service                                masked         
zfs-import-cache.service                   masked         
zfs-import-scan.service                    masked         
zfs-load-module.service                    masked         
zfs-mount.service                          masked         
zfs-share.service                          masked         
zfs-zed.service                            masked         
machine.slice                              static         
system.slice                               static         
user-783016879.slice                       transient      
user.slice                                 static         
acpid.socket                               enabled        
apport-forward.socket                      enabled        
avahi-daemon.socket                        enabled        
cups.socket                                enabled        
dbus.socket                                static         
saned.socket                               disabled       
snapd.socket                               enabled        
ssh.socket                                 disabled       
syslog.socket                              static         
systemd-fsckd.socket                       static         
systemd-initctl.socket                     static         
systemd-journald-audit.socket              static         
systemd-journald-dev-log.socket            static         
systemd-journald.socket                    static         
systemd-networkd.socket                    disabled       
systemd-rfkill.socket                      static         
systemd-udevd-control.socket               static         
systemd-udevd-kernel.socket                static         
uuidd.socket                               enabled        
swapfile.swap                              generated      
basic.target                               static         
bluetooth.target                           static         
cryptsetup-pre.target                      static         
cryptsetup.target                          static         
ctrl-alt-del.target                        disabled       
default.target                             static         
emergency.target                           static         
exit.target                                disabled       
final.target                               static         
friendly-recovery.target                   static         
getty-pre.target                           static         
getty.target                               static         
graphical.target                           static         
halt.target                                disabled       
hibernate.target                           static         
hybrid-sleep.target                        static         
initrd-fs.target                           static         
initrd-root-device.target                  static         
initrd-root-fs.target                      static         
initrd-switch-root.target                  static         
initrd.target                              static         
kexec.target                               disabled       
local-fs-pre.target                        static         
local-fs.target                            static         
multi-user.target                          static         
network-online.target                      static         
network-pre.target                         static         
network.target                             static         
nss-lookup.target                          static         
nss-user-lookup.target                     static         
paths.target                               static         
poweroff.target                            disabled       
printer.target                             static         
reboot.target                              disabled       
remote-cryptsetup.target                   disabled       
remote-fs-pre.target                       static         
remote-fs.target                           enabled        
rescue.target                              static         
rpcbind.target                             static         
runlevel0.target                           disabled       
runlevel1.target                           static         
runlevel2.target                           static         
runlevel3.target                           static         
runlevel4.target                           static         
runlevel5.target                           static         
runlevel6.target                           disabled       
shutdown.target                            static         
sigpwr.target                              static         
sleep.target                               static         
slices.target                              static         
smartcard.target                           static         
sockets.target                             static         
sound.target                               static         
spice-vdagentd.target                      static         
suspend-then-hibernate.target              static         
suspend.target                             static         
swap.target                                static         
sysinit.target                             static         
system-update.target                       static         
time-sync.target                           static         
timers.target                              static         
umount.target                              static         
zfs-import.target                          masked         
zfs.target                                 masked         
anacron.timer                              enabled        
apt-daily-upgrade.timer                    enabled        
apt-daily.timer                            enabled        
fstrim.timer                               enabled        
mdadm-last-resort@.timer                   static         
motd-news.timer                            enabled        
snapd.snap-repair.timer                    enabled        
systemd-tmpfiles-clean.timer               static         
ureadahead-stop.timer                      static         

375 unit files listed.


(Dan Mac Donald) #8

Success at last! :smiley:

I noticed 3.0.3 has been out for almost a month but still hasn’t made it into 16.04 backports so I decided to install the lxd (3.8) snap. For some reason, I ‘lost’ my existing lxd container and lxd configuration (or the lxd snap couldn’t see it?) upon installing the lxd snap and I had to reconfigure lxd by running lxd init before I was able to use lxd-p2c with the lxd 3.8 snap. Is this a known issue with switching from apt packages to the snap?

lxd-p2c 3.8 doesn’t give any indication of how much data needs to be transferred (I’ll open a feature request on github for that) but at least it does finalise the transfer and create a container when all the data has been transferred.

(Stéphane Graber) #9

Hmm, that’s odd, the way to upgrade from deb to snap is to have both installed on your system at the same time (do not uninstall the debs), then run lxd.migrate which will move the data from deb to snap and cleanup the debs. All commands from the snap version of LXD would fail until this step is done.