Docker containers can't run on LXC (_NOT LXD_)

nobody · April 24, 2020, 7:05am

Ubuntu 18.04 , lxc 3.0.3 (and ubuntu 20.04, lxc 4.0.1)

sudo lxc-create -t download -n centos7-3 -B lvm --vgname vg1 --fssize 5G --fstype ext4 -- -d centos -r 7 -a amd64
config:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d centos -r 7 -a amd64
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = lvm:/dev/vg1/centos7-3                                                                   
lxc.uts.name = centos7-3                                                                                   
                                                                                                           
# Network configuration                                                                                    
lxc.net.0.type = veth                                                                                      
lxc.net.0.link = br0                                                                                       
lxc.net.0.flags = up                                                                                       
lxc.net.0.name = eth0
lxc.net.0.hwaddr = 00:16:3e:0f:d2:8c
lxc.net.0.ipv4.address = 192.168.51.220/22
lxc.net.0.ipv4.gateway = 192.168.50.250

# Local additions
lxc.start.auto = 1
lxc.cgroup.memory.limit_in_bytes = 1024M
lxc.cgroup.devices.allow = a
lxc.cap.drop =
raw.lxc = 'lxc.apparmor.profile=unconfined'
lxc.mount.auto = proc:rw sys:rw

Inside container docker installed as said here: https://docs.docker.com/engine/install/centos/
Inside container docker run hello-world

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:46: preparing rootfs caused \\\"permission denied\\\"\"": unknown.
ERRO[0001] error waiting for container: context canceled

What should i do else, to run docker’s hello-world inside LXC (NOT LXD)?

nobody · April 24, 2020, 7:05am

With ubuntu 20.04 and lxc 4.0.1 - the same trouble. Hey guys, nobody ran docker inside LXC (NOT LXD)? No way, i don’t believe…

stgraber · April 24, 2020, 12:50pm

LXC and LXD handle apparmor namespacing differently. We test running Docker in LXD daily, we don’t for LXC.

nobody · April 24, 2020, 1:37pm

hey, @stgraber , u a cool guy, but ur last posts makes me sad more and more
Yeah, i googled many examples running docker in LXD, but i can’t use LXD cuz i strongly need to have containers configs as files so i have to use LXC. Well, maybe someone else have good experience with docker and LXC…

stgraber · April 24, 2020, 1:39pm

You can probably replicate what LXD does, but that means manually generating your apparmor profiles, loading them with apparmor namespacing, making sure to be on a compatible filesystem and if the container is unprivileged, then you’ll also need to write a service which handles syscall interception for mknod and setxattr or you’ll have some surprises with some layers.

nobody · April 27, 2020, 5:42am

too hard 4 me :’(

simos · April 27, 2020, 9:05am

Can you expand on that?

nobody · April 27, 2020, 9:15am

For provisioning, create, delete, etc containers i use Puppet. So i have to use config files instead of sql-like configs.

simos · April 27, 2020, 10:11am

You can query the LXD database using SQL statements but I am unaware of using SQL to manage a LXD container. In fact, such a thing would really be hardcore.

There is this addon for LXC on Puppet 3.0, https://forge.puppet.com/sjimenez/lxc/readme
Last updated in 2015, developed on Ubuntu 14.04 for Puppet 3.0.
Is this the one you are currently using?

Someone did something with LXD and Puppet, https://github.com/jhughes01/lxd-puppet
What you need is support for LXD in Puppet.
Can you frame what is needed to get proper support for LXD in Puppet?

nobody · April 27, 2020, 10:33am

No i can’t, cuz it’s very bad practice manage something with Puppet through SQL, it should be configs as files, stateful files. I don’t want to use LXC puppet module from forge, i will write it by myself. I have a lot of experience with that.

https://github.com/jhughes01/lxd-puppet : ehmm, it’s just create container with shebang and install inside some packages

All i need it’s just stateful containers config as it in LXC, that’s all

simos · April 27, 2020, 10:50am

First of all, I am not aware of anyone using SQL to manage LXD.
Do you have any actual evidence of using SQL to manage LXD?

Have you created your own LXC Puppet module? Is this something that can be shared?
This forum caters for LXC as well.

LXD uses YAML for the configuration. Are you familiar with YAML?

nobody · April 27, 2020, 11:38am

github.com

lxc/lxd/blob/master/doc/database.md

# Database

## Introduction
So first of all, why a database?

Rather than keeping the configuration and state within each instance's
directory as is traditionally done by LXC, LXD has an internal database
which stores all of that information. This allows very quick queries
against all instances configuration.


An example is the rather obvious question "what instances are using br0?".
To answer that question without a database, LXD would have to iterate
through every single instance, load and parse its configuration and
then look at what network devices are defined in there.

While that may be quick with a few instance, imagine how many
filesystem access would be required for 2000 instances. Instead with a
database, it's only a matter of accessing the already cached database
with a pretty simple query.

This file has been truncated. show original

Ok, lxc launch images:centos/7/amd64 centos7-5 so where i can find now config of this container?
If i create container with lxc-create -t download -n centos7-4 -B lvm --vgname vg1 --fssize 5G --fstype ext4 -- -d centos -r 7 -a amd64 i am quite sure exactly, there is stateful /var/lib/lxc/centos7-4/config who can be managed with Puppet or something else. So, i don’t need to create containers with YAML like this lxc launch images:centos/7/amd64 centos7-5 < my_config.yaml , i need to have stateful config of that container

P.S. also, see Change containers config storage on LXD

simos · April 27, 2020, 1:15pm

You asked a very specific question and got the corresponding answer.
There was no background as to what you are trying to do.
LXD maintains maintains the configuration in a SQL database,
and your question was phrased about changing the architecture of the whole project.

You, as a user, are not directly accessing the SQL database in any case.
The access is through the LXD REST API.
The lxc client is using this REST API to communicate with the LXD server.
The robust way to support LXD in Puppet is to use directly the REST API, or if you are really inclined, through the lxc client).

nobody · April 27, 2020, 1:40pm

ohh, i never asked “about changing the architecture of the whole project”. I’m just find the ways, but i already saw - there are not. To teach Puppet to use LXD REST API - bad way, bad practice, only one simple and good choice - use files as it in LXC, but I have long understood (many posts ago) - it isn’t possible. So i’m just tried to be clarify and explain you, why i have to use LXC instead of LXD, that’s all…

simos · April 27, 2020, 3:41pm

I could accept that you probably did not intend to ask an architecture change.

You say that using the LXD REST API is “bad way, bad practice”. This is certainly not the case.
It might be more difficult for someone who is familiar with something else and would require to learn more new things, but it is definitely not bad way, bad practice.

You can convert your lxc-create command into the LXD equivalent. See

https://lxd.readthedocs.io/en/latest/storage/

nobody · April 27, 2020, 4:34pm

I think it looks like holywar now. I never didn’t asked about how to: “use LXD REST API”, “how to convert lxc-create to something”, etc. And yes, it’s realy bad way, bad practice use LXD REST API with Puppet cuz Puppet haven’t provider that, he is can only manage config files cuz it’s universal method. Anyway, how this all refers to the main topic where i’m just want to run docker over LXC? I already found out from @stgraber there is no simple way to do that only if use LXD API. Why you repeat it from post to post?

toby63 · April 28, 2020, 6:48pm

simos only wanted to help you.
And your english doesn’t really make it easy to understand what you are about.

And I don’t really understand your problem.
If we take aside the puppet aspect:
You can easily manage everything through lxd.
You can look at the configs via commands, change the configs via commands and you can automatize the use of configs by creating profiles.

Regarding puppet:
How about using an alternative to puppet?
Like ansible: https://docs.ansible.com/ansible/latest/modules/lxd_container_module.html
I know thats not convenient at the first moment, but maybe the pros outweigh the cons.