Incus OCI Docker Containers mapped volume issue

Scott_T · July 26, 2024, 5:29pm

@stgraber Just trying to learn the Docker OCI containers. I can’t seem to make the mapped volume work.

 scott  demo-1  ~/ghost-data  incus config device add ghost hostport8080 proxy connect="tcp:127.0.0.1:2368" listen="tcp:0.0.0.0:8080"
Device hostport8080 added to ghost

 scott  demo-1  ~/ghost-data  incus config device add ghost storage disk source=/home/scott/ghost-data path=/var/lib/ghost
Device storage added to ghost

 scott  demo-1  ~/ghost-data  incus start ghost
Error: Failed to run: /opt/incus/bin/incusd forkstart ghost /var/lib/incus/containers /run/incus/ghost/lxc.conf: exit status 1
Try `incus info --show-log ghost` for more info

 scott  demo-1  ~/ghost-data  incus info --show-log ghost
Name: ghost
Status: STOPPED
Type: container (application)
Architecture: x86_64
Created: 2024/07/26 17:24 UTC
Last Used: 2024/07/26 17:25 UTC

Log:

lxc ghost 20240726172521.536 ERROR    utils - ../src/lxc/utils.c:lxc_mkdir_p:240 - Permission denied - Failed to create directory "/opt/incus/lib/lxc/rootfs/var/lib/ghost/content"
lxc ghost 20240726172521.536 ERROR    conf - ../src/lxc/conf.c:mount_entry_create_dir_file:2376 - Permission denied - Failed to create directory "/opt/incus/lib/lxc/rootfs/var/lib/ghost/content"
lxc ghost 20240726172521.536 ERROR    conf - ../src/lxc/conf.c:lxc_setup:3915 - Failed to setup mount entries
lxc ghost 20240726172521.536 ERROR    start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "ghost"
lxc ghost 20240726172521.536 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc ghost 20240726172521.550 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "eth0" to its initial name "veth93025942"
lxc ghost 20240726172521.550 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc ghost 20240726172521.550 ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "ghost"
lxc ghost 20240726172521.550 WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 20131
lxc 20240726172521.674 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240726172521.674 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

stgraber · July 26, 2024, 7:16pm

Took me a little while to track that one down.

Basically the issue is that you’re passing a disk at /var/lib/ghost which will not be writable by the container. This then causes LXC to fail as the container asked it to create /var/lib/ghost/content and put a tmpfs on that.

If you put shift=true in your disk device, that should all start working fine.

Scott_T · July 26, 2024, 8:47pm

Interesting. I was trying to use the old idmap at one point. Thanks.

Scott_T · July 26, 2024, 9:23pm

So the shift=true worked. However, what does this mean?

Error occurred when starting proxy device: Error: No such file or directory - Failed to safely open namespace file descriptor based on pidfd 3

There is nothing in the log and the proxy defines a folder that does exist.

Scott_T · July 27, 2024, 6:03am

So, now that I understand this I decided to try and get the container to run with a database. Here’s what I am doing:

incus create docker:ghost:5.75-alpine ghost \
  -c environment.url="http://localhost:2368" \
  -c environment.database__client="mysql" \
  -c environment.database__connection__host="ghost-db.incus" \
  -c environment.database__connection__user="ghost" \
  -c environment.database__connection__password__host="somepass" \
  -c environment.database__connection__database="ghostdb"

incus config device add ghost hostport8080 proxy connect="tcp:127.0.0.1:2368" listen="tcp:0.0.0.0:8080" 

mkdir -p ./ghost/content ./ghost/mysql

incus config device add ghost content disk source=/home/scott/ghost/content path=/var/lib/ghost/content shift=true

incus create docker:mysql:8.0 ghost-db \
  -c environment.MYSQL_ROOT_PASSWORD=rootpass \
  -c environment.MYSQL_USER=ghost \
  -c environment.MYSQL_PASSWORD=somepass \
  -c environment.MYSQL_DATABASE=ghostdb

incus config device add ghost-db ghostdb disk source=/home/scott/ghost/mysql path=/var/lib/mysql shift=true  
incus start ghost-db
incus start ghost

The database starts fine. Assuming that the ghost container will talk to the ghost-db container via the incus.ghost-db DNS name we discussed.

Problem is that after working many different iterations of this over the last 12 hours, I am routinely getting the following error when I try to start the ghost container. I am feeling like something in the data structure isn’t working and the error has nothing to do with it.

incus start ghost --console
To detach from the console, press: <ctrl>+a q
[2024-07-27 05:56:40] INFO Ghost is running in production...
[2024-07-27 05:56:40] INFO Your site is now available on http://localhost:2368/
[2024-07-27 05:56:40] INFO Ctrl+C to shut down
[2024-07-27 05:56:40] INFO Ghost server started in 1.782s
[2024-07-27 05:56:40] ERROR The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received an instance of Object

The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received an instance of Object

"Unknown database error"

Error ID:
    500

Error Code: 
    ERR_INVALID_ARG_TYPE

----------------------------------------

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received an instance of Object
    at /var/lib/ghost/versions/5.75.3/node_modules/knex-migrator/lib/database.js:57:19
    at new NodeError (node:internal/errors:405:5)
    at Hash.update (node:internal/crypto/hash:107:11)
    at sha1 (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/auth_41.js:31:8)
    at Object.token [as calculateToken] (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/auth_41.js:57:18)
    at new HandshakeResponse (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/packets/handshake_response.js:28:26)
    at ClientHandshake.sendCredentials (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/commands/client_handshake.js:71:31)
    at ClientHandshake.handshakeInit (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/commands/client_handshake.js:162:12)
    at ClientHandshake.execute (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/commands/command.js:45:22)
    at Connection.handlePacket (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/connection.js:478:34)
    at PacketParser.onPacket (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/connection.js:97:12)
    at PacketParser.executeStart (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/packet_parser.js:75:16)
    at Socket.<anonymous> (/var/lib/ghost/versions/5.75.3/node_modules/mysql2/lib/connection.js:104:25)
    at Socket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)

[2024-07-27 05:56:40] WARN Ghost is shutting down
[2024-07-27 05:56:40] WARN Ghost has shut down
[2024-07-27 05:56:40] WARN Your site is now offline
[2024-07-27 05:56:40] WARN Ghost was running for a few seconds
Error: stat /proc/-1: no such file or directory

It is worthwhile to point out that I tried six different database containers/versions feeling like the error had to do with the interface between the container and the database. The “data” argument must be type string makes no sense. I ended up making everything a string as you can see and it made no difference. This does work in a docker compose file. So, something is not quite right in the OCI incus world I fear.

simos · July 27, 2024, 9:58am

Thanks for going deep into the issue. The error is quite specific (helpful), it’s about an argument that is of the wrong type and it mentions where it can be found. It requires a bit more digging to figure out what went wrong and how to fix.

Scott_T · July 27, 2024, 3:42pm

Yes, notably the error is specific, but is also bogus by its description. Something upstream is causing issues. I’ve been testing a variety of different containers as type OCI and the errors are sometimes pretty obscure.