After Incus upgraded, I cannot start mysql oci container anymore

Incus
21

It works for me. That is,

$ incus version
Client version: 6.5
Server version: 6.5
$ incus launch docker:mysql mysql \
 -c environment.MYSQL_DATABASE=wordpress \
 -c environment.MYSQL_USER=wordpress \
 -c environment.MYSQL_PASSWORD=wordpress \
 -c environment.MYSQL_RANDOM_ROOT_PASSWORD=1
Launching mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 9.0.1 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| performance_schema |
| wordpress          |
+--------------------+
3 rows in set (0.00 sec)

mysql> CONNECT wordpress;
Connection id:    20
Current database: wordpress

mysql> SHOW TABLES;
Empty set (0.00 sec)

mysql> quit
Bye
$
22

Restart your PC/Server while Mysql container is working and see if will start after boot

23

It should not require to shutdown the system. I think incus restart --force has the same exact effect. And in fact, indeed the mysql application container fails predictably to restart.

$ incus restart --force mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password:    
<<<back to shell>>>
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$

Then,

$ incus start mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$

Can you test in a VM whether the docker:mysql in Docker fails, when you incus restart --force the VM? Because if it fails there are well, then it’s a container problem. Perhaps you are supposed to fix manually during an abrupt shutdown?

24

I just tried it and the container started normal without issue inside the VM

25

Since you tried docker:mysql with Docker in a VM and it works when you abruptly restart the VM (incus restart --force docker-in-a-vm) multiple times, but fails when you restart the application container (incus restart --force application-container-mysql), then something unexpected happens in the application container that requires investigation.

How do you investigate? When you start (again) the application container, add --console to see the console messages.

$ incus start mysql --console
To detach from the console, press: <ctrl>+a q
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 9.0.1-1.el9 started.
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Initializing database files
2024-10-01T13:07:43.566534Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-10-01T13:07:43.568025Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 9.0.1) initializing of server in progress as process 97
2024-10-01T13:07:43.576523Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-01T13:07:43.693780Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-01T13:07:44.283930Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-10-01T13:07:45.453367Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-10-01 13:07:45+00:00 [Note] [Entrypoint]: Database files initialized
2024-10-01 13:07:45+00:00 [Note] [Entrypoint]: Starting temporary server
mysqld will log errors to /var/lib/mysql/mysql.err
2024-10-01T13:07:46.034833Z 0 [ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log.
2024-10-01 13:07:46+00:00 [ERROR] [Entrypoint]: Unable to start server.
Error: stat /proc/-1: no such file or directory
$

What you need to do, is enable logging/debugging in MySQL. I did not see and interesting logs. Are they disabled until you enable them?

When you first launch the docker:mysql container, you can get a shell in there with this. Then, looks into /etc/my.cnf on enabling more debugging.

$ incus exec mysql -- /bin/bash
26

The issue occurred only on Incus OCI

@stgraber should I open issue in GitHub?

As I understand the container get 2 filesystems or something similar that execute mysql process twice

27
28

@stgraber thank you so much

29

@stgraber thank you for the fix, I don’t face that issue anymore but I notice something wired when restart my server the MySQL container will start again but without database (like it remove or something similar)

that issue occurred many times from last month till now

30

Check the image you’re using, it may need storage to be attached to specific paths for persistency.

In some cases containers set up a ramdisk when persistent storage isn’t attached, that may be the case here.

31

the database is stored at /var/lib/mysql folder which is part of container volume

32

With OCI, it’s your responsibility to create a persistent data volume and mount it on /var/lib/mysql, just as you would have to do if you were using docker.

From mysql docker container docs:

Important note: There are several ways to store data used by applications that run in Docker containers. We encourage users of the mysql images to familiarize themselves with the options available, including:
…
docker run --name some-mysql -v /my/own/datadir:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag

In the case of incus you probably want to create a custom storage volume.

33

in docker, you can stop the container and it will remain storing the volume and you will need to mount directory only when you want to delete/recreate the container so the data will be stored

34

But you shouldn’t rely on that, because when you next come to upgrade the container (e.g. new version of mysql) you will lose all your data.

It’s pretty poor that docker lets you run stateful containers in a way that will lose their state.

35

I’m worry that behavior not implement in Incus, I think it important to be implemented

36

IMO incus implements the correct behaviour.

It was a really bad idea for docker not to reinitialize containers when they are stopped and restarted; it makes people treat them like “pets” instead of “cattle”.

37

with my respect, I don’t agree with you on that

for now I can solve the issue with

incus config device add mysql mysql_data disk source=/opt/mysql_data path=/var/lib/mysql shift=true

but I don’t think that step is necessary

38

I would take a step back and think about why there is an option to define a separate data volume during instance creation. It is actually best practice to separate or decouple your data from the OS. Usually your data is growing and need much more space but doesn’t need fast storage or you backup / snapshot the data more often etc.