After Incus upgraded, I cannot start mysql oci container anymore

It works for me. That is,

$ incus version
Client version: 6.5
Server version: 6.5
$ incus launch docker:mysql mysql \
 -c environment.MYSQL_DATABASE=wordpress \
 -c environment.MYSQL_USER=wordpress \
 -c environment.MYSQL_PASSWORD=wordpress \
 -c environment.MYSQL_RANDOM_ROOT_PASSWORD=1
Launching mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 9.0.1 MySQL Community Server - GPL

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| performance_schema |
| wordpress          |
+--------------------+
3 rows in set (0.00 sec)

mysql> CONNECT wordpress;
Connection id:    20
Current database: wordpress

mysql> SHOW TABLES;
Empty set (0.00 sec)

mysql> quit
Bye
$ 

Restart your PC/Server while Mysql container is working and see if will start after boot

It should not require to shutdown the system. I think incus restart --force has the same exact effect. And in fact, indeed the mysql application container fails predictably to restart.

$ incus restart --force mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password:    
<<<back to shell>>>
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$ 

Then,

$ incus start mysql
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
$ incus exec mysql -- mysql -h localhost -u wordpress -p
Error: Instance is not running
$

Can you test in a VM whether the docker:mysql in Docker fails, when you incus restart --force the VM? Because if it fails there are well, then it’s a container problem. Perhaps you are supposed to fix manually during an abrupt shutdown?

I just tried it and the container started normal without issue inside the VM

Since you tried docker:mysql with Docker in a VM and it works when you abruptly restart the VM (incus restart --force docker-in-a-vm) multiple times, but fails when you restart the application container (incus restart --force application-container-mysql), then something unexpected happens in the application container that requires investigation.

How do you investigate? When you start (again) the application container, add --console to see the console messages.

$ incus start mysql --console
To detach from the console, press: <ctrl>+a q
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 9.0.1-1.el9 started.
2024-10-01 13:07:43+00:00 [Note] [Entrypoint]: Initializing database files
2024-10-01T13:07:43.566534Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-10-01T13:07:43.568025Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 9.0.1) initializing of server in progress as process 97
2024-10-01T13:07:43.576523Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-01T13:07:43.693780Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-01T13:07:44.283930Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-10-01T13:07:45.453367Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-10-01 13:07:45+00:00 [Note] [Entrypoint]: Database files initialized
2024-10-01 13:07:45+00:00 [Note] [Entrypoint]: Starting temporary server
mysqld will log errors to /var/lib/mysql/mysql.err
2024-10-01T13:07:46.034833Z 0 [ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log.
2024-10-01 13:07:46+00:00 [ERROR] [Entrypoint]: Unable to start server.
Error: stat /proc/-1: no such file or directory
$ 

What you need to do, is enable logging/debugging in MySQL. I did not see and interesting logs. Are they disabled until you enable them?

When you first launch the docker:mysql container, you can get a shell in there with this. Then, looks into /etc/my.cnf on enabling more debugging.

$ incus exec mysql -- /bin/bash

The issue occurred only on Incus OCI

@stgraber should I open issue in GitHub?

As I understand the container get 2 filesystems or something similar that execute mysql process twice

@stgraber thank you so much