LXC Containers are created in 16.04 . Now after OS upgrade to 18.04 this version the lxc containers are not starting not stopping.
File Systems giving read only error
Filesystem being read-only is likely a hardware failure or some filesystem corruption that’s unrelated to LXC. LXC startup failures after 16.04 to 18.04 upgrades may be caused by incompatible config options which can be fixed with lxc-update-config.
Can you show dmesg?
Thanks for your quick response Stephane.
The output of last 20 lines of dmesg is as follows
[25607954.672435] [UFW BLOCK] IN=eno3 OUT= MAC=ac:1f:6b:65:fd:3a:84:3d:c6:3d:4c:81:08:00 SRC=185.176.26.104 DST=147.135.130.142 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=9115 PROTO=TCP SPT=40872 DPT=222 WINDOW=1024 RES=0x00 SYN URGP=0
[25607965.355426] [UFW BLOCK] IN=eno3 OUT= MAC=ac:1f:6b:65:fd:3a:84:3d:c6:3d:4c:81:08:00 SRC=46.3.96.67 DST=147.135.130.142 LEN=40 TOS=0x00 PREC=0x00 TTL=244 ID=38176 PROTO=TCP SPT=46298 DPT=9813 WINDOW=1024 RES=0x00 SYN URGP=0
[25607970.804539] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25607970.805198] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25607970.805821] BTRFS info (device md3): no csum found for inode 160265 start 0
[25607970.823170] BTRFS warning (device md3): csum failed root 594 ino 160265 off 0 csum 0x33113a41 expected csum 0x00000000 mirror 1
[25607993.468916] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25607993.469581] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25607993.470282] BTRFS info (device md3): no csum found for inode 160265 start 0
[25607993.481749] BTRFS warning (device md3): csum failed root 594 ino 160265 off 0 csum 0x33113a41 expected csum 0x00000000 mirror 1
[25608002.099402] [UFW BLOCK] IN=eno3 OUT= MAC=ac:1f:6b:65:fd:3a:84:3d:c6:3d:4c:81:08:00 SRC=162.243.151.176 DST=147.135.130.142 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=39666 DPT=993 WINDOW=65535 RES=0x00 SYN URGP=0
[25608004.336770] BTRFS error (device md3): parent transid verify failed on 236271091712 wanted 1451217 found 1450823
[25608004.338373] BTRFS error (device md3): parent transid verify failed on 236271091712 wanted 1451217 found 1450823
[25608004.342181] BTRFS error (device md3): parent transid verify failed on 236271091712 wanted 1451217 found 1450823
[25608004.342751] BTRFS error (device md3): parent transid verify failed on 236271091712 wanted 1451217 found 1450823
[25608009.882572] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25608009.883234] BTRFS error (device md3): parent transid verify failed on 236270829568 wanted 1451217 found 1450822
[25608009.883858] BTRFS info (device md3): no csum found for inode 160265 start 0
[25608009.898742] BTRFS warning (device md3): csum failed root 594 ino 160265 off 0 csum 0x33113a41 expected csum 0x00000000 mirror 1
[25608034.915855] [UFW BLOCK] IN=eno3 OUT= MAC=ac:1f:6b:65:fd:3a:84:3d:c6:3d:4c:81:08:00 SRC=209.17.97.114 DST=147.135.130.142 LEN=44 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=58754 DPT=3000 WINDOW=65535 RES=0x00 SYN URGP=0
Right, your filesystem is in bad shape, you’ll want to check/fix that before doing much anything else on your system.
Ohh.
Is this recoverable?
Are my lxc containers recoverable?
As you said earlier is this a hardware failure?
Can you please help how can I fix this.
This was production environment and I am really in trouble
The btrfs filesystem on top of /dev/md3 is showing corruption.
This may be caused by physical drive failures or it may just be a software bug in the kernel/btrfs that caused corruption.
You should run btrfs scrub start on that filesytem, possibly btrfsck and probably reboot after those to see if things stabilize. You should also attempt to backup whatever you can in case you’re dealing with a drive failure as those may be the early signs of a much worse problem.
None of this is really related to LXD, so you’ll have more luck looking for guides/howtos around btrfs only and first only dealing with fixing btrfs as much as possible. Once you know what data is accessible and functional, we can talk about how to get LXD back online.
Thanks Stephane for your guidance.
I was roaming about lxc whereas the problem was elsewhere.
Just one query,
When I am going to take backup from /var/lib/lxc/containers/mycontainer/rootfs somewhere it is allowing me to take backup whereas somewhere it is showing me input output error and not accessible.
How can I take lxc file system backup?
Those errors happen because of the broken filesystem. Try to make a tarball of that data and for now, ignore all those errors. Until you’ve attempted filesystem repairs, you can’t really know if that data will be lost or not.
Thanks Stephane for guiding so much.
When I am trying to make tarball by using the command
tar -czvf folder.tar.gz directory it is giving me read only error.
Wondering how can I take backup?
Hi Stephane,
We are following the file reparing procedure and shall reboot the cloud server.Thankfully we had backup in our local storage.
After that I shall get back to you regarding lxc.
Your assistance was really helpful.
Thanks
Hello Stephane
Thanks for your advice. We are working on it. We may come back with our observations and seek your guidance, later if the issue remains unsolved.
Rgds
Shamik
VP - FinlabsIndia , Kolkata
Hello Stephane
Please find below the output . While to run btrfs check /dev/md3, it says the device is mounted and to use with --force option. I want to do that, but need your opinion. Rgds Shamik
root@# btrfs scrub start /dev/md3
scrub started on /dev/md3, fsid 1ab9b6cd-ccdc-422b-b185-dfe1c5f201db (pid=19507)
root# btrfs scrub status /dev/md3
scrub status for 1ab9b6cd-ccdc-422b-b185-dfe1c5f201db
scrub started at Mon Jul 1 08:16:59 2019 and was aborted after 00:00:00
total bytes scrubbed: 0.00B with 0 errors
Rgds Shamik
Thanks a ton Stephane.
The btrfs repair worked and after reboot We can log in to individual containers.
Start stop working perfectly.
Thanks again.
Our domain is finexa.in
But testing.finexa.in(testing container) is not accessible from the browser
Now LXC containers are still on 16.04 while root container is on 18.04.
Do I need to update the config file?