Hi,
Couple of older (IBM 8cpu/15GB ram) test servers experience swap full usage, running 18.04 LTS. Each server holding 11 containers 14.04 LTS and 18.04 LTS
Ram usage is around 5GB. SWAP 3.2/4GB vm.swappiness=30. This happens after full container export every night. During day I reboot containers and SWAP usage goes down to 200MB and holding until container export.
Is this normal?
Doing some tests on that, so far memory usage is steady at 35MB or so during the export process, but that’s during the copy of the container and its compression, there may still be something wrong during the download stage which causes the tarball to be loaded in memory.
What memory counters were you tracking for LXD? Anything other than RSS isn’t very relevant for this.
All what I can pull from nagios. On host and guest. Host example
Just did a full export of a 9GB large container and LXD stayed at a constant < 40MB the whole time.
What LXD version was that with? (3.10 has a bunch of fixes for such issues)
And if that was with 3.10, exactly what flags did you pass to lxc export, does your container have snapshots and what storage driver is used for its storage pool?
dir storage backed
3.0 LTS, cant move to snap
echo “Backing up $CONT… Please wait…” >> $LOG
/usr/bin/lxc snapshot $CONT backup_$CONT
echo “Snapshost done…” >> $LOG
date >> $LOG
/usr/bin/lxc publish $CONT/backup_$CONT --alias lxd-image-backup-$CONT
echo “Container published…” >> $LOG
date >> $LOG
/usr/bin/lxc image export lxd-image-backup-$CONT $DIRECTORY2/$CONT-$NOW
echo “Image exported…” >> $LOG
date >> $LOG
echo “Size:” >> $LOG
ls -lahS $DIRECTORY2/$CONT-$NOW.tar.gz >> $LOG
/usr/bin/lxc image delete lxd-image-backup-$CONT
/usr/bin/lxc delete local:$CONT/backup_$CONT
echo “Cleanup done…” >>$LOG
Maybe its because I reached limit of the server, this started after 10 th container on server.
Oh, okay, so that’s not using lxc export then, that’s using snapshot+image+publish.
This also means that XZ will be used for compression and may be quite memory intensive explaining a push for inactive memory over to SWAP.
Did you try changing images.compression_algorithm to none or to a less intensive compression algorithm?
no, I am new to LXD so I try to stay on defaults
Ok, so yeah, I’d say try exporting with none see if disabling compression does the trick, if it does, then maybe try gzip as a less aggressive compression algorithm. gzip is the default for container exports in LXD releases that have that feature as those backups aren’t usually shared on the public internet so having them a bit bigger isn’t as much of an issue as for images.
ok, thank you, I will give it a try.
I’ve also just tested current LXD to see if we had any memory issue during snapshot+publish+export and I’ve only seen memory usage go up by about 5MB during the publication and memory usage has since gone down (so just temporary allocation, not a leak).
