How to deal with non-obvious lack of free space when using lxd snap

Inspect and solve non-obvious lack of free space when using lxd snap.

Display LXD snap version.

$ snap list lxd
Name  Version  Rev    Tracking      Publisher   Notes
 lxd   4.0.4    18150  4.0/stable/…  canonical✓  -

The symptoms

Identify the problem during the export operation.

$ sudo lxc export laboratory laboratory.tar.gz
Error: Create backup: Backup create: Error adding "/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/laboratory/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar" as "backup/snapshots/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar" to tarball: Failed to copy file content "/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/laboratory/dns/rootfs/var/opt/gitlab/backups/1596300891_2020_08_01_13.2.1_gitlab_backup.tar": io: read/write on closed pipe

Identify the problem during the import operation.

$ sudo lxc import laboratory.tar.gz
Importing instance: 39% (43.36MB/s)Error: write /var/snap/lxd/common/lxd/backups/lxd_backup_084016629: no space left on device

The issue

The issue is that the /var/snap/lxd/common/lxd/backups directory that is used during export/import operations is by default on the root filesystem.

$ sudo df -h /var/snap/lxd/common/lxd/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0         46G   36G  7.8G  83% /

This is an issue when you want to perform the mentioned operations on virtual machines that use more disk space than a small amount available on the root filesystem.

The solution

The solution is to create a new pool. I will stick with ZFS but use LVM or whatever is available.

$ sudo zfs create tank/lxd_backup

Ensure that it will mount at boot. Use a proper solution or a bind mount.

$ echo "/tank/lxd_backup /var/snap/lxd/common/lxd/backups       none    bind,nofail      0 0" | sudo tee -a /etc/fstab
/tank/lxd_backup /var/snap/lxd/common/lxd/backups       none    bind,nofail      0 0

Mount it afterward.

$ mount /var/snap/lxd/common/lxd/backups

Restart lxd service, to ensure that it will take advantage of the new directory.

$ sudo snap restart lxd

Problem solved.