How to upgrade lxd guests

It is time mention about simple sequential or simultaneous upgrade process on lxd guests. It is adapted to Debian-like operating systems using apt as I am using these on daily basis.

Distribution-specific information.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

The container hypervisor client/daemon version.

$ lxc --version
2.0.10

As you can see the following operations are performed on the Ubuntu 16.04 using the container hypervisor lxc 2.0.10.

Upgrade guests sequentially

Update package index.

for guest in $(sudo lxc list -c ns |  awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
  sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get update; echo;";
done | tee /tmp/guest_update.log

Inspect the /tmp/guest_update.log log file.

$ tail /tmp/guest_update.log
[...]
dokuwiki-tests
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists...
[...]

This step is purely optional, but you can list packages to be upgraded.

for guest in $(sudo lxc list -c ns |  awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
  sudo lxc exec $guest -- /bin/bash -c "echo $guest; apt-get --dry-run dist-upgrade | grep ^Inst | cut -d\  -f2 | sort | tr '\n' '\ '; echo;echo;";
done | tee /tmp/guest_upgrade_list.log

Inspect the /tmp/guest_ugrade_list.log log file.

$ tail /tmp/guest_upgrade_list.log
[...]
dokuwiki-tests
apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools cloud-init
cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils iproute2
isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0 libapt-pkg5.0 libblkid1
libdrm2 libfdisk1 libkmod2 liblxc1 libmount1 libpam-systemd libphp7.0-embed
libsmartcols1 libssl1.0.0 libsystemd0 libudev1 libuuid1 linux-libc-dev lxc-common
lxcfs lxd lxd-client mount open-iscsi openssl overlayroot php7.0-common php7.0-intl
php7.0-phpdbg php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade
python3-software-properties python3-update-manager snap-confine snapd
software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher
ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core
util-linux uuid-runtime vlan
[...]

Perform upgrade process.

for guest in $(sudo lxc list -c ns |  awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}'); do
  sudo lxc exec $guest -- /bin/bash -c "export DEBIAN_FRONTEND=noninteractive; echo $guest; apt-get upgrade -y; echo;";
done | tee /tmp/guest_upgrade.log

Inspect the /tmp/guest_upgrade.log log file.

$ tail /tmp/guest_upgrade.log
[...]
dokuwiki-tests
[...]
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following package was automatically installed and is no longer required:
  snap-confine
Use 'apt autoremove' to remove it.
The following packages will be upgraded:
  apt apt-transport-https apt-utils base-files binutils bsdutils btrfs-tools
  cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf coreutils
  iproute2 isc-dhcp-client isc-dhcp-common kmod less libapt-inst2.0
  libapt-pkg5.0 libblkid1 libdrm2 libfdisk1 libkmod2 liblxc1 libmount1
  libpam-systemd libphp7.0-embed libsmartcols1 libssl1.0.0 libsystemd0
  libudev1 libuuid1 linux-libc-dev lxc-common lxcfs lxd lxd-client mount
  open-iscsi openssl overlayroot php7.0-common php7.0-intl php7.0-phpdbg
  php7.0-readline php7.0-sqlite3 php7.0-xml python3-distupgrade
  python3-software-properties python3-update-manager snap-confine snapd
  software-properties-common sudo systemd systemd-sysv ubuntu-core-launcher
  ubuntu-release-upgrader-core udev unattended-upgrades update-manager-core
  util-linux uuid-runtime vlan
63 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 36.6 MB of archives.
After this operation, 1794 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 base-files amd64 9.4ubuntu4.5 [68.4 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 bsdutils amd64 1:2.27.1-6ubuntu3.3 [52.0 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 coreutils amd64 8.25-2ubuntu3~16.04 [1174 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 util-linux amd64 2.27.1-6ubuntu3.3 [849 kB]
[...]
Hint: Some lines were ellipsized, use -l to show in full.
Setting up snapd (2.25) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Setting up ubuntu-core-launcher (2.25) ...
Setting up snap-confine (2.25) ...
Setting up iproute2 (4.3.0-1ubuntu3.16.04.1) ...
Setting up isc-dhcp-client (4.3.3-5ubuntu12.7) ...
Setting up isc-dhcp-common (4.3.3-5ubuntu12.7) ...
[...]

You can use additinal conditions inside awk program line to match only specific guests.

Upgrade guests in parallel

You can speed up a whole process by using the amazing parallel utility.

$ sudo apt-get install parallel

Update package index simultaneously on four guests.

sudo lxc list -c ns |  \
  awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \
  tr -d ' ' | \
  parallel  --jobs 4 --progress --no-run-if-empty \
    --results /tmp/parallel_update/ --joblog /tmp/parallel_update_status.log \
      sudo lxc exec {1} -- apt-get update

Perform upgrade process simultaneously on four guests.

export DEBIAN_FRONTEND="noninteractive"; \
  sudo lxc list -c ns |  \
  awk 'BEGIN{FS="|"} NR%2==0 && NR>2 && $3 ~ "RUNNING" {print $2}' | \
  tr -d ' ' | \
  parallel  --jobs 4 --progress --no-run-if-empty \
    --results /tmp/parallel_upgrade/ --joblog /tmp/parallel_upgrade_status.log \
    --env DEBIAN_FRONTEND \
      sudo lxc exec {1} -- apt-get upgrade -y

List of the executed jobs will be stored in /tmp/parallel_update_status.log and /tmp/parallel_update_status.log log files.

$ cat /tmp/parallel_update_status.log
Seq     Host    Starttime       JobRuntime      Send    Receive Exitval Signal  Command
1       :       1501795764.764       4.626      0       0       0       0       sudo lxc exec dokuwiki-tests -- apt-get update
4       :       1501795764.793       4.599      0       0       0       0       sudo lxc exec jira-tests -- apt-get update
5       :       1501795769.392       4.840      0       0       0       0       sudo lxc exec nextcloud-tests -- apt-get update
6       :       1501795769.395       4.841      0       0       0       0       sudo lxc exec wallabag-tests -- apt-get update
7       :       1501795774.236       4.055      0       0       0       0       sudo lxc exec wekan-tests -- apt-get update
2       :       1501795764.767      20.569      0       0       0       0       sudo lxc exec draw-tests -- apt-get update
3       :       1501795764.775      20.561      0       0       0       0       sudo lxc exec foswiki-tests -- apt-get update

Detailed logs will be located in the specified /tmp/parallel_update/ and /tmp/parallel_upgrade/ directories.

$ cat /tmp/parallel_update/1/jira-tests/std{err,out}
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 1s (239 kB/s)
Reading package lists...
Milosz Galazka's Picture

About Milosz Galazka

Milosz is a Linux Foundation Certified Engineer working for a successful Polish company as a system administrator and a long time supporter of Free Software Foundation and Debian operating system.

Gdansk, Poland https://sleeplessbeastie.eu