How to to fix container which cannot be started

The short story of my experience with container is marked for removal and cannot be started error which was caused by driver "overlay" failed to remove root filesystem for ...: remove /var/lib/docker/overlay/.../merged: device or resource busy.

I tried to stop and start systemd service that was using docker internally.

$ sudo systemctl stop docker-compose-grafana.service
$ sudo systemctl start docker-compose-grafana.service

It didn't go well...

$ sudo systemctl status docker-compose-grafana.service
● docker-compose-grafana.service - Grafana
   Loaded: loaded (/etc/systemd/system/docker-compose-grafana.service; disabled; vendor preset: disabled)
   Active: deactivating (stop-post) (Result: exit-code) since sun 2020-03-01 09:08:33 CET; 1s ago
  Process: 19459 ExecStop=/usr/bin/env docker-compose stop (code=exited, status=0/SUCCESS)
  Process: 19378 ExecStart=/usr/bin/env docker-compose up (code=exited, status=1/FAILURE)
 Main PID: 19378 (code=exited, status=1/FAILURE);         : 19472 (docker-compose)
   Memory: 17.3M
   CGroup: /system.slice/docker-compose-grafana.service
           └─control
             └─19472 /usr/bin/python2 /usr/bin/docker-compose down

mar 01 02:08:33 analytics systemd[1]: Starting Grafana...
mar 01 02:08:33 analytics env[19378]: Creating network "grafanaanalytics_default" with the default driver
mar 01 02:08:33 analytics env[19378]: Starting grafana_analytics ...
mar 01 02:08:33 analytics env[19378]: Starting grafana_analytics
mar 01 02:08:33 analytics env[19378]: [45B blob data]
mar 01 02:08:33 analytics env[19378]: ERROR: for grafana_analytics  Cannot start service grafana: container is marked for removal and cannot be started
mar 01 02:08:33 analytics env[19378]: ERROR: for grafana  Cannot start service grafana: container is marked for removal and cannot be started
mar 01 02:08:33 analytics env[19378]: Encountered errors while bringing up the project.
mar 01 02:08:33 analytics systemd[1]: docker-compose-grafana.service: main process exited, code=exited, status=1/FAILURE
mar 01 02:08:34 analytics env[19472]: Removing grafana_analytics ...

This service failed miserably.

$ sudo systemctl status  docker-compose-grafana.service
● docker-compose-grafana.service - Grafana
   Loaded: loaded (/etc/systemd/system/docker-compose-grafana.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

mar 01 02:08:33 analytics env[19378]: ERROR: for grafana  Cannot start service grafana: container is marked for removal and cannot be started
mar 01 02:08:33 analytics env[19378]: Encountered errors while bringing up the project.
mar 01 02:08:33 analytics systemd[1]: docker-compose-grafana.service: main process exited, code=exited, status=1/FAILURE
mar 01 02:08:34 analytics env[19472]: Removing grafana_analytics ...
mar 01 02:08:35 analytics env[19472]: [45B blob data]
mar 01 02:08:35 analytics env[19472]: ERROR: for grafana_analytics  driver "overlay" failed to remove root filesystem for 6752b12b582b80166e31510c6947bb337ed19de689122598574f75a2aa9efbb1: remo... resource busy
mar 01 02:08:35 analytics env[19472]: Removing network grafanaanalytics_default
mar 01 02:08:35 analytics systemd[1]: Unit docker-compose-grafana.service entered failed state.
mar 01 02:08:35 analytics systemd[1]: docker-compose-grafana.service failed.
mar 01 02:08:48 analytics systemd[1]: Stopped Grafana.
Hint: Some lines were ellipsized, use -l to show in full.

As expected the same error can be seen when using docker-compose directly.

$ docker-compose down
Removing grafana_analytics ... error

ERROR: for grafana_analytics  driver "overlay" failed to remove root filesystem for 6752b12b582b80166e31510c6947bb337ed19de689122598574f75a2aa9efbb1: remove /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged: device or resource busy
Removing network grafanaanalytics_default
WARNING: Network grafanaanalytics_default not found.
$ docker-compose up -d
Creating network "grafanaanalytics_default" with the default driver
Starting grafana_analytics ...
Starting grafana_analytics ... error

ERROR: for grafana_analytics  Cannot start service grafana: container is marked for removal and cannot be started

ERROR: for grafana  Cannot start service grafana: container is marked for removal and cannot be started
ERROR: Encountered errors while bringing up the project.
$ docker-compose ps
    Name       Command   State    Ports
---------------------------------------
grafana_analytics   /run.sh   Exit 1
$ docker-compose rm -f
Going to remove grafana_analytics
Removing grafana_analytics ... error

ERROR: for grafana_analytics  driver "overlay" failed to remove root filesystem for 6752b12b582b80166e31510c6947bb337ed19de689122598574f75a2aa9efbb1: remove /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged: device or resource busy

I tried to restart docker service.

$ sudo systemctl restart docker.service

It didn't work.

$ rm -rf /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged/
rm: cannot remove '/var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged/': Device or resource busy

So I checked wchich process was using this mount.

$ grep 489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26 /proc/*/mountinfo
/proc/18295/mountinfo:340 233 0:74 / /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged rw,relatime shared:157 - overlay overlay rw,lowerdir=/var/lib/docker/overlay/df$975bec0c7cb27e9770e30d574a23aff530f6e331f064bb4faf025aa91a4cc/root,upperdir=/var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/upper,workdir=/var/lib/docker/overlay/c489302a4$ea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/work
/proc/19019/mountinfo:340 233 0:74 / /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged rw,relatime shared:157 - overlay overlay rw,lowerdir=/var/lib/docker/overlay/df$975bec0c7cb27e9770e30d574a23aff530f6e331f064bb4faf025aa91a4cc/root,upperdir=/var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/upper,workdir=/var/lib/docker/overlay/c489302a4$ea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/work
/proc/19020/mountinfo:340 233 0:74 / /var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/merged rw,relatime shared:157 - overlay overlay rw,lowerdir=/var/lib/docker/overlay/df$975bec0c7cb27e9770e30d574a23aff530f6e331f064bb4faf025aa91a4cc/root,upperdir=/var/lib/docker/overlay/c489302a4bea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/upper,workdir=/var/lib/docker/overlay/c489302a4$ea143469f8c29915354ee505903bae2a3fee6fd40a890b8cdb9d26/work

It was nginx on the host.

$ ps -p 18295 19019 19020
    PID TTY      STAT   TIME COMMAND
  18295 ?        Ss     0:00 nginx: master process /usr/sbin/nginx
  19019 ?        S<     0:19 nginx: worker process
  19020 ?        S<     0:16 nginx: worker process

I restarted nginx to free the process's mount namespace.

$ sudo systemctl restart nginx

Now I can start service without any issues.

$ docker-compose rm
Going to remove grafana_analytics
Are you sure? [yN] y
Removing grafana_analytics ... done
$ sudo systemctl start docker-compose-grafana.service