Magnus May 23, 2022

Introduction

Some of you might have encountered the issue that it takes some time (ca 10 seconds) before a container stops after you issue docker stop or docker-compose stop. This is happening either because the script you use in your ENTRYPOINT is trap:ing the wrong kill signal, or you are not properly dealing with signals your init script/command. I always use a bash script, so this post is mainly focusing on that. If you use other scripting languages for your ENTRYPOINT script, you might have to solve this issue differently.

Stopping a container is dependent on a few things:

The ENTRYPOINT in your Dockerfile, and how it behaves when receiving a signal
The STOPSIGNAL in your Dockerfile (default: SIGTERM, but this is not always used in all base containers, see php:8.0-fpm and nginx).
The stop_signal in your docker-compose.yml file

Bonus items:

The stop_grace_period in your docker-compose.yml file

How docker stops your containers

The default behavior is that docker sends the SIGTERM signal to the entry point process (normally with process id 1 in the container). If the container is still running after 10 seconds, docker stop and docker-compose down will send the SIGKILL signal, which will remove the process from the OS scheduler. This can be overridden:

In the Dockerfile
In the docker-compose.yml
In the docker stop command

For example, the php-fpm and nginx containers does not use the default SIGTERM as stop signal.

$ docker inspect nginx:latest | jq '.[].Config.StopSignal'
"SIGQUIT"
$ docker inspect php:7.4-fpm | jq '.[].Config.StopSignal'
"SIGQUIT"

In the case of nginx container, it has to do with how the nginx process itself deals with signals. SIGTERM is used to quickly stop an nginx process, which might cut off a request and leave some garbage behind. The signal SIGQUIT is used instead, which within the nginx process is interpreted as “we will slowly shut down, and the users will not be affected”.

Start/stop containers

I have a very simple example, to show a few ways to test this.

#--- create the init.sh script
cat<<EOT > init.sh
#!/bin/bash

echo "This container will not stop immediately after SIGTERM"

sleep infinity
EOT

chmod 755 init.sh

#--- create the Dockerfile
cat<<EOT > Dockerfile
from php:8.0-fpm

COPY . /

ENTRYPOINT ["/init.sh"]
EOT

#--- build the container
docker build -t stop-container-demo:latest .

#--- run the container (in one terminal window)
docker run --name=stop-demo --rm -it stop-container-demo

#--- stop the container (in a different terminal window), this will take 10 seconds
docker stop stop-demo

The container will stop after 10 seconds, as we have multiple issues with the init.sh script.

php:8.0-fpm does not use the default SIGTERM signal (it uses SIGQUIT), so the init.sh script does not receive the expected signal
We don’t trap the signal, so that we can exit the script
We sleep infinity in a way so that our bash script can’t trap the signal

We have to add a trap to the init.sh script, and we have to send sleep to the background:

cat<<EOT > init.sh
#!/bin/bash

#--- we add a function to exit nicely (perhaps kill a few processes and remove some temp files)
function exit_container_SIGTERM(){
  echo "Caught SIGTERM"
  exit 0
}

#--- trap the SIGTERM signal
trap exit_container_SIGTERM SIGTERM 

echo "This container will stop immediately after SIGTERM"

sleep infinity &
wait
EOT

chmod 755 init.sh

Now you can test again:

#--- build the container
docker build -t stop-container-demo:latest .

#--- run the container (in one terminal window)
docker run --name=stop-demo --rm -it stop-container-demo

#--- stop the container (in a different terminal window), this will take 10 seconds
docker stop stop-demo

Well, our container still did not exit immediately. This is because the php:8.0-fpm container is built to use SIGQUIT instead of SIGTERM. We must either build our container and select the SIGTERM as the stop signal, or we have to rewrite our init.sh script to trap the SIGQUIT signal.

In your Dockerfile, you select which signal to use with the STOPSIGNAL keyword:

cat<<EOT > Dockerfile
from php:8.0-fpm

COPY . /

#--- override the SIGQUIT used in php:8.0-fpm
STOPSIGNAL SIGTERM

ENTRYPOINT ["/init.sh"]
EOT

#--- build the container
docker build -t stop-container-demo:latest .

#--- run the container (in one terminal window)
docker run --name=stop-demo --rm -it stop-container-demo

#--- stop the container (in a different terminal window), the container will terminate immediately. 
docker stop stop-demo

Bash - sleep, wait, trap

This section is basically repeating what is already written above to some extent. Your ENTRYPOINT script will not react on a signal if you don’t take care of how you sleep at the end of the script (bash). Many write their entry point script such, that they are using sleep infinity to make the script “blocking forever”. If this is not done properly, your script will not catch any signals sent to it, even if you have a trap in your bash script.

This does not work:

function exit_script(){
  echo "Caught SIGTERM"
  exit 0
}


trap exit_script SIGTERM

#--- my init.sh script
./start/my/program &
sleep infinity

This works:

function exit_script(){
  echo "Caught SIGTERM"
  exit 0
}

trap exit_script SIGTERM

#--- my init.sh script
./start/my/program &

#--- send sleep into the background, then wait for it.
sleep infinity &
#--- "wait" will wait until the command you sent to the background terminates, which will be never.
#--- "wait" is a bash built-in, so bash can now handle the signals sent by "docker stop"
wait

docker-compose.yml

You can override the STOPSIGNAL in a container by using the attribute stop_signal in your docker-compose.yml file:

version: '2'
services:
    demo:
      stop_signal: SIGTERM
      image: stop-container-demo:latest

Navigation

Docker: stop containers with the proper signals

Introduction

How docker stops your containers

Start/stop containers

Bash - sleep, wait, trap

docker-compose.yml

References

KMG Group

Company

Products