AI-translated. Read the original in KoreanEmbedded slides, videos and images remain in their original language.
Breaking Down asyncio, Part 3: Handling SIGTERM (CTRL+C) Properly
Software Engineer
A server using asyncio needs to support graceful shutdown. Because work is registered as tasks on an event loop, shutting down without care can discard requests from users or other servers.
The previous posts explained how asyncio's core components work. This time, we will apply that knowledge by learning about signals and adding signal handlers to the event loop.
- Breaking Down asyncio, Part 1: Coroutines and the Event Loop
- Breaking Down asyncio, Part 2: Working with Futures
- (This post) Breaking Down asyncio, Part 3: Handling SIGTERM (CTRL+C) Properly
- (WIP) Breaking Down asyncio, Part 4: Graceful Shutdown in APScheduler
What is a signal?
In Unix, a signal is a form of inter-process communication (IPC) that sends a particular meaning to a process. Signals come in different types according to that meaning. On a POSIX-compatible OS, including Linux and macOS, you can run kill -l in a terminal to see the signals supported by your OS.
A signal can occur at any time while a process is running. When it does, the process pauses its execution to handle the signal. At any time is the key point: it may arrive while a newly started server is checking a database's health, while processing a user request, or even while handling another signal.
Trying signals locally
You can test this with the kill command. Running kill -KILL <PID> sends SIGKILL to the process whose ID is <PID>. Replace KILL with a name listed by kill -l to send that signal.
Some signals can also be sent through a TTY without using kill. For example, CTRL + C, commonly used to stop a process in the terminal, sends SIGINT.
Common signals
The Linux signal manual page lists many signals. If I had to choose three that server engineers should care about, they would be SIGTERM, SIGKILL, and SIGINT.
SIGTERM
Think of this as asking a process to terminate. The process should finish its work and exit normally, making it suitable for graceful shutdown. Both docker stop and kubectl delete pod send SIGTERM immediately.
SIGKILL
This terminates a process immediately, without giving it time to finish. docker kill sends SIGKILL to the container immediately. Both docker stop and kubectl delete pod initially send SIGTERM, but if the process does not exit within the time configured through the CLI, they send SIGKILL to force termination.
SIGINT
As explained above, SIGINT is sent when you press Ctrl + C to stop a process in a terminal. Docker and Kubernetes do not use it for this purpose, but it is common during local development. Gunicorn also uses it for an immediate shutdown.
Signal handlers
A signal handler is code that runs when a signal arrives. The OS provides a default handler for each signal. For example, SIGKILL's default action terminates the process. A process can define its own handler, called a user-defined signal handler. The OS runs that handler if one exists, otherwise it uses the default. There are two exceptions: SIGKILL and SIGSTOP always use their default actions.
But I never added a handler to my server
Most server frameworks have their own signal handlers. Gunicorn provides basic handling, and frameworks such as FastAPI and Django support running handlers when a server shuts down. They abstract the signals into framework features. If your server does not use a web framework, though, you need to consider signals yourself.
Signals in Python
Python supported signal handling through the signal module before asyncio existed. When SIGINT (Ctrl + C) occurs, Python raises KeyboardInterrupt. Because this is a subclass of BaseException, you can catch and handle it with exception handling.
asyncio added loop.add_signal_handler. When using asyncio, use this instead of signal.signal(). Looking at the implementation, it ultimately calls signal.signal(), but also includes a mechanism to ensure the event loop wakes up.
Why not just use KeyboardInterrupt?
⚠
KeyboardInterruptresponds only to Ctrl + C (SIGINT), used to stop a program from the terminal. It has nothing to do with SIGTERM or SIGKILL. We use it here to illustrate why handlers are necessary and how signals behave.
import asyncio
loop = asyncio.new_event_loop()
try:
loop.run_until_complete(some_coroutine_function())
except KeyboardInterrupt:
print('Interrupted')
some_terminating_task()
finally:
loop.close()
print('Eventloop closed')
This program prepares for Ctrl + C using KeyboardInterrupt, without a signal handler. It looks simple and clear, but it is not sufficient for handling signals. Because signals can arrive at any time, one could occur inside terminating_some_tasks() in the except block, or inside finally.
Would wrapping the contents of except or finally in another try-except KeyboardInterrupt solve the problem? Unfortunately, a signal could arrive inside that new except block too. Ultimately, KeyboardInterrupt alone cannot safely handle SIGINT.
Signals in asyncio
To handle SIGINT and SIGTERM—the latter used in production lifecycle management—properly, use loop.add_signal_handler.
import asyncio
import signal
loop = asyncio.new_event_loop()
def sig_handler():
print('Shutdown')
some_terminating_task()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, sig_handler)
try:
loop.run_until_complete(some_coroutine_function())
finally:
loop.close()
print('Eventloop closed')
The important point is to call loop.add_signal_handler as early as possible. Signals can arrive at any time. If time-consuming initialization happens first, the handler cannot handle signals that arrive during that initialization.
Example 1: controlling the event loop from a signal handler
Handlers can receive parameters, so we can pass the event loop to one and manipulate it. Let's change part of the previous example.
def sig_handler(loop):
print('Shutdown')
loop.stop()
some_terminating_task()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, sig_handler, loop)
Example 2: registering an async function through a signal handler
A signal handler must ordinarily be a regular function without the async keyword. If you want to use await inside your handling logic, however, you can use loop.create_task.
async def sig_handler(loop):
pass
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, lambda l: l.create_task(sig_handler(l)), loop)
One important distinction: a regular function registered as a handler runs immediately when the signal is handled, whereas work scheduled with loop.create_task may not. As explained in Part 1, loop.create_task does not execute the coroutine immediately; it registers it with the event loop. If too many tasks are registered, or a CPU-intensive task is not yielding control, there may be a delay between the signal arriving and the handling coroutine running.
Example 3: cancelling all tasks running on the current event loop
Since SIGTERM and SIGINT are used for shutdown, sometimes we want to terminate all tasks running on the current loop.
async def sig_handler(loop):
tasks = []
for t in asyncio.all_tasks():
if t == asyncio.current_task():
continue
tasks.append(t)
t.cancel()
await asyncio.gather(*tasks)
loop.stop()
Calling t.cancel() raises asyncio.CancelledError in the task, so each task must handle it appropriately. asyncio.gather waits for the tasks to finish handling that exception and complete their coroutines.
Putting it together: a runnable example
Here is a runnable example combining these approaches. The script runs an infinite loop that prints sleep once per second. We add handlers for SIGINT and SIGTERM that terminate that loop.
To make the behavior visible just by running the script, I also added send_sigterm_after_5_seconds, which sends SIGTERM to the script itself after five seconds. If the SIGTERM handler is configured correctly, you will see stopping loop... after five seconds.
import asyncio
import signal
import os
shutdown: bool = False
def sig_handler() -> None:
""" signal handler. `print_and_sleep`의 무한루프를 종료시킨다. """
print('stopping loop...')
global shutdown
shutdown = True
async def print_and_sleep(loop) -> None:
""" 1초에 한번 `sleep`이라는 문자를 출력하는 무한루프 """
while not shutdown:
print('sleep')
await asyncio.sleep(1)
loop.stop()
async def send_sigterm_after_5_seconds() -> None:
""" 5초 이후에 스크립트 자신에게 SIGTERM을 보낸다. """
await asyncio.sleep(5)
os.kill(os.getpid(), signal.SIGTERM)
def main() -> None:
loop = asyncio.new_event_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, sig_handler)
loop.create_task(print_and_sleep(loop))
loop.create_task(send_sigterm_after_5_seconds())
try:
loop.run_forever()
print('loop stopped')
finally:
loop.close()
print('loop closed')
if __name__ == '__main__':
main()
# SIGINT 없이 스크립트 실행 후 5초 동안의 출력
# sleep
# sleep
# sleep
# sleep
# sleep
# stopping loop...
# loop stopped
# loop closed
A note on containers
An incorrectly written Dockerfile ENTRYPOINT or a wrapper script can prevent your application from receiving signals. https://hynek.me/articles/docker-signals/ explains the causes and provides guidance.
Closing thoughts
This post introduced signals and how to handle them correctly in asyncio. In the next post, I will build on the series so far to share how we implemented graceful shutdown in a real production server.
Apply for an engineering role at Buzzvil
Have a coffee chat with a Buzzvil tech recruiter