Production Checklist

This document describes what to check before deploying a newly developed service. Before deploying a new service to Kubernetes, we recommend verifying that it meets the following requirements.

Docker Images

Preparing a Docker image for production requires more than making a development environment run with docker-compose up. Follow the Docker guide to write a lightweight, secure Dockerfile suitable for production.

Logs

Send all logs to STDOUT or STDERR. A containerized service should focus on its single responsibility of running the server or main command, leaving log files, directories, log rotation, and similar concerns to the container orchestrator (Kubernetes). If you need to send metrics to external storage, you can deploy tools such as fluentd as sidecars, but this makes the pod lifecycle more complex and is discouraged unless necessary. One future option is to archive metric logs written to STDOUT in S3, then add an Airflow task to filter and process those metrics separately.

Environment Variables

Important application settings, such as external service credentials and database connection details, should be configurable through environment variables. This follows the config principle of the Twelve-Factor App. In a container orchestration environment such as Kubernetes, it makes it easier to separate configuration from build artifacts (container images) and to add deployment environments, such as CI or end-to-end test environments, without changing the code.

Dependencies

Pin system and language-specific package dependencies to explicit versions. If a version is ambiguous—for example, only the major version is specified—building an image from the same source code later can produce a different dependency graph. Keep lockfiles such as package-lock.json under version control.

Dependencies can easily fall behind in projects that receive little maintenance. Use tools such as dependabot to review and update package versions continuously.

Writing a README

The project README should explain the application’s basic usage, including prerequisites, starting the server, and running migrations.

Graceful Shutdown

Events such as rolling updates and node drains put existing pods into the Terminating state. Pod termination is a routine part of operating in Kubernetes. Each application is responsible for handling it gracefully.

When an application receives SIGTERM, it should stop accepting new requests, finish processing requests already in progress, and then exit.

• golang

For details on the pod lifecycle, see the Google Cloud blog post Kubernetes best practices: terminating with grace.

Readiness and Liveness Probes

Every application should provide a way to check its health. Kubernetes offers two kinds of checks:

• readiness: Checks whether the application is ready to handle requests. A successful response means it can serve traffic.
• liveness: Detects when an application has a problem that requires a restart. If the liveness probe fails beyond the configured threshold, the orchestrator restarts the pod.

Health checks can use commands as well as HTTP endpoints. A gRPC server can implement the grpc-health-probe protocol and run the grpc-health-probe command to check its health.

Monitoring

You also need to configure error tracking, application performance management (APM), and related tools.

TBD