March 9, 2021 · TECH

Gracefully Shutdown your Go Application

A walkthrough on implementing graceful shutdown for Go applications — from signal handling to concurrent cleanup with timeouts. Originally published on Tokopedia Engineering.

Originally published on Tokopedia Engineering.

TL;DR — Listen for SIGTERM, execute cleanup operations concurrently, enforce a timeout to prevent hangs, and exit cleanly. Use os.Exit(1) on timeout to signal abnormal termination.

A Common Problem

During deployment, services terminate to load new code. If the application shuts down abruptly while handling requests, those requests fail. Graceful shutdown ensures in-flight requests complete before exit.

At Tokopedia, we perform numerous deployments daily across multiple services. Graceful shutdown prevents request errors during deployments and applies broadly to any Go application.

What should we do?

Graceful shutdown allows a process to safely terminate by completing cleanup tasks before exit: finishing in-flight requests, closing database connections, flushing buffers, etc.

This is the Disposability principle from The Twelve-Factor App—applications must start and stop quickly without losing data.

Implement this by listening for termination signals from the process manager, executing cleanup concurrently, and enforcing a timeout to prevent hangs.

Specifically:

  1. Listen for the termination signal/s from the process manager like SIGTERM
  2. We block the main function until the signal is received
  3. After we received the signal, we will do clean-ups on our app and wait until those clean-up operations are done
  4. We also need to have a timeout to ensure that the operation won’t hang up the system

The Code

Here’s a complete implementation. This pattern is reusable, scales as your application grows, and addresses all four requirements:

Listen for termination signals

s := make(chan os.Signal, 1)
// add any other syscalls that you want to be notified with
signal.Notify(s, syscall.SIGINT, syscall.SIGTERM, syscall.SIGHUP)
<-s

Listen to at least SIGTERM (standard for process managers). Add SIGINT for local development and SIGHUP if you support config reloads.

Execute cleanup operations

wait := gracefulShutdown(context.Background(), 2*time.Second, map[string]operation{
    "database": func(ctx context.Context) error {
        return db.Shutdown()
    },
    "http-server": func(ctx context.Context) error {
        return server.Shutdown()
    },
    // Add other cleanup operations here
})

<-wait

Register all cleanup operations here: database closes, HTTP server shutdown, Redis cleanup, metric flushing, profiling teardown, etc. Each operation is a function that returns an error.

Execute cleanup concurrently

var wg sync.WaitGroup

for key, op := range ops {
    wg.Add(1)
    innerOp := op
    innerKey := key
    go func() {
        defer wg.Done()

        log.Printf("cleaning up: %s", innerKey)
        if err := innerOp(ctx); err != nil {
            log.Printf("%s: clean up failed: %s", innerKey, err.Error())
            return
        }

        log.Printf("%s was shutdown gracefully", innerKey)
    }()
}

wg.Wait()

Run operations in parallel to minimize shutdown latency. The explicit capture (innerOp := op; innerKey := key) is necessary for Go versions before 1.22, which had different loop variable scoping semantics. Go 1.22+ fixes this, but explicit capture remains a safe pattern.

Ensure no resource sharing between operations to avoid race conditions.

Enforce a timeout

timeoutFunc := time.AfterFunc(timeout, func() {
    log.Printf("timeout %d ms has been elapsed, force exit", timeout.Milliseconds())
    os.Exit(1)
})

defer timeoutFunc.Stop()

If cleanup doesn’t complete within the timeout, force exit with status code 1 to signal abnormal termination to the process manager. This prevents the application from hanging indefinitely during deployment.

Conclusion

Graceful shutdown is essential for reliable deployments. Combined with proper request routing and load balancing, this pattern eliminates request errors during updates and prevents resource leaks.

Implement this in production. It handles the common case (timeout expires and operations complete) and the edge case (forced termination). Your users won’t notice deployments.

Related resources: