Skip to content
:tech
← Articles

AI-translated. Read the original in KoreanEmbedded slides, videos and images remain in their original language.

awsdynamodbhttpgo

Optimizing Latency in a Go Service Using DynamoDB, Part 2: TLS Handshakes

R
Raf Kim

Data Engineer

Optimizing Latency in a Go Service Using DynamoDB, Part 2: TLS Handshakes

Hello, I'm Raf, a data engineer at Buzzvil. Following the previous post on AWS credential tokens, I would like to share another attempt to optimize latency in a Go service using DynamoDB. This time, we did not find a complete solution, but we learned a great deal about DynamoDB and Go's HTTP client.

High DynamoDB latency

Traffic gradually increased in the months after we launched the rewards service. At some point, Datadog began alerting us to a very small number of gRPC requests with high latency. Looking at the DynamoDB query-latency dashboard, we saw maximum values exceeding 100 ms.

text Figure 1) DynamoDB query-latency dashboard

According to DynamoDB Metrics and Dimensions, this dashboard includes only server-side processing time, not network or client time. During load testing, we had assumed DynamoDB latency was low and had not seen server-side spikes. This time it was DynamoDB itself taking longer, so we considered it a completely different problem.

We found Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications, which describes Java AWS SDK settings that time out DynamoDB requests and retry them. The Go AWS SDK did not support the same feature, so we implemented retry-on-timeout logic ourselves.

The code attempts a request up to three times, with a 30 ms timeout. Errors that would inevitably recur are returned immediately; retryable errors, including timeouts, trigger another attempt. We expected Datadog to show gRPC requests completing within 100 ms after deployment.

TLS handshakes became a source of latency

After deployment, we monitored slow gRPC requests. In some cases, when a DynamoDB request timed out, the next attempt also timed out. I had expected occasional network problems to cause slow responses, with a retry then completing quickly. Instead, the timeouts appeared causally related rather than independent, so we added traces through httptrace.ClientTrace. For details, see the first post, HTTP Connection Pool in Go Explained.

Figure 2) Datadog APM

The traces showed that a timeout on the first attempt was generally followed by a TLS handshake on the second. Every gRPC request exceeding 100 ms involved a handshake, and some handshakes themselves took more than 100 ms. APM showed handshakes continuing after a timeout because connections are created in a separate goroutine. Since a TLS handshake is only needed when establishing a connection, we confirmed that most slow requests were trying to create new connections.

If we could reduce handshake latency, the problem seemed straightforward to solve. We looked into the handshake process.

Figure 3) TLS handshake (source: TLS Handshake)

The first messages, Client Hello and Server Hello, exchange metadata needed to establish the connection. In particular, TLS Session Resumption can simplify the handshake and reduce latency. After an initial handshake, a new TLS connection can reuse the session received earlier. Session ID resumption caches the session on the server, while session ticket resumption does not require server-side caching. With these properties in mind, we explored several options and checked their feasibility.

1) Could we avoid TLS?

This is not something we should do, but it was the simplest option, so we checked whether it was even possible. Infrastructure Security in Amazon DynamoDB stated that DynamoDB required TLS 1.0 or later and recommended TLS 1.2 or later. We would have to reduce latency while retaining TLS.

2) Could we make the TLS handshake faster?

Next, we considered minimizing network hops. Handshakes normally took around 10 ms; only some failing requests took around 100 ms. We therefore suspected network latency rather than handshake computation was the main bottleneck.

DynamoDB is normally accessed over the public internet, but a VPC endpoint allows communication within AWS's network without passing through a public gateway (Using Amazon VPC Endpoints to Access DynamoDB). Our rewards service ran on Amazon EKS, so using an endpoint would require infrastructure changes. Without confidence that it would reduce latency, we deferred that option. Later, during a DynamoDB clinic with AWS Solutions Architects, we asked about it. They advised that a VPC endpoint was unlikely to provide much benefit if the goal was lower latency.

3) Could we avoid the handshake itself?

If we could use session resumption, we could optimize latency without much difficulty. We could not find a clear official answer, so we opened a support case asking whether DynamoDB supported it. While waiting, we noted that ServerHello tells the client whether resumption is supported and logged this information in Go's TLS layer. Even when we sent ticketSupported: true in ClientHello, ServerHello returned false. We also confirmed that Go did not support Session ID resumption.

AWS later replied:

  • TLS sessions are honored only on same DynamoDB IP the initial TLS session was created on. We do not support TLS session resumption across DynamoDB IPs.
  • We do not support Extended Master Secret RFC 7627. If the client does use Extended Master Secret TLS session resumption will not operate.

We concluded that we had no way to avoid the handshake in this setup. It was disappointing that, despite DynamoDB's emphasis on low latency, we could not use session resumption to address handshakes that substantially affected actual response times.

4) Could we reuse connections to minimize handshakes?

If TLS handshakes were unavoidable, we needed to reuse connections as much as possible. Yet seeing handshakes during retries suggested reuse was failing.

The first post explained how connections return to the pool. Here, they were not returning, so we investigated when that can happen. If tryPutIdleConn() cannot insert a connection into idleConn, it returns an error and calls persistConn.close() to close the connection. That method also causes the readLoop() and writeLoop() goroutines to exit.

Following the callers of persistConn.close(), we found that a context timeout notifies the request's readLoop() and writeLoop() goroutines, which then close the connection. Our short timeout was causing retries and repeated handshakes—the opposite of what we wanted. We needed a timeout that avoided excessive retries and gave a handshake enough time to succeed when a retry did occur.

I raised the timeout to 70 ms to tolerate moderately slow DynamoDB responses. If a retry's handshake took around 100 ms, the second attempt would still fail. However, connection creation runs in a different goroutine from the one processing the gRPC request. If a connection finishes opening with no goroutine waiting to use it, it becomes idle and enters the pool. Thus, although the third attempt may try to open yet another connection, an idle connection can become available in the meantime and carry the DynamoDB request.

5) Could we keep enough connections in the pool?

Another option was to retain enough idle connections to replace any that were closed immediately. As explained in HTTP Connection Pool in Go Explained, tuning MaxIdleConnsPerHost and IdleConnTimeout can keep as many connections in the pool as the server actually uses.

The default IdleConnTimeout is 90 seconds, but our service tends to receive a traffic burst every ten minutes. Ten minutes would fit that pattern, yet raising the timeout that far did not seem reasonable. The pool also uses LRU behavior, with no round-robin option, so keeping every connection alive for a long time seemed difficult.

An AWS Solutions Architect explained that increasing IdleConnTimeout might not work anyway, because DynamoDB could close the connections from the server side. Later, a detailed architecture talk taught me that DynamoDB has many Request Router instances that receive requests and forward them to partition-hosting Storage Nodes. Given that architecture, I wondered whether closing client connections could help maintain availability by avoiding a client's persistent association with one Request Router instance.

AWS also suggests sending dummy traffic over connections to keep them alive. We did not adopt that approach because it seemed likely to increase code complexity.

Closing thoughts

Our initial goal was to keep gRPC request timeouts within 100 ms. Trying to optimize even the occasional slow request, however, could actually harm the connection pool. An appropriate timeout resolved most slow requests, but a tiny fraction that remained slow unfortunately still had to be dropped.

Investigating Go's HTTP connection pool and DynamoDB in depth was a valuable experience. In particular, it made me more interested in DynamoDB's architecture and how its features are implemented. If you are curious too, I recommend the following two talks. Thank you for reading this long post!

Apply for an engineering role at Buzzvil

Have a coffee chat with a Buzzvil tech recruiter

All articles