AI-translated. Read the original in KoreanEmbedded slides, videos and images remain in their original language.
Optimizing Latency in a Go Service Using DynamoDB, Part 1: AWS Credential Tokens
Data Engineer
Hello, I'm Raf, a data engineer at Buzzvil. In my previous post, I explained Go's HTTP connection pool (HTTP Connection Pool in Go Explained). This post describes how tracing HTTP requests in a Go service helped us discover and optimize credential token issuance logic that had a major effect on DynamoDB request latency.
Amazon DynamoDB is a NoSQL database supporting low latency and strong consistency. Although the process is invisible to users, it can scale horizontally while reads and writes are in progress, providing excellent scalability and availability. Its data model supports not only partition and sort keys, but also LSIs with alternative sort keys and GSIs with alternative partition keys, accommodating many access patterns.
At Buzzvil, we mainly use DynamoDB for write-intensive tables. For these workloads, it reduces operational burdens such as migrations compared with MySQL. Our rewards table records the rewards users earn from viewing ads. It is write-intensive and requires low latency and high throughput, so we chose DynamoDB. While implementing the new rewards service, however, we encountered errors we had not noticed before.
Intermittent latency spikes at low traffic
While building the rewards service, we also introduced the Locust load-testing framework. Tests can be written in Python or Go, matching Buzzvil's stack and making adoption easy. We also chose it because it does not restrict the target to HTTP: it can make gRPC calls, send events to a message queue, and more.
Our goal was to increase traffic gradually until the service could reliably handle 1,000 RPS. After writing a simple Locust agent that called the rewards service, we began monitoring at around 3 RPS. Once every hour or two, one or two requests would spike to nearly two seconds. Since the proportion of slow requests could increase with traffic, we needed to find the cause.
Adding traces to find the cause
First, we checked the DynamoDB CloudWatch dashboard. If it showed a maximum latency that high, we could conclude that processing inside DynamoDB was taking the time. But there were no requests approaching two seconds.
That suggested the delay was in the rewards service sending the request. Buzzvil instruments each server layer with traces that can be viewed in Datadog APM, making slow functions easy to identify. We used Datadog's AWS tracing to instrument the AWS session. Figure 1 illustrates the result; I no longer have the original screenshot.
The trace around the rewards service repository's DynamoDB Query call lasted about two seconds, while the DynamoDB command itself took less than roughly 20 ms. In other words, the Go AWS SDK was spending nearly two seconds doing something other than executing the command.
Figure 1) Datadog APM after adding DynamoDB traces
We examined how the Go AWS SDK processes requests. Two likely sources of delay were the HTTP request to DynamoDB and the SDK's HTTP request to renew temporary credentials, or tokens. Figure 1 showed that the DynamoDB request took less than 20 ms, so credential issuance was the likely culprit. The SDK sent that request only when the token expired, which also explained the intermittent spikes. We decided to trace the issuance logic to verify this directly.
Go supports tracing several stages of an HTTP request, and the AWS SDK supports this through an injected http.Client. We wrapped http.Transport in a struct to trace the client. For details, see the previous post, HTTP Connection Pool in Go Explained.
Figure 2) Datadog APM after tracing http.Client
Figure 2 shows a slow request after adding the client traces. As expected, the HTTP request to renew the token took nearly two seconds. We began researching that logic.
Refreshing tokens in the background
While researching AWS SDK token issuance, we found documentation explaining that using a regional STS endpoint rather than the global endpoint reduces latency (Managing AWS STS in an AWS Region). We applied it immediately by adding the environment variable AWS_STS_REGIONAL_ENDPOINTS=regional.
Figure 3) Datadog APM after switching to a regional endpoint
Token renewal dropped from as much as two seconds to around 100 ms. That was a major improvement, but a gRPC request was still noticeably slower if it included renewal. Because this happened intermittently, average latency looked fine. Still, the increased chance of a gRPC timeout whenever a token expired was a potential risk.
Possible solutions were to use non-expiring access-key credentials, remove token expiration, or refresh tokens in the background. Non-expiring authentication would introduce security weaknesses, so the first two were not viable. For the third option, we found a similar issue in the Go AWS SDK repository and a comment proposing background refresh. Because the code used locks, we wanted to understand the logic before adopting it.
We revisited the SDK's credential structure to check whether the proposed code would work. It is organized as follows.
credentials.Credentialsrequests tokens and checks expiration through thecredentials.Providerinterface.stscreds.WebIdentityProviderimplementscredentials.Providerusing temporary credentials. The implementation is selected when the SDK initializes, based on environment variables.credentials.EnvProvider, which uses access keys supplied through environment variables, does not expire.
Figure 4) Token issuance through stscreds.WebIdentityProvider
Figure 4 shows the interactions between components and token state over time. When the token in credentials.Credentials expires, it requests one through stscreds.WebIdentityProvider, which sends an HTTP request to the AWS STS endpoint. The provider sets the expiration time internally and returns the token.
The background refresh code works like this: RefreshProvider, a struct wrapping credentials.Provider, periodically obtains a token in a goroutine and caches it. When credentials.Credentials calls Provider.Retrieve(), it returns that cached token. Since actual issuance is separated from requests to the rewards service, this looked likely to solve the problem.
Making sure we always use a single token
After deploying and testing, we encountered another error: intermittent InvalidSignatureException responses. There were no other changes, so the new code was clearly involved, but the error came from DynamoDB and we could not identify its underlying cause. The documentation said an expired signature could produce this error (Troubleshooting AWS Signature Version 4 errors).
We wondered whether STS might invalidate the previous token because the rewards service held two tokens simultaneously. In the code, RefreshProvider obtained a new token, but it was not used immediately. credentials.Credentials kept using the old one until it expired, then switched to the new cached token. As Figure 5 shows, even after the goroutine—the blue arrow—obtained a new token, credentials.Credentials continued using the old one for a long time, until expiration.
Figure 5) Using the older of two simultaneously unexpired tokens
We therefore changed the rewards service to use just one token. If RefreshProvider.periodicRefresh() made credentials.Credentials use the new token immediately after issuance, the service would no longer maintain two tokens at once.
Figure 6 illustrates the change. The updated goroutine, shown in blue, requests a token every ten minutes. On success, it calls Credentials.Expire() to expire the token held by credentials.Credentials, prompting it to retrieve the cached token from RefreshProvider. The code is available in this GitHub Gist.
Figure 6) Expiring the token in credentials.Credentials immediately after RefreshProvider obtains a new one
After monitoring for several days without an InvalidSignatureException, we considered the token issuance problem resolved. One aspect of the implementation still bothered me: both credentials.Credentials and credentials.Provider cache tokens, and forcibly calling Expire() to update both feels like a circular dependency rather than a clean design.
Tracing http.Client with Datadog APM revealed AWS SDK credential logic we had not previously noticed. Moving it off the RPC call's critical path eliminated the intermittent latency spikes. We then increased load-test traffic over several days, verified that the service remained stable with more requests, and released it.
Several months after release, however, traffic gradually increased and more records accumulated in DynamoDB. We saw high latency again. I will share that issue in the next post.