Improving Feature Flag API p99 Latency—and Contributing the Fix Upstream

The latency experienced by the 1%

After introducing feature flags, our Supply group needed infrastructure to store evaluation results and make better use of A/B testing. We used the export functionality in the open-source go-feature-flag library to build S3Export, which writes each pod's flag-evaluation results to S3 at minute intervals. The API was low-latency at p95, around 1 ms, but we intermittently encountered S3 export retry problems and p99 latency above three seconds. On May 1 alone, the evaluation API received 68.5 million calls, meaning roughly 680,000 calls at the p99 tail experienced delays of three seconds or more. Because evaluation runs within another API call, such as loading the BenefitHub main screen, we needed to minimize its impact on the parent API's latency.

Metric	Value
API	GET /피쳐플래그-evaluation
p95	~1ms
p99	3 seconds or more
Gap between p95 and p99	Around 3,000× or more

The feature flag infrastructure uses Redis for flag metadata and S3 for evaluation results. Metadata is fetched from Redis once a minute, cached locally, and used for evaluation. Even with many pods reading once per minute, that seemed unlikely to explain p99 above three seconds. We run a dedicated Redis instance for feature flags, so its latency metrics made it easy to rule out.

That left S3. Digging deeper, we found two distinct latency problems, both related to a mutex, or shared-resource lock.

Problem 1 — p99 spikes: p99 suddenly jumps to seconds or tens of seconds.
Problem 2 — maximum-latency baseline: even after resolving the spikes, each flush still blocks for 60–70 ms.

A mutex allows only one goroutine through at a time. If it is held while waiting for a long operation, all subsequent contenders must wait. This article follows how we traced and solved the two lock-related problems.

---

How the feature flag evaluation API works

When a client requests evaluation of a flag key, the API reads flag data from the local cache and returns a value such as a boolean, string, or int. Source data is refreshed from Redis once a minute. With sub-millisecond evaluation, a normal p95 below 1 ms made sense.

The go-feature-flag library records each evaluation as an event and exports events to S3 when certain conditions are met. Usually it only appends events to an in-memory buffer, with little effect on response time. That changes when a flush occurs.

The flush mechanism

The library's DataExporter triggers a flush either from a timer every 60 seconds or when the buffer reaches 100,000 events. In both cases, it acquires a mutex and performs S3 export synchronously.

Trigger	Condition	Execution context
Timer	Every 60 seconds	Daemon goroutine
Buffer	100,000 events	Request goroutine

AddEvent() runs for every evaluation request and uses the same mutex to append to the buffer. While a flush holds the lock, every incoming request's AddEvent() waits, increasing API latency.

AddEvent() 호출
     │
     ▼
mutex.Lock()  ◀── flush 중이면 여기서 대기
     │
     ▼
len(cache) >= 100,000?
     │           │
    YES          NO
     │           │
     ▼           ▼
  flush()     cache에 이벤트 추가
  (동기식)    
     │
     ▼
S3Exporter.Export()
     │
     └─▶ S3 Upload (~60-70ms)

This is the shared background to both problems: other requests must wait while the flush holds the mutex.

---

Problem 1: p99 latency—jitter sleeping while holding the mutex

We needed to determine whether flushing itself was slow or something else was delaying it. We added debug logs and analyzed real traffic.

Metric	Value	Interpretation
event_count	~5,000	Only 5% of the 100,000-event buffer
export_interval	~60 seconds	Normal timer-based flush
duration (S3 upload)	~60–70ms	The upload itself is fast

S3 uploads took only 60–70 ms, and the buffer never reached 100,000 events. Where did the three-second latency come from? The applyJitter() function in the application's S3 exporter.

The error above occurs when multiple pods upload simultaneously—a thundering herd—and S3 returns 503 until the maximum retry count is exceeded. We had added a random delay of 0–7 seconds (jitter) to each pod's retry to spread requests out. This did resolve S3 throttling, but the sleep happened while holding the mutex. A flush could sleep for up to seven seconds, making every other request's AddEvent() wait.

func (f *S3Exporter) applyJitter(ctx context.Context) error {
    if f.MaxJitterMs <= 0 {
        return nil
    }
    jitter := time.Duration(rand.IntN(f.MaxJitterMs)) * time.Millisecond
    select {
    case <-ctx.Done():
        return fmt.Errorf("interrupted")
    case <-time.After(jitter):  // ← 최대 7초 sleep
        return nil
    }
}

Disabling jitter

The most immediate improvement came from a PR that temporarily disabled jitter. Once we were confident in the cause, we simply commented out applyJitter() in Export. The effect was dramatic: immediately after deployment, the daily maximum latency fell from 55 seconds to around 200 ms and remained stable.

After removing jitter, we mitigated the thundering herd by extending the export interval from one minute to three. This reduced how often pods flushed simultaneously and brought S3 throttling under control. It did not eliminate the errors completely, but they became very rare.

---

Problem 2: maximum latency—synchronous flush blocks the API

Disabling jitter and adjusting the interval resolved the p99 spikes, but another problem remained. During each flush, the mutex was held for the duration of the S3 upload, making all requests wait and intermittently increasing maximum latency. Pausing every API request at once during a flush was still undesirable.

An AsyncExporter wrapper

We concluded that making export asynchronous was the fundamental improvement. Previously, a flush held the mutex until the S3 upload finished. With an AsyncExporter wrapper, Export() hands events to a channel and returns immediately. The library's scheduler still releases the mutex, but because Export returns at once, lock hold time falls to microseconds. S3 uploads run asynchronously in a separate background goroutine.

Why a wrapper rather than changing the library?

We first considered upgrading go-feature-flag, but even the latest version, v1.52.1, retained the DataExporter scheduler's pattern of holding a mutex while exporting synchronously. An upstream upgrade alone would not solve the structural problem. We also considered forking the library, but the maintenance burden was high. Instead, we implemented the library's Exporter interface in an asynchronous wrapper around the existing S3Exporter.

Execution flow

Scheduler.flush()
    │
    ▼
AsyncExporter.Export()
    │  이벤트를 채널로 전달하고 즉시 리턴
    │
    ▼
buffered channel (size: 10)
    │
    ▼
background goroutine
    │  채널에서 이벤트를 꺼내 실제 업로드 수행
    │
    ▼
S3Exporter.Export()  →  S3 업로드 (수백 ms)

When the scheduler calls AsyncExporter.Export(), it sends the event data to a channel and returns immediately. A background goroutine takes jobs from the channel and uploads them to S3. From the scheduler's perspective, Export is just a channel send, reducing mutex hold time to microseconds.

The core implementation of Export() is:

type exportJob struct {
	ctx    context.Context
	events []ffexporter.FeatureEvent
}

type AsyncExporter struct {
	inner     ffexporter.Exporter
	eventsCh  chan exportJob
	...
}

func (a *AsyncExporter) Export(
    ctx context.Context, logger *fflog.FFLogger, events []ffexporter.FeatureEvent,
) error {
    copied := make([]ffexporter.FeatureEvent, len(events))
    copy(copied, events)

    select {
    case a.eventsCh <- exportJob{ctx: ctx, events: copied}:
    default:
        log.Warnf("channel full, dropping %d events", len(events)) // 채널 포화 시 블로킹 없이 drop
    }
    return nil
}

• Buffered channel, size 10: Delivers export jobs asynchronously. With a three-minute interval, ten entries buffer up to 30 minutes of batches. That provides room for an extreme situation in which S3 uploads remain delayed for more than 30 minutes. Under normal operation, the channel is almost empty.
• Shallow copy: Export uses copy() to shallow-copy the event slice before sending it. Reference fields such as FeatureEvent.Value (any) and Metadata (map[string]any) are not deep-copied. Since the calling scheduler does not mutate events after Export, this avoids unnecessary deep-copy costs.
• Drop when full: If the channel is full, a select-default branch logs a warning and drops the batch without blocking. This ensures that the API request path remains nonblocking.

Situation	Impact	Response
S3 export fails	Batch events are lost without retry	Monitor error logs; analytics data is lost, but service behavior is unaffected
Prolonged S3 delay fills channel	Excess event batches are dropped	Ten slots × three-minute interval gives up to 30 minutes of buffering; log a warning

After deploying the AsyncExporter wrapper to production, maximum latency that had reached 25 seconds in the worst case stabilized at around 200 ms.

Contributing upstream: reducing mutex hold time

Beyond the service-side improvement, the structural issue in go-feature-flag itself seemed worth fixing. The DataExporter scheduler held a mutex throughout reading the cache, synchronously calling Export, and clearing the cache. Network I/O inside that critical section made lock hold time unnecessarily long.

// 기존 코드 — ProcessPendingEvents
func (e *eventStoreImpl[T]) ProcessPendingEvents(...) error {
    e.mutex.Lock()
    defer e.mutex.Unlock()   // ← S3 업로드 끝날 때까지 유지
    eventList, _ := e.fetchPendingEvents(consumerID)
    processEventsFunc(...)   // ← S3 업로드 (수십~수백 ms 블로킹)
    e.updateConsumerOffset(...)
}

// 변경 후 — 락 분리 (3단계)
func (e *eventStoreImpl[T]) ProcessPendingEvents(...) error {
    // Step 1: 이벤트 조회 (짧은 락)
    e.mutex.Lock()
    eventList, err := e.fetchPendingEvents(consumerID)
    e.mutex.Unlock()

    // Step 2: Export 실행 (락 없음 — 느린 I/O)
    err = processEventsFunc(context.Background(), eventList.Events)

    // Step 3: offset 업데이트 (짧은 락)
    e.mutex.Lock()
    err = e.updateConsumerOffset(consumerID, eventList.NewOffset)
    e.mutex.Unlock()
}

I opened a PR to fix it. The change copies cached data into a local variable, releases the mutex immediately, and performs the actual export outside the lock. Even if export's network I/O takes a long time, other goroutines can call AddEvent() without being blocked.

The PR was merged, turning the investigation into an open-source contribution. With this change, even re-enabled jitter sleeps outside the mutex, preventing thundering herds without affecting API latency. We plan to apply it in production when the next library version is released.

---

Closing thoughts

Both problems came down to the same cause: performing I/O while holding a mutex. We initially suspected S3 throttling or network latency, but the real issue was slow operations inside the lock scope. In the first case, jitter slept for up to seven seconds under the lock. In the second, synchronous flushing held it during an S3 upload. The scale differed, but the pattern was the same. Both were addressed by one principle: minimize the lock scope.

Investigating a production latency issue led us to fix a root cause in a dependency and contribute it upstream. It was a valuable opportunity to look beyond our own service code and broaden the ways we solve problems.

Maximum latency stabilized at around 200 ms

Stage	Action	Improvement
Problem 1	Disable jitter and adjust export interval	Maximum 7 seconds → ~200ms
Problem 2	Introduce AsyncExporter wrapper	Maximum 25 seconds → within ~200ms

References

• go-feature-flag
• fix(exporter): reduce mutex hold time in process pending events