Behind BuzzBenefit: The Backend Team Serving Korea’s Five Leading Financial Institutions

Hello, I'm Wynn, lead of the Product Backend team in Buzzvil's Supply group.

We're currently looking for a new teammate to help us grow. A job posting can only tell you so much about a team and its work, so I wanted to share a fuller picture here.

Our team is responsible for the backend systems that power BuzzBenefit. In this post, I'll explain why the Product Backend team was formed and what we do today, with examples from our work.

Why the Product Backend Team Was Formed

For roughly ten years, from its launch through early 2025, the BuzzBenefit SDK primarily focused on serving rewarded ads. Early this year, however, it began rapidly evolving beyond an advertising platform into a platform that uses gamification to engage users and offer a wider range of benefits.

This year, we secured partnerships with all five of Korea's leading financial institutions, expanding beyond digital banks and fintech companies such as KakaoBank and Kakao Pay and strengthening our position as Korea's leading rewards monetization platform. Even with the largest market share already, our user base grew by more than 50%.

That rapid product growth meant our backend systems needed to change, too.

To build a platform that encourages users to participate, stay longer, and enjoy richer benefits, the product organization planned a range of gamification features. It also wanted to expand data analysis to measure each feature's effectiveness and run more experiments to test hypotheses.

Our backend, however, was optimized for ad delivery. Each new feature introduced different data structures and logic, fragmenting the flow of data. Connecting data to create synergies between features became increasingly complicated, and we were reaching the limits of both scalability and maintainability.

Above all, collecting and processing data at the experiment level was difficult. The architecture was making it harder for product teams to validate hypotheses at the pace they needed or connect user behavior data to uncover insights. We needed a system that could handle traffic reliably while making new features and experiments easier to build.

The Product Backend team was formed to address these problems at their roots and support BuzzBenefit's growth, today and into the future.

Our Principles and Goals

We established three core principles when we formed the team:

1. Go beyond building APIs to design data models and processing flows across the product.
2. Build a scalable architecture.
3. Build systems that support an experimentation-driven product culture, including A/B testing and feature flags.

These principles translate into the following day-to-day work:

• Developing APIs that serve data to apps and the web
• Implementing business logic for new benefits features
• Designing and building data pipelines
• Supporting feature experiments
• Monitoring systems
• Improving systems to make operations more efficient

Our Technology Stack

Our current stack includes:

• Applications: Golang, Python, MySQL, AWS DynamoDB, Redis, Kafka, Airflow, Confluent Kafka Connector, Argo Workflows
• Infrastructure: Kubernetes, Istio, Docker, Terraform, AWS, Helm, Argo CD, Datadog, Loki, Grafana, Prometheus

Examples of Our Work

To give you a better sense of the team, here are a few projects we've worked on.

1. Launching the Group Account Mission Challenge for KakaoBank

We recently launched the Group Account Mission Challenge. This project ranks members of KakaoBank group accounts based on their ad participation and lets them collect rewards according to their ranking.

Key Challenges

The project presented three main challenges:

1. Generate user rankings from an existing, large-scale stream of ad participation data.
2. Keep data consistent across services in a microservices environment.
3. Build a ranking pipeline that is both reliable and scalable.

Data Flow

Before walking through the data flow, let me introduce the main server components:

• Ad participation API server: Records users' ad participation and requests reward credits from KakaoBank's systems.
• Ranking API server: Provides user ranking lookups and group account registration.

Rankings are calculated from user participation events. To publish those events, the ad participation server needed group account information. But that information existed only in the ranking server's database, so we needed to synchronize, or replicate, data between the two databases.

Building a CDC (Change Data Capture) Pipeline

We built a CDC pipeline that detects database changes in real time and propagates them to another database, using the Kafka-based Confluent Platform.

Confluent Platform is a Kafka-based platform that makes it easier to build real-time data pipelines and streaming applications.

• JDBC Source Connector → Detects changes in the MySQL database and publishes them to a Kafka topic.
• JDBC Sink Connector → Subscribes to the Kafka topic and automatically loads the data into another database.

This let us maintain consistency between databases without writing separate synchronization logic, while making the extract/load pipeline straightforward to implement.

The ad participation server could now use group account information to publish participation events correctly.

The Event Pipeline and Ranking Calculations

Participation events are published to Kafka topics that multiple services can subscribe to. We considered two approaches to calculating rankings:

• Real-time aggregation
• Periodic aggregation

After evaluating both, we chose periodic aggregation, taking into account Kafka consumer maintenance costs and the project's pilot status.

We used the Confluent Managed S3 Sink Connector to load events into S3, and Argo Workflow to automate everything from data queries to ranking calculations and database writes on a schedule.

Ranking and participation data are also stored in S3 through a separate ETL workflow for analysis. Buzzvil already has a self-service data pipeline, so any backend engineer familiar with YAML can write their own analytics pipeline.

The Kafka-based architecture also made it easier to scale out as increased participation drove sharp growth in API requests, published events, and data volume. After launch, we maintained uninterrupted service even during traffic spikes.

This scalable pipeline helped us successfully complete season one of the Group Account Mission Challenge, which opened to 12 million group account users. Season two is now running reliably as well.

2. Building Our Monitoring System

Why Monitoring Matters

To manage product data flows reliably, we need to monitor both the components that generate source data, such as APIs and workflows, and the infrastructure beneath them. We have to track whether a malfunction or outage has affected the product, whether data has been corrupted, and whether synchronization is working correctly.

Our Monitoring Tools: Datadog and Grafana

At Buzzvil, we primarily use Datadog and Grafana for monitoring.

We collect traces in Datadog by default, so its APM capabilities let us effectively monitor API latency, errors, and traffic.

We use Datadog for overall service monitoring, and Grafana for log monitoring with Loki and metrics monitoring with Prometheus.

Our monitoring coverage was limited when the team first formed. After several incidents, we quickly recognized the need to establish these systems. Fortunately, other teams already had solid implementations we could learn from, which helped us set things up reliably.

Our team uses Datadog dashboards and alerts to monitor components such as APIs and workflows, along with infrastructure including Kafka, MySQL, Redis, and DynamoDB, in real time.

For components that aren't connected to Datadog, such as Kafka connectors, we use Loki and Grafana Alerting.

Through ongoing tuning, we now operate the systems reliably while meeting our SLO targets.

3. Building a Feature Flag System

Background

Reliable product operations and successful feature launches require fine-grained control over data flows. That control helps us respond to risks such as outages, makes testing easier, and enables more agile product experimentation.

As BuzzBenefit has gained more large enterprise clients that are cautious about service changes, and as the number of features has grown, our product operations team has needed more precise feature controls and faster ways to respond to risk. Our team has taken the lead on a feature flag system, from the initial implementation to organizing a task force to develop it further.

What Is a Feature Flag?

A feature flag is a mechanism for dynamically enabling or disabling functionality without changing code. For example, it can limit a new feature to selected users or clients, or immediately disable a feature if something goes wrong.

Feature flags help developers manage and test functionality more effectively and give operators an easier way to control product releases.

Here is the overall architecture we implemented:

We focused on separating the Client interface, which evaluates flags, from the Manager interface, which registers them. This makes it easy to replace either implementation independently later. Feature flags can be used from both the frontend and backend.

What's Next

Feature flags have enabled testing in production for BuzzBenefit: experimenting with features for a limited group of users under real traffic. They also make it easier to release features to specific user groups or clients and collect data.

We plan to extend this foundation to support A/B testing, helping the product organization validate a wider range of hypotheses more quickly and efficiently.

Closing Thoughts

These three examples illustrate what our team does and the environment we work in. Beyond the projects discussed here, we have many others underway or planned:

• Improving the benefits module architecture to speed up development
• Migrating an RDB that handles 1.5 million data writes per day to NoSQL without downtime
• Building version two of our real-time analytics infrastructure

We're also preparing other projects, including initiatives that use AI as an assistant to improve productivity and deliver business impact.

When I first set up the team, I set a personal goal: to build a team with strong skills in designing data-intensive applications. As we broaden our scope and deepen our expertise, we're steadily moving toward that goal.

If you're interested in data-intensive application design, creating business impact through backend development, or growing with a team while working with large-scale traffic and data, please get in touch and apply for our [BuzzBenefit] Backend Developer role (3+ years of experience). Coffee chats are always welcome, too!

I hope this story sparks someone's interest and encourages them to take the next step.

Thank you for reading.