Introducing Buzzvil’s Backend Stack

Hello, I'm Liam from Buzzvil's DevOps team.

As a rewarded ad-tech platform, Buzzvil works to deliver optimized ads to a monthly audience of more than 20 million and maximize performance. We recommend relevant ads using audience behavior collected from multiple data sources, and use rewards to reduce friction along the journey to purchase. Over time, the volume and speed of traffic and data we handle have grown rapidly. To work more flexibly and efficiently, we have spent several years splitting our monolithic service into dozens of microservices.

We use a wide range of technologies to operate this complex advertising system, and share our stack on StackShare. But a list alone does not explain how each technology is used. This post briefly introduces the backend stack behind our high-volume advertising platform.

Programming languages

Most backend microservices are written in Python or Go. As of October 2021, our internal service catalog lists 45 backend microservices: 21 in Python, 16 in Go, seven in Node.js, and others. Python and Go adoption within the product team is fairly balanced, but since microservice-oriented development became widespread, new gRPC services in Go have been appearing slightly more often.

Python has been Buzzvil's primary language for the longest time, with Django as the main framework. Its excellent ORM makes business logic around varied models easy to write, and it remains popular internally. Microservices often focus on a specific domain and do not need all of Django's capabilities, so we also use lighter frameworks such as Flask and FastAPI. More recently, we have been writing more gRPC-based Python services.

Previously, a Django-based Python web server handled most APIs, including ad allocation. As traffic grew, infrastructure usage increased substantially. Porting ad and content allocation—the largest part of that workload—to Go brought major performance gains (Developing a Go Server). Since then, Go has been a frequent choice for high-performance APIs. Its simplicity fits Buzzvil's engineering philosophy, and goroutines provide excellent concurrency support. There is no absolute rule for language selection, but we recommend Go when performance and resource use deserve particular attention.

Engineers who prefer each language form expert groups to discuss best practices for libraries and project structure, and improve style guides and build pipelines.

Because our microservices approach means creating services frequently, we maintain templates to take projects from creation to deployment quickly. They include recommended configurations for each language, helping even less experienced users set up projects easily. We use Cookiecutter and plan to integrate templates with Backstage for easier project creation.

The IDL repository

We chose gRPC for inter-service communication for more than performance. From the initial microservices blueprint, we expected many services, and manually maintaining API documentation for each would struggle to keep pace with change.

gRPC supports Protocol Buffers as an Interface Definition Language (IDL). Teams define a new service's interface and RPC messages in protobuf and open a PR in the IDL repository. Both the provider and consumer teams review it to verify that it reflects decisions made in meetings or design documents. As API definitions evolve, Buf checks whether changes break backward compatibility.

Once a proposed definition is merged, the repository's CI generates client and server code for each language and publishes it to the relevant package repository. Go code is packaged as Go modules with tags pushed directly to GitHub; PyPI and NPM packages go to our internal package registry. For environments or legacy clients where gRPC is difficult to use, we add configuration to transcode gRPC into JSON APIs. The required proto descriptor sets are generated and uploaded to S3 as well; see Use gRPC and Get REST for Free!?.

Databases

Our main databases are MySQL on RDS, DynamoDB, Elasticsearch, and Redis on ElastiCache. The principle is that each microservice owns and manages its domain data in its own database rather than sharing a central database.

Without special requirements, most services use the relational database RDS for MySQL. We monitor it with pmm-server and CloudWatch, and use GitHub's gh-ost for zero-downtime schema changes.

We actively use fully managed DynamoDB for workloads requiring high read/write performance or significant scale, such as ad frequency caps and reward issuance. Monitoring provisioned read/write capacity and query throttling makes scaling straightforward and minimizes operational overhead, so it serves many use cases.

Elasticsearch quickly selects candidate ads matching numerous targeting conditions for a user request. Redis provides caching and real-time statistics.

Service provisioning

Except for a few AWS Lambda endpoints, most services deploy to Kubernetes. Our goal is that any service can be deployed once it builds as a container, regardless of its internal technology. To manage resource manifests consistently, we maintain Helm chart templates with recommended settings and guide developers on important deployment considerations such as graceful shutdown and readiness/liveness probes.

Like the IDL repository, company-wide Helm charts live in one monorepo. Service developers adapt the template to their application's requirements, and DevOps engineers review the charts.

We use an in-house Spinnaker installation to configure different deployment approaches by team. Managed Pipeline Templates reduce the burden of maintaining pipelines for each environment.

Deployments are triggered through ChatOps. A request to our internal botkit-based chatbot starts deployment and reports the change summary and status in Slack. This makes deployments visible and naturally onboards people into the process simply by letting them follow the deployment channel.

Service mesh

We chose Istio as our service mesh from the beginning of our microservices adoption. The goal was to move complex client logic for inter-service communication into the infrastructure so services could focus on core business logic. We intended to use gRPC over HTTP/2 extensively, but Kubernetes Services offer limited load balancing for long-lived connections (reference). Handing that concern to the mesh instead of implementing it in every client was a major advantage.

Advanced traffic features such as gRPC–JSON transcoding, traffic shifting, and mirroring also help test new features and move capabilities out of the Django monolith. After moving functionality to a new microservice, we mirror existing traffic to verify equivalent results and capacity. We then gradually shift a proportion of traffic to the new service until migration is complete.

Observability

Buzzvil uses Datadog, Prometheus, and Grafana for observability. We have used Prometheus and Grafana extensively since establishing Kubernetes. As Kubernetes accounted for more of our infrastructure, however, high metric-label cardinality made Prometheus memory difficult to manage and caused frequent OOM kills. We introduced Thanos for highly available Prometheus, stored metrics in S3, and compacted and downsampled older time-series data.

Operating the stack ourselves was still burdensome. As microservices multiplied, demand also grew for visibility into inter-service calls through distributed tracing. We considered Jaeger, which fits well with Istio and Grafana, but chose managed Datadog to reduce operational work. In particular, Datadog APM made identifying cross-service bottlenecks, managing SLO metrics, and detecting anomalies much easier. We also use Incident Management and PagerDuty integration.

Logging

We use logs widely, from service visibility and debugging to collecting core business metrics, producing more than 1 TB a day. Services emit metrics through log streams, and we operate several forwarding mechanisms for Kubernetes workloads.

Services created before our extensive Kubernetes adoption run fluentd as a sidecar in each pod, forwarding logs to S3 or Kinesis Data Streams and onward into the data lake. Newer services follow the Kubernetes Logging Architecture: write everything to stdout and stderr, then use node-level fluent-bit to forward logs through a fluentd deployment into S3. For JSON structured log lines, fluent-bit routes records to agreed data-lake paths according to a reserved type key.

We also operate a Loki stack for short-term log storage and queries over a few days, mainly for debugging and troubleshooting.

Closing thoughts

That was a brief tour of Buzzvil's backend stack. We have addressed many challenges while organizing a complex advertising system into domain-based microservices. This post covered the overall stack; when the opportunity arises, we will explore individual areas in greater depth with concrete examples.

There are still many problems to solve, and we are looking for colleagues who would like to tackle them with us. If you are interested, feel free to ask for a tea-time conversation!

Apply for an engineering role at Buzzvil

Have a coffee chat with a Buzzvil tech recruiter