AWS re:Invent 2021, Through the Eyes of Buzzvil’s CTO

re:Invent is AWS’s biggest conference. Held in Las Vegas, it costs roughly KRW 2 million just for a ticket. The event is so large that a single hotel cannot accommodate it: sessions run simultaneously across several hotels. The keynotes unveil the major new services AWS has spent the year preparing. I used to follow the announcements through the AWS RSS feed whenever re:Invent came around, and this year I finally had the opportunity to attend in person. International travel was difficult because of COVID, but I made it there and back safely. I initially wrote these notes for colleagues, hoping they would help anyone attending in the future. Afterward, I thought others might find them useful too, so I decided to publish them here. The tone is perhaps a little casual for an official company blog—please bear with me!

Booking sessions

Once you buy a ticket, you can reserve sessions. I decided to attend fairly late, so most were already fully booked. With around 2,000 sessions to choose from, the website was frustrating: it did not let me filter for sessions with available seats. I later discovered that the AWS Events app does have that filter, which made the search much easier.

Keynotes

There are keynotes every day. The main one, delivered by AWS CEO Adam Selipsky on Tuesday morning, was enormous. Rather than simply announcing products, he first explained why each service was needed, then introduced it in that context. The transitions were smooth, and I was impressed by how fluently he delivered the presentation. The Nasdaq CEO also appeared to talk about how Nasdaq uses AWS, which certainly lent credibility to the story. I found myself imagining the AWS CEO asking the Nasdaq CEO to come and say good things about AWS.

The announcements that stood out to me were Graviton3, the Arm-based processor, and the machine learning services. The first Graviton arrived in late 2018, and AWS has released a new version almost every year since—remarkable progress in just three years. With equivalent performance at a 20–30% lower cost, adoption is growing. For services such as RDS and ElastiCache, switching is as simple as changing the instance type. Web applications face a slightly higher hurdle because they need to be rebuilt for Arm. Until recently, developing locally on Intel while deploying to Arm was inconvenient, but Apple’s M1 is changing that picture. With Arm spreading across both personal computers and servers, the shift feels increasingly inevitable. The pace of Graviton3 development reinforced that impression.

Most of the machine learning announcements centered on SageMaker. AWS is clearly investing heavily in it and promoting it actively. Vanguard and Tapjoy presented substantial production ML use cases built on SageMaker. When the audience was asked who used it, roughly 15% raised their hands. It looks worth considering.

Thursday’s keynote by CTO Werner Vogels was much more technical. At one point he went into the details of deriving signing keys, which was surprisingly deep for a keynote. He is an expert in distributed systems, and since we have been studying Designing Data-Intensive Applications together at work, it was a privilege to see him in person.

Workshops

Get rolling with machine learning fast on AWS DeepRace

Regular sessions are recorded, but workshops are not, so I focused on reserving workshops that still had space. I was not particularly interested in DeepRacer itself; it was available and seemed worthwhile. We sat at tables in groups of six.

After a roughly 15-minute introduction, we were given a link and told to work through it ourselves. That was less interactive than I had expected. Unlike another workshop that supplied accounts, this one required a personal AWS account and provided credits, so I created a new account rather than using our company account. All we really had to write was a reward function for reinforcement learning; everything else was automated. It was impressive how easily we could get a model working. The problem was that training took about an hour. Changing hyperparameters meant another wait, leaving us reading documentation in the meantime. That was not quite what I wanted from a workshop, so I left early.

There were several instructors available to answer questions. If you work through the material beforehand and arrive with questions, I think you would get much more out of the workshop.

Introduction to Working backwards

I had recently read the book, so this workshop caught my attention. I had not reserved a seat, but getting in through the walk-up line was easy. We defined a customer and their problem, chose a solution, and then wrote a press release for it. That included a customer quote and a headline that summarized the idea while making people want to read more. We discussed our work with the person next to us and exchanged feedback.

My partner worked on solutions for the US government. There was not enough time to produce something I was happy with, but the interaction made it a good workshop. It reminded me of the sessions James runs for our Employee Experience team—another reminder of how good those are. I asked whether we could get the slides or run a workshop internally, and was told to contact our account manager. I also asked how Amazon had made this part of its culture. The answer was that senior leaders repeatedly emphasize it and actively support it.

Advertising sessions

There were only a few advertising sessions. They seemed less popular than other topics: the rooms were smaller, and some had plenty of empty seats.

Under the hood at Amazon Ads

This was the most useful session for me. It was early in the conference, and I did not take detailed notes, intending to watch it again later. Three speakers covered use cases, the details of ad serving, and machine learning. They described moving campaign management from RDS to DynamoDB to handle scale, and searching DynamoDB data through Elasticsearch, much as we do. Real-time BERT inference using dedicated inference chips was impressive. The system scores hundreds of millions of ads at tens of thousands of requests per second, with p99 latency of 20 ms.

I had questions for the second speaker, but many others did too, and we kept him talking in a corner for about 30 minutes. I did not fully understand everything, so I got his name with the intention of connecting on LinkedIn and asking more questions once I returned to Korea. I sent the connection request the next day, and he immediately offered to meet while we were still at the conference. It pays to get in touch early! Here are the questions I asked.

Why use Solr rather than Elasticsearch for ad search?

Elasticsearch’s scatter-gather approach returns results after gathering everything, without a way to intervene along the way. Solr can be customized to stream intermediate results.

What is Solr’s latency?

Around 20–30 ms. Most results are cached.

With so many parameters and different queries, can you really cache everything?

Yes—the cache is about 300 GB. It is not invalidated in real time. The application filters the results again, and a query returns around 1,000 ads.

How do you keep the delay in budget updates to one second?

Kinesis Analytics aggregates the data, and atomic counters decrement the budget in DynamoDB. Normally the aggregation window is longer, but when the remaining budget drops below $10, it aggregates every second.

Are there more AWS references for advertising architecture? Google publishes material such as advertising platforms arthitecture; it would be helpful if AWS did too.

He said there was no additional material he could share because it was confidential. His advice was not to copy Google-scale architecture when you are not operating at Google’s scale, but to focus on the problems you have today. He builds what is needed and fixes issues when they arise. I agree, though I was hoping to get a broader picture of where we might be heading—perhaps that was just my excuse.

He had a very pragmatic approach. Amazon, too, seems to prioritize delivering the features it needs quickly over optimizing in advance. There were similarities with Buzzvil: using a general-purpose search engine and Kinesis Analytics for budgets, for example, although we use Redis rather than DynamoDB. One difference was the 300 GB cache of search results, with stale entries handled by filtering in the application. We have focused instead on optimizing the index itself. For more on our approach, see Introducing Buzzvil’s backend technology stack.

Best practices for cloud-based real-time ad platforms

This was a chalk talk with the head of advertising and marketing and two practitioners: about 40 minutes of presentation followed by 20 minutes of questions. These were the best practices they highlighted.

Low cost

• Aim for 100% Spot instances.
• Aerospike.
• Reduce data transfer costs: negotiate private pricing for external traffic, use PrivateLink, and avoid cross-region replication.

Low latency

• SO_REUSEPORT.
• Change from one bidder on 64 CPUs to 32 bidders on 64 CPUs (two CPUs each). They explained the reason, but I did not understand it.
• FastJSON.
• fastHTTP.
• Nonblocking streams.
• Zstandard for Kinesis.
• Optimize Prometheus summaries.

It seems Amazon uses Go for its ad server. I asked about Aerospike versus DynamoDB, but with my limited English, what I took away was the distinction between a managed service and running it yourself. The two people sitting on either side of me both spoke to me in Korean at the same time. We exchanged business cards; both worked in advertising, and it was fun to talk with them.

Applying Amazon SageMaker to real-time advertising workloads

This session featured the same advertising lead and a speaker from Tapjoy, again with 40 minutes of presentation and 20 minutes of questions. The key slide came near the end. Their bidder calls SageMaker inference directly through the SDK, with no Lambda function or separate inference server to maintain, which looked very convenient. Seventy percent of Tapjoy’s workloads run on SageMaker.

• ~5 single-model and ~4 multi-model Amazon SageMaker endpoints handling ~70% of our production ad requests
• Using ~40 c5.2xlarge instances
• Model training and hyperparameter tuning achieved on Amazon SageMaker
• Inference via Client SDK - didn’t use AWS Lambda; no need to expose inference calls publicly
• Model deploys and retraining is performed manually
• Couchbase used to serve user-level features, contextual features stored in memory of our bidding engine

Expo

The expo had booths from a huge number of companies. On the first day I collected T-shirts, then regretted carrying so many around. On the last day I was less tempted by the giveaways and spent more time learning about the companies. There was so much to see that I had to decide quickly, based on a company’s name and a single catchphrase, whether to stop. It made me wonder what Buzzvil’s one-line pitch should be.

Two categories stood out: cloud operations and FinOps, and data platforms such as Firebolt. We already use Spot by NetApp, formerly Spotinst before its acquisition. Apptio had a large booth, and I watched a demo. The price was 2% of the total cloud bill, which felt substantial; I wondered whether the value justified it. It would be useful at a lower price. There were also many analytics companies I had never heard of. With the success of Snowflake and Databricks, competition in this space seems fierce.

AWS had booths for EC2, containers, AI/ML, and other areas, where solutions architects were available to discuss questions at a whiteboard. I had not prepared any questions and missed the opportunity. I recommend bringing a list of things you want to ask.

Megazone events

Megazone, Buzzvil’s MSP partner, rented a pub and hosted two free evening networking events. The conference is so large that it is hard to run into even your own colleagues during the day, let alone people from other companies. These events were a useful chance to meet people from Korean companies.

Other observations

The conference spans several hotels, with the Venetian serving as the main venue for keynotes. Because of COVID, there were fewer venues this year: the Venetian, Wynn, Encore, and Caesars Forum. A pedestrian bridge near the Venetian keynote hall leads to Caesars Forum; shuttles for the other venues leave from the first floor. For convenience, I mostly stayed at two venues. Breakfast and lunch are included and were good. Having lunch outside at Caesars Forum was a welcome chance to get some fresh air.

Not every seat is reserved, so you can also join the walk-up line. For popular sessions, arrive 30–60 minutes early; for less popular ones, you can often just walk in. I could not get into the EKS session. Kubernetes is clearly popular.

AWS speakers frequently wove Amazon’s leadership principles into their presentations, especially Customer Obsession. Amazon’s vision is “to be Earth’s most customer-centric company, where customers can find and discover anything they might want to buy online, and endeavors to offer its customers the lowest possible prices.” Seeing that commitment repeatedly put into practice was a lesson in itself.

Thursday’s re:Play party featured DJ Zedd. Seeing Squid Game imagery in the animation gave me a little burst of Korean pride. There were three large tents, one with free food and drinks, so there is no need to eat dinner before you go.

New services

Here are some of the new services that might be relevant to Buzzvil. There are many announcements, so I will leave out detailed explanations. SageMaker alone had a long list of updates; I have included only Canvas here.

1. Amazon ECR adds pull-through cache repositories
2. AWS Karpenter v0.5 is generally available
3. Amazon Athena previews ACID transactions with Apache Iceberg
4. Amazon EC2 C7g instances powered by AWS Graviton3 processors
5. Preview of Amazon EC2 Trn1 instances
6. Public preview of Amazon MSK Serverless
7. Amazon Redshift Serverless
8. Amazon Kinesis Data Streams on-demand
9. Preview of Amazon EMR Serverless
10. Amazon SageMaker Canvas: build accurate machine learning models visually, without code
11. Amazon DynamoDB Standard-Infrequent Access table class: save up to 60%
12. Amazon S3 reduces prices by up to 31% for three storage classes

Closing thoughts

Session recordings remain available online for a while. If your goal is simply to absorb information, watching online may be better: you can skip the introductions and speed up playback. I went looking for opportunities that only being there could offer, such as workshops and conversations with speakers. Meeting a presenter separately gave me information I would not have found in the recording, and made the trip worthwhile. I hope I get to go again.

Apply to join Buzzvil’s engineering team (click)

Have a coffee chat with Buzzvil’s tech recruiter (click)