A Departure Retrospective from Jacob

Introduction

Hello. Gloria, who was there at both the beginning and the end of my time at Buzzvil, encouraged me to write this departure retrospective. Netflix has a practice of asking departing employees to email colleagues about why they are leaving, what they learned, and what could have been better. They call it a departure postmortem. To close out my own time at Buzzvil, I decided to look back on these two years in the same way.

1. Why Am I Leaving?

In one sentence: to keep growing. A fuller explanation requires a rather long story beginning in graduate school. Two courses I took there—discrete optimization and reinforcement learning—stand out. They gave me not only knowledge but also ways of looking at life.

Back when I was in LA graduate school...
Source: What Are You Doing Here? (KCC’s Park Chan-ho commercial)

Choosing a Career as an Optimization Problem

From an optimization perspective, choosing a career path means selecting the available option that best maximizes what you value. Everyone values different things: benefits, salary, growth, aptitude, or a preferred field. We often consider several together, perhaps trying to maximize something like (0.7*salary + 0.3*benefits).

Different career options
Source: The Importance of Values in Career Decision Making

Every career path has costs, time requirements, and uncertainties. If I suddenly decided to become a lawyer, I would need to invest at least three years and the associated tuition. Moving to another technology company as a developer requires time and effort researching companies and preparing for interviews. Every choice has benefits and costs; choosing a career means considering them together to make the best decision for yourself.

The answer depends on what you value. I seek choices that maximize growth, because experience has taught me that growing is when I feel most satisfied. Looking back, I remember difficult periods as meaningful when I grew through them, while comfortable, stress-free periods leave me disappointed if I did not. I also believe growth gives me greater negotiating power over salary, working conditions, and the domains I work in. Pursuing growth should therefore make other values easier to pursue in the long run.

Which career will maximize growth? Salary is written explicitly in an offer letter, making it relatively easy to compare options and choose the highest. But “How much will I grow in this career?” is hard to answer without experiencing it. We cannot predict it precisely and select a mathematically optimal option. We must instead rely on reasonable ideas and heuristics.

Exploration and Exploitation in Reinforcement Learning

I am particularly fond of several ideas from reinforcement learning. One is the distinction between exploration and exploitation. Exploration means venturing into unfamiliar territory to gain new information; exploitation means making the best choice based on what you already know. Relying only on exploitation confines us to strategies we have already discovered. Better long-term results require exploration at appropriate times and in appropriate proportions. Reinforcement learning research itself studies how, when, and how much to explore.

Choosing a restaurant: somewhere you have never tried may be more satisfying.
Source: https://medium.com/analytics-vidhya/the-epsilon-greedy-algorithm-for-reinforcement-learning-5fe6f96dc870

It has been almost three years since I finished graduate school and began working as a developer, and I still feel there is far more to learn. The less you know, the greater exploration’s expected value. I believe I am at a stage where exploration deserves more weight than exploitation. For me, venturing into a new area is likely to be the choice that maximizes long-term growth.

If exploration is a good strategy for long-term growth, the next question is which adventure to choose. There are countless possibilities. Recently, my options included:

1. Changing teams within the company to solve new problems and gain experience.
2. Changing roles within the company—for example, to PM, DevOps, or data engineer—to gain experience.
3. Moving to another company to solve new problems with a new team.

Let me expand on the third option. Which company and team would help me grow most? There is no definitive answer, but possible guides include the fastest-growing company, the strongest technology, or a team with outstanding colleagues. People define growth differently, so each person can reach their own answer. To make a choice aligned with my goal, I needed to consider what growth means to me and which option would best support it.

Curiosity-Driven Exploration

I am intensely curious. In board games, for example, I often make the irrational decision to pay a “curiosity tax” just to see someone’s hand because I desperately want to know whether they are bluffing. Curiosity arises around something we cannot predict well because we do not understand it, yet want to know. It can provide a criterion for answering “Which choice will help me grow most?”: choose the path that makes you most curious. Reinforcement learning even has a paper titled Curiosity-driven Exploration by Self-supervised Prediction, which attracted attention from publications including The Wall Street Journal and Wired.

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life.

In many real-world situations, an agent receives very few external rewards, or none at all. Curiosity can then act as an internal reward signal, encouraging it to explore its environment and learn skills that may become useful later.

Source: the abstract of Curiosity-driven Exploration by Self-supervised Prediction

The paper trained a Super Mario agent through curiosity-driven exploration without external rewards.
Source: noreward-rl GitHub

While working on ad recommendation and performance optimization at Buzzvil, I often wondered whether I was doing well enough and how other excellent teams around the world made decisions and developed software. Other companies’ engineering blogs and papers never quite satisfied that curiosity. Then I had a coffee chat with the lead at the company whose offer I eventually accepted. After ten years at Google, this lead was now “spreading a sound engineering culture” and “solving diverse problems across the full search stack.” Hearing that made me want to see what problems they were solving and how. That curiosity was hard to resist, and it is why I am leaving Buzzvil.

2. What I Learned Here

Great Colleagues Are the Best Benefit

People say, “Great colleagues are the best benefit.” At Buzzvil, I felt how true that is.

Before joining, I noticed “great colleagues” appearing repeatedly in employee reviews and wondered who these people could be. After two years here, I can say the company is full of colleagues who are kind, smart, and deeply engaged in their work. As a result, I rarely experienced stress unrelated to the work itself, and always felt secure in knowing the team was working toward our goals together.

Many people are committed to growth, and I often saw colleagues inspire one another and grow together. Engineering study groups and knowledge sharing are active and easy to join. I participated in design-pattern and domain-driven design (DDD) groups, and gave a Dynamic AD presentation at an advertising-domain seminar for new hires. Some colleagues speak at leading Korean conferences such as PyCon and Droid Knights. Perhaps that environment encouraged me: I recently applied to speak at PyCon Korea 2023 and was accepted. Smaller gatherings flourish too, such as watching engineering seminar videos together over lunch. Beyond formal groups, many wonderful people simply work hard and keep growing. Being around them makes the value of great colleagues tangible.

Study-group invitations. Many colleagues voluntarily organized and ran groups.

Retrospectives That Solve Problems and Improve How We Work

My favorite part of Buzzvil’s culture is retrospectives. Experiencing retrospectives that genuinely solved problems convinced me of their value so strongly that I started doing them in my personal life too.

At the end of 2022, my recommendation team merged with the engine team, which primarily handled ad allocation. This let us approach ad-performance optimization more effectively with the allocation context in mind. The downside was more context to learn, more kinds of issues to handle, and greater cognitive load. Meetings and other communication also seemed less productive.

As those disadvantages became clearer over several retrospectives, we split the team’s work between ad-performance optimization and reliability. Engineers handling reliability could then concentrate on it without also tracking optimization work. We gradually improved meeting formats and task workflows through retrospectives as well. Once the process had matured, we reflected on how much productivity and focus had improved since the merger’s early days. It reinforced the importance of retrospectives for me.

Retrospective examples. Our team held regular retrospectives using Confluence and Figma.

Communication in Service of a Goal

Communication is a means to an end. When absorbed in a conversation, it is easy to drift into discussion that does not advance the goal. We might begin by clarifying a solution to problem A, then realize we are debating whether problem B matters. Such conversations can be useful, but often consume time without achieving the original objective. Conversely, well-prepared, goal-conscious discussions often both achieve their purpose and produce meaningful ideas.

There are many ways to communicate. Asynchronous discussion around a document may serve the goal better than gathering everyone in a meeting. Listening carefully over a cup of tea might be more effective. Sometimes giving people time to learn independently, without explicit discussion, is better still. Choosing the method most effective for the goal can be wiser than choosing what feels comfortable or familiar.

Asynchronous communication examples. Our team discussed technical direction through RFCs, or Requests for Comments.

Colleagues with strong communication habits helped me improve my own. In particular, I now pay more attention to:

1. What goal I want to achieve.
2. Which communication method would best serve that goal.
3. Whether the conversation I am having is actually helping achieve it.

As a result, when an issue arises that would once have made me call a meeting immediately, I can instead communicate through a document, wait and observe, or think it through further myself and discuss the resulting work.

3. What Could Have Been Better?

Netflix’s departure emails reportedly frame this section as “If Netflix had done this, I would not have left.” As explained above, I see this as a time when exploration has greater expected value, so I probably would have sought a new environment within a few years regardless of conditions. I am not leaving because of dissatisfaction, and there is no “If only Buzzvil had done this, I would have stayed.”

Still, leaving behind ideas that might help the company improve seemed more useful than saying I had no regrets at all. I discussed one point several times during my exit conversations, and have summarized it here.

Many ad tech companies work on ad-performance optimization. Papers and engineering blogs show how global companies such as Criteo, LinkedIn, and Google approach it. They appear to have invested considerable time and effort and developed sophisticated solutions.

Buzzvil began tackling this problem in earnest this year and achieved meaningful results. Because it was new to everyone, we improved through repeated discussions and experiments based on many ideas. At times, though, I felt there was no clear agreement on the roadmap and priorities. Perhaps that is why I struggled to picture our solution one or two years ahead, or how advanced our approach would be.

I find ad-performance optimization exciting enough to want to solve it better and compete with other ad tech companies. If we had shared a clear agreement on the roadmap and priorities, and I could look forward to what our solution and results would be in one or two years, I might have wanted to stay longer and achieve those results together.

I do not take that to mean “We must quickly establish a roadmap, priorities, and a detailed development direction.” I believe the team has already made the best choices it could under constraints such as limited staffing and resources, urgent issues, new business requirements, and alignment with related teams. Investing in a detailed roadmap that will inevitably change may be less effective than a flexible, short-term approach that solves problems step by step. Please take this simply as a light reflection on something I occasionally wished for.

Beyond that, I have no regrets. I enjoyed working with outstanding colleagues, and I leave grateful that so many people helped me grow.

4. What Comes Next?

I will be a developer on the search team at my new company. This is how the team describes its work.

We aim to deliver a high-quality service across diverse content through reliable systems and a range of ranking models. Our focus includes developing search features that improve the user experience, experimenting with and applying ranking models to improve quality, and building and improving search infrastructure for a reliable service.

I expect plenty of interesting challenges and plan to write about my new work on my personal blog. Thank you for reading this long reflection!

Apply for an engineering role at Buzzvil

Have a coffee chat with a Buzzvil technical recruiter