An Engineer’s Guide to Shopping for Open Source

1. Introduction

Startup engineers who rely heavily on open source are constantly weighing choices. Just as shoppers compare prices and read reviews, we compare tools against our criteria and learn from others’ experiences. Sometimes, almost like compulsive shopping, the comparison itself becomes enjoyable.

When we began building HoneyScreen’s server, we faced many such decisions. One particularly useful reference was an article about Instagram’s server architecture. Instagram chose its systems according to three principles:

Keep it very simple.

Don’t reinvent the wheel.

Use proven, stable technology whenever possible.

I was astonished that just three engineers maintained a service with more than ten million users. Startups need results from limited resources, so this approach felt especially useful. All three principles ultimately point to one idea: keep the system as simple as possible, with minimal customization. That means using existing technology well—and preferably choosing technology that is proven and stable.

What interested me was how to find the right wheels and fit them properly. Distinguishing a rotten wheel from a sturdy one suited to the job isn’t as easy as it sounds. Choosing well at the outset also matters: changing wheels after the car is already moving is difficult. Anyone operating a service will recognize how hard it can be to change a running system without interrupting it.

So how do you find a good wheel? It helps to have someone to ask, but most of us search online. With gratitude to Google, that reliable ray of light for an engineer who doesn’t know enough, here are the criteria I use.

Choosing search keywords

The most important part of searching is choosing the keywords. I first appreciated this in a university class about patents. A guest lecturer from a patent-related company demonstrated how to find prior patents, combining keywords in many different ways. It made the seemingly obvious point memorable: keyword choice is fundamental to searching.

Suppose A and B are libraries with similar functionality. Here are some queries I try first:

[blockquote]

• A B
• A vs B
• A B comparison
• A B which is better
• A alternative
• Why A sucks
• Why moved from A
• Switch from A [/blockquote]

It may look simplistic, but “vs” often produces useful comparisons. “Alternative” can uncover a new option B that does something similar to A. Searching helps compare known candidates and discover ones you hadn’t considered. A broad vocabulary helps: keep asking which words will lead to the answer you need.

Looking for deal-breaking drawbacks

This is where “Why A sucks” comes in. Think of people who say they like everything about an iPhone except its non-removable battery, which is enough to rule it out. A drawback can decide the choice.

Open-source project pages usually list their strengths, so you need to investigate weaknesses separately. Adopting a tool for a desired feature only to discover an unacceptable drawback later is a recipe for regret. Looking for drawbacks also teaches you what to watch out for.

Redis, for example, is powerful, but you need to understand its single-threaded behavior and the possibility of data loss. Incorrect failover in a replication setup can lose data. Dae Myung Kang’s presentation “Redis: Use It Wrong and You’re in Trouble” explains this well. Twilio also documented an incident involving partial data loss.

Checking whether development is active

I look at repository commit history to see whether maintenance continues. For example, if the last commit was more than six months ago, I may consider the project inactive. It might work today but become an obstacle when a bug appears or another library upgrade introduces incompatibility. Very simple libraries, such as JSON serializers, are exceptions: they may need little maintenance and be easy to replace.

A project may be maintained now but stop soon. Judging that requires some intuition. I look at contributors’ profiles and the general atmosphere of the repository. Small libraries maintained by one person can be more vulnerable to losing maintenance.

Checking whether it has reached 1.0

A version above 1.0 isn’t automatically safe, nor is a version below 1.0 necessarily incomplete. Still, reaching 1.0 suggests the author feels confident recommending real-world use. Node.js and SQLAlchemy are examples of open-source projects widely used in production while still carrying 0.x versions. A later move to 1.0 may, however, introduce breaking changes.

I consider age alongside version numbers. SQLAlchemy, for example, spent nearly ten years on 0.x versions. One practical rule I use: if the two latest releases are 3.9.28 and 4.0.0, I choose 3.9.28. A new major version signals substantial changes and potentially more bugs. A stable release is often preferable to the newest one. I tend to feel more comfortable after a major release has been out for about six months. Browsing GitHub issues also helps identify noteworthy problems.

Looking for real-world references

Evidence that recognizable organizations use a library provides some validation. When selecting HoneyScreen’s web framework, we considered Django and Flask. One major reason we chose Django was its references: Disqus, Instagram, Pinterest, Eventbrite, and others used it successfully. We had difficulty finding comparable information about Flask deployments at that scale.

Many people used Flask too, but examples at the scale of Django’s prominent adopters were harder to find. This is absolutely not a claim that Flask is bad; it describes our comparison in 2013.

Checking Google Trends

If I still can’t choose between A and B, I turn to Google Trends to compare search frequency. Its trend charts provide another piece of objective information to inform the decision.

2. Closing thoughts

People will disagree about which open-source tools to use. Choosing good tools doesn’t by itself make someone a good engineer. But we do need to identify tools likely to cause problems. A thoughtful initial choice can reduce unexpected production issues and help us develop quickly and reliably.

Writing this makes me feel a little guilty about simply using what others have built. I hope to make my own contributions to open source someday, and will keep working toward that.