A bug appears

One of the moments developers dread most is an urgent Slack notification on the weekend. That’s why unwritten rules such as “no Friday deployments” and “no Friday rollouts” exist. Yet last Sunday afternoon, a notification mentioned our entire client team. A partner in Dubai reported that clicking content in the feed did not open its destination.

Buzzvil provides partners with products that display content and rewarded ads, sharing the resulting advertising revenue. Content makes the feed more than a stream of ads: it offers something worth reading, creates additional value for users, and acts as the glue that keeps them engaged with the product. Broken content links were therefore a serious issue.

Fortunately, the partner discovered the problem during QA. The message arrived on Sunday because Sunday was a working day there. That didn’t mean they worked without a break all week: their weekend was Friday and Saturday, with the working week running Sunday through Thursday.

Finding the cause

Several engineers gathered to investigate. When a user clicks content, the client requests a redirect URL from the server and includes a checksum as a parameter. To verify the request’s integrity, it uses a cryptographic algorithm to create that checksum from user information, campaign information, rewards, and other parameters.

The checksum being sent was wrong. The server calculated:

• 6371ab08199c6fcaf73cdc5ed17c712d

But the client sent:

• ed85f6f40927a1b61e0702cc43c9dad1

According to the partner’s email, it happened with Arabic content or when the device language was set to Arabic.

1. Our first hypothesis was that the content title was encoded incorrectly when creating the checksum. But the title of the reported content was in English, ruling that out.
2. Our second hypothesis was that the client and server escaped whitespace differently, though that also seemed unlikely.

We set a breakpoint where the checksum was generated and began debugging. Here is an example of Buzzvil’s checksum generation:

public static String generateChecksum(String deviceId, String contentsName, String userToken, int reward) {
    String str = String.format("example:%s:%s:%s:%d", deviceId, contentsName, userToken, reward);
    return encrypt(str);
}

It uses the device ID, content name, user token, and reward value. We substituted the content, user information, and reward provided by the partner:

• example:20014fea6bcc820c:[tumblr] tumblr Pictures:2ahUKEwjrmqr5pv:1

Calculating the checksum with these parameters worked perfectly, producing the same value as the server. We tried removing spaces and adding escape characters, as suggested by our second hypothesis, but could not reproduce the incorrect client value.

Then we returned to the clue that the bug occurred with Arabic as the device language. We changed our test device to Arabic and tried again. Immediately, the client produced the exact incorrect checksum the partner had reported:

• ed85f6f40927a1b61e0702cc43c9dad1

The parameters seemed identical, so why was the result different? On closer inspection, the string passed to the cryptographic function had changed slightly:

• example:20014fea6bcc820c:[tumblr] tumblr Pictures:2ahUKEwjrmqr5pv:١

Can you spot the difference? With the device set to Korean, the final digit appeared as the familiar 1. With Arabic selected, it appeared as an unfamiliar character. Searching Google for “arabic number” brought up similar characters, and one result showed a comparison table of Arabic and English numerals.

We learned that the Arabic numeral system used here represents digits with symbols such as ٠,١,٢. Those digits were why the checksum calculation differed.

Now that we knew the cause, we needed a fix. The Android client used String.format to convert a number to a string, and that conversion was the culprit.

Without an explicit locale, String.format() builds the string using the user’s locale or the system default. With Arabic selected, numeric conversion used characters such as ٠,١,٢ rather than the Western Arabic digits 0, 1, 2 that we expected. These are called Eastern Arabic numerals. For a user-facing string, displaying them in an Arabic-speaking locale is perfectly valid—and may be preferable. But a checksum must remain consistent regardless of locale. Allowing its input to change with locale caused this bug.

The solution

There are two ways to fix this:

• Set a locale (recommended).

    String.format(Locale.US, "%d", 1);

• Use %s. This also converts the number to the string 1 regardless of locale.

    String.format("%s", 1);

Running a global service sometimes brings problems you would never have anticipated. This one taught me that Arabic numerals have an Eastern variant. Isn’t it fascinating that debugging can teach you about another language’s writing system?

In fact, we found this bug relatively quickly because Brice had encountered and solved exactly the same issue at a previous company. Now, if anyone at Buzzvil encounters a bug caused by Arabic numerals again, they’ll know where to look. So will you. Better still, knowing that String.format without a locale can be dangerous may help you prevent the bug altogether.

As programmers, we gradually collect more examples of code that makes us nervous on sight. I take that as a sign that we’ve learned to work more carefully and avoid familiar pitfalls. Here’s hoping we don’t meet this particular bug yet again.