Finding Constellations in Photos: Building a Plate Solver, Seven Years in the Making

1. Introduction

Hi, I'm Brice, a frontend developer at Buzzvil. This tech blog is usually full of serious technical stories, so this time I'd like to share something a little more romantic. How often do you look up at the stars?

When I was a child, my father taught me the Big Dipper, Cassiopeia and Orion. The Big Dipper and Cassiopeia always sat on opposite sides of Polaris in the northern sky, and what you could see, and where, changed depending on when you looked. High in the sky they looked small, but when they hung over a mountain ridge they looked huge, and I used to wonder whether they were really the same constellations.

Orion is supposed to be a winter constellation, yet I saw it in the night sky more often than I expected. In high school, when I looked up on my way home after evening self-study, Orion was often there to greet me. Even in Seoul, where far fewer stars are visible than in my hometown, Orion still shone brightly, so I could find it on my way home from university assignments or from hanging out with friends.

Which constellations are in this photo? You already saw the answer the moment you opened this post, didn't you?

Whether I was looking at the night sky itself or at a photo of it, I always wondered which stars were in it and which constellations they formed. As I studied, I found out there were far more stars and constellations than I had known, and that long ago people used these stars to work out where they were and what time it was.

2. How it started: a 2019 workshop photo from Saipan

Buzzvil used to hold global workshops abroad. The June 2019 global workshop was in Saipan, and one photo posted on Slack at the time caught my curiosity. It showed people on a beach looking up at the night sky, but I was more curious about the stars in the background.

I looked them up myself in a stargazing app, took a screenshot of the app, and left this in the thread:

I was curious about the stars in the background, so I looked them up. It was the southern sky!

Left: the Saipan photo posted on Slack in June 2019. Right: the same sky found in a stargazing app (Constellation Map). I found the same arrangement of the eight stars visible to the naked eye in the photo (yellow circles) in the app.

Seven months later, I added something new I had learned about that star to the same thread:

I just learned that the bright star in the photo goes by names like "Rigil Kentaurus" and "Alpha Centauri," and among the stars you can see with the naked eye in the night sky, it's the closest one after the Sun ...

There is an excellent service called Astrometry.net that tells you where in the sky the stars in a photo are, so I uploaded the Saipan photo. But it couldn't solve it, because people, the beach, clouds and other things on the ground were in the shot along with the stars. That made me think it would be nice to build a tool that works well on photos taken by ordinary people like this one. I left this message in the machine learning study group that was running inside the company at the time:

One thing I always wanted to build once I learned machine learning was a way to get more information out of the stars in a photo (e.g. latitude/longitude, time, direction). Will I ever be able to build it? lol

August 19, 2019

3. Studying: how plate solving works (2022)

To build it, I first had to understand it. I read the paper behind Astrometry.net to learn how it solves photos. In astronomy, working out where in the sky the stars in a photo are is called plate solving. The name comes from the old days, when astronomers found stars on glass photographic plates of the night sky.

Left: the part of a Harvard College Observatory photographic plate where Williamina Fleming first discovered the Horsehead Nebula in 1888 (red circle). Right: the Horsehead Nebula photographed today. (Left: Harvard College Observatory photographic plate. Right: ESO, CC BY 4.0)

Based on what I had learned, I gave a talk titled "Introduction to Plate Solving" at our internal Dev Seminar in June 2022. I demonstrated plate solving by hand on a few night sky photos, and, drawing on the papers and talks published by Astrometry.net, shared what structures and algorithms would be good for implementing it on a computer.

A plate solver does four things:

1. Find stars in the photo: look for points brighter than their surroundings.
2. Find the same arrangement on the star map: turn the shape formed by a few stars into a "fingerprint" and look for similar fingerprints in a star map prepared in advance.
3. Check whether it's right: overlay the star map on the photo as the match suggests and see whether other stars line up too.
4. Draw: draw star names and constellation lines on the photo.

The key is the fingerprint in step 2. People look at a constellation's shape and say, "Oh, Orion!" To have a computer do that, you need a "shape fingerprint" that doesn't change no matter how the photo was taken.

The method in the Astrometry.net paper (Lang et al., 2010) uses four stars (a quad). Move, rotate and scale the quad so that its two farthest stars, A and B, sit at (0,0) and (1,1), and what's left is four numbers: the coordinates of the other two stars, C and D. Whether you rotate the camera or use a telephoto lens, those four numbers stay the same.

The same fingerprints are prepared on the star map side in advance, and we look for the one most similar to the photo's fingerprint. Once found, the pairs of those four stars give the transformation between photo coordinates and sky coordinates, which is used to draw the surrounding stars and constellations on the photo.

While preparing the talk I learned one more thing: constellations alone can't pin down the date and time a photo was taken, because the same sky comes back two hours earlier every month.

I closed the talk like this:

Based on what I've learned, I want to build my own plate solver. For now, as a PoC that just checks whether what I learned works: working with bright stars, working even when the ground or clouds are in the shot, and if it's hard to make it work for every photo, at least working for one or two photos.

4. First attempt: the 2023 Dream Challenge

Buzzvil has a program called Buzz Challenge, where each of us picks a challenge every quarter. There are two kinds: the Focus Challenge, which cheers you on as you set and reach an ambitious goal, and the Dream Challenge, where you share the failures and trial and error along the way. I felt it was finally time to actually build it, so in the first quarter of 2023 I signed up for building a plate solver as my Dream Challenge and spent three focused months on it.

I originally wanted to use four-star quads like the paper. But implementing the logic turned out to be hard. Time flew, the promised three months were nearly over, and the presentation was a week away. To make the deadline somehow, I simplified the goals as much as I could:

• Give up on four-star quads and search with simpler three-star triangles.
• Skip picking the single most accurate answer among the candidates, and only show that the right answer is among them.
• Count the slimming-down needed to host it for anyone to use as a failure this time, and put it off.

But the 9,000 stars produced as many as 2.84 million triangles, and far too many of them looked like the triangles in the photo.

The shape value of a triangle made from three stars in the photo (cross), and a zoomed-in view of the star map triangles around it (blue dots). There are more than 20,000 similar triangles inside the small circle alone, and the right answer (green) ranks 3,049th by similarity.

After many attempts, on April 20, the day before the presentation, a run tested on the Big Dipper finally returned the right answer as the top candidate, after 5 hours and 35 minutes. I started to think, "Maybe I can actually make it work before the presentation?" Just before the talk, I tested it on Orion and got it down to 2 minutes and 41 seconds. But it didn't work for every photo, it couldn't tell which of the candidates was correct, and I never hosted it, so I had to declare it a failure.

I ended the talk with the failure and the words "Teaching a computer is hard." But to teach something precisely, you have to know it precisely. In the end, I couldn't teach it precisely because I didn't know it precisely.

Looking back at that code while writing this post, I found the causes were layered. The star map only built triangles within small patches of the sky (about 5 degrees in radius), so most of the triangles of the Big Dipper, which spans 25 degrees, weren't in the star map at all. Even the correct triangles that were in the star map ranked 632nd among more than 3,000 similarly shaped candidates, and the final check only passed when the angles were practically identical, so it answered "no" even when given the right answer. But at the time I didn't know the exact causes, and tracking down and fixing all of them one by one would have taken a lot of time, so it was hard to go further.

I promised myself I'd improve it someday, but a lot of time went by. Then the changes in the world gave me the courage to try again.

5. Second attempt: 2025, with ChatGPT

By 2025, AI had become good enough to try debugging with, and I wondered whether AI could help me quickly figure out what was wrong and fix it. I described the situation to ChatGPT, it gave me scripts to check the state, I ran them and reported the results back, and we fixed things one at a time like that.

When I asked how to detect stars in a photo, ChatGPT immediately produced Python code using OpenCV, and when I asked it to convert that to OpenCV.js so it would run in the browser, it did that too. The same went for code that connects the detected stars into triangles. Still, in my talk I made a promise to myself: "Let's not become someone who just does what the AI says and then presents it."

This time I went back to four-star quads as in the paper. I split the sky into 192 cells of equal area (HEALPix) and cut down the number of quads by building them only from nearby stars within each cell. For the fingerprint search I used a KD-tree. Implementing HEALPix and a KD-tree myself would have been daunting, but when I asked ChatGPT it whipped up the code, and all I had to do was put it into the product and check that it worked. After trying all sorts of improvements this way, I presented again at the Dev Seminar in March 2025.

In the second attempt, I decided to solve the problem with people and computers working together. Writing an algorithm to tell stars apart from lights along the edges of buildings was still hard, so a person would review the result. The computer detects stars first, a person reviews them and marks the ones they think are stars, and the computer does plate solving based on that. As a result, the computer alone wasn't very accurate, but once a person cleaned things up, accuracy was high.

Without a human in the loop, it got only 4 of the 16 sample photos right. The very first step, finding stars, was already picking up building lights as stars.

On top of that, it took the candidate with the closest fingerprint as the answer, and that was often wrong.

It took about 4 seconds per photo before the candidates came out. Deciding correctly which answer was right was still lacking, so once again I couldn't get it ready for hosting. Solving a single star photo still meant downloading 28 MB. The last slide of the talk listed three pieces of homework:

1. Make the right call: judge on its own whether the answer it found is correct
2. Correct the drift in where surrounding stars are drawn
3. Publish it as a website anyone can try

I had worked with ChatGPT, but the debugging process of finding, analyzing and fixing the causes was still rough, and improving it further would have taken a lot of time. I only renewed my promise to improve it someday, and once again time went by.

Then, before long, another chance came along.

6. Third attempt: 2026, finished with Claude Code

In 2026, barely a year later, AI had improved dramatically, and building software without writing code yourself had become the norm. At Buzzvil, too, nobody writes code by hand anymore. By the second half of 2026 we were even giving AI access to the browser and letting it debug directly, and it handled most problems remarkably well without any human help. I thought that if I applied this approach to the plate solver, this time I might be able to finish it.

So this time I handed it to Claude Code without writing a single line of code myself. Instead, we agreed on rules to follow together:

• Don't judge by gut feeling; measure on many photos.
• Measure under the same conditions as the app (8 seconds per photo).
• Record every attempt and finding.

A year earlier I had said, "Let's not become someone who just does what the AI says," and this time I became the one giving the AI work to do. I decided what to do, checked whether the results were right, and corrected the course when it drifted. Claude Code wrote the code, and fixed things while opening the site in a browser and looking at the screen itself. Over four months starting in June 2026, about 500 commits piled up.

The three rules saved us more often than I expected. We believed the Milky Way photos were taken with fisheye lenses and built a separate fisheye solver, but measuring on many photos showed they were actually perspective-corrected. There were also improvements that helped when measured with no time limit but made things worse once we cut off at 8 seconds like the app does. Because we wrote down every attempt and finding, we ended up with a log of more than 30 chapters, and this post is based on it too.

We worked through the three pieces of homework from 2025 one by one.

The same 16 samples solved without a human in the loop. Top: the second attempt. Bottom: the third. The four blurred photos are ones I picked from the Naver café Byeolhaneul Jigi because I liked them and used as development samples, so they are hidden in this post.

So how did the third attempt solve these problems?

Judging on its own whether the answer is right

The most similar fingerprint isn't always the answer. But you can't simply say "if a photo star isn't on the map when you overlay the star map as the answer suggests, the answer is wrong," because even a right answer always leaves points in the photo that aren't on the map. So, as in the Astrometry.net paper, we compute "how much more plausible this answer is than chance."

1. Overlay the star map on the photo as the answer suggests.
2. Count how many stars not used to find the match land on stars in the photo.
3. Work out how many would land on photo stars by chance if the map were placed anywhere. With 60 photo stars and 168 map stars, that's about 2.4.
4. If 26 land when 2.4 are expected, the probability that it's chance is 10 to the power of −17.8.

Only when this probability is lower than one in a billion does it say "Found it"; otherwise it says "I don't know."

Imitating what people used to do

When people look at a star, they also look around the point. A real star is surrounded by an even night sky, while a window light is surrounded by walls and other lights. So we score each point on how even the brightness of the ring around it is, call that "star-likeness," and use the most star-like points first.

When people look at a photo, they cover the ground first and look only at the sky. So if it can't solve the whole photo, it tries again keeping only the top 75%, 55% and 45%.

We also brought back the triangles that had failed in the first attempt. To keep the star map small, quads are only built from combinations where all four stars are within 10 degrees of each other. So for widely spread constellations like the Big Dipper, there are few shapes to build. Triangles can be built almost anywhere as long as there are three stars, but as we saw earlier, there are far too many look-alike fakes.

The key to cutting the fakes was size. Triangles in a photo are in pixels, so we don't know how many degrees they span in the sky, which meant we could only compare shapes (similarity). But if you guess the lens's field of view, you can tell how many degrees a triangle in the photo spans in the sky. After fetching the eight candidates with the most similar shape, we throw away, without checking, any whose size isn't between half and double. Not having to check fakes that match in shape but not in size makes it that much faster. We don't know the field of view, so we try all nine guesses from 10 to 75 degrees. A wrong guess can throw away the right answer too, but only one of the nine needs to be right.

And we use both quads and triangles. They're good at different things, and at first each solved photos the other couldn't.

	Triangle (three stars)	Quad (four stars)
Real shapes buildable from the photo	304	64 (only within 10°)
Fakes more similar than the answer (median)	3,052	1
Photos solved on its own (of 14)	14	10
Time taken (longest among solved photos)	3.3 s	0.3 s

Quads have almost no fakes, so they reach confidence quickly, while triangles can be built anywhere, so they hang on to the end. So we try quads first for a fast solve, and fall back to triangles if that fails.

I ended the 2023 talk with "Teaching a computer is hard," and what I learned this time was that having it imitate, one at a time, the simple things people actually do works better than a clever method.

Correcting where surrounding stars are drawn

The second attempt computed the transformation between the photo and the sky from only the three or four matched stars. So it was right near the matched stars, but the farther you went, the more the drawn constellations drifted off the real stars. In screen ④ of the second attempt shown earlier, you can see the lines near Polaris straying from the real stars.

The third attempt, after finding the answer, refits the transformation with every star that lined up with a star in the photo. If eight or more line up, it refines once more with a perspective transformation that even fits the trapezoid-shaped distortion from shooting at an angle.

Getting ready to ship

There were four reasons the second attempt couldn't ship:

• Accuracy: It didn't solve arbitrary photos as well as hoped, and it couldn't judge whether the answer it found was right.
• Speed: It took about 4 seconds per photo before the candidate list came out. Almost all of that was unpacking a 65 MB pattern DB and building the search tree.
• Size: You had to download 28 MB before solving a single photo, so I was worried about server costs.
• Ease of use: It only solved photos after a person corrected the star marks. I know which points to remove, but someone visiting for the first time doesn't.

Accuracy and ease of use were solved above. Wrong answers are filtered out by "say you don't know when you don't know," and star selection now works without a human by imitating what people used to do. For speed, it stops as soon as it's confident. Preparation happens in advance while you're choosing a photo (0.75 seconds), and solving takes 0.13 seconds at the median, with the slowest photo taking 3.9 seconds. Photos it can't solve are cut off at 8 seconds.

What remained was size. Measuring what was downloaded, the pattern DB was 22 MB, the image processing library OpenCV.js was 3.9 MB, and, surprisingly, the font was 2 MB.

• Dropping OpenCV.js: All OpenCV did was grayscale, blur, subtract and pick bright points. Writing that myself took a little over 120 lines, and it was almost as fast (0.16 vs. 0.15 seconds on the same photo). Yet we were downloading 3.9 MB for it.
• Only the characters in use: A single font file contained all 11,000-plus Korean characters. We split the characters into 92 pieces and only download the pieces containing characters on screen, which brought it to 0.31 MB.
• Build the pattern DB instead of downloading it: The pattern DB can always be rebuilt identically from star positions and a rule (dividing the sky into 192 cells of equal area). So instead of downloading it, the browser builds it in 0.75 seconds from the 1,624 stars visible to the naked eye while you're choosing a photo.

In the second attempt, solving one star photo meant downloading about 28 MB. Dropping the image processing library (OpenCV.js), downloading only the font characters in use, and having the browser build the pattern DB instead of downloading it brought this to 0.68 MB, one-fortieth of the original.

Validating it fairly

The 16 samples are photos I had run countless times during development. To check that it wasn't just tuned to pass the samples, we validated it separately on photos never seen during development.

• Photos: 200 night sky photos picked from Wikimedia Commons. Besides auroras, the Milky Way, constellations, and photos with the ground and buildings, we deliberately included 91 photos in hard conditions such as moonlight, twilight, star trails, telephoto and phone cameras. We excluded zoomed-in telescope photos of nebulae and photos with constellation lines drawn on them.
• Conditions: Just like the app, each photo had an 8-second limit.
• Judging: Even when the solver said "Found it," we didn't take it at face value. We first tuned the judging criteria on samples with known answers, then drew the constellation lines on every photo it was confident about and checked each one by eye.

It was confident on 102 of the 200 photos, and all 102 were correct. Not a single photo got a confident wrong answer. The median solving time for confident photos was 0.09 seconds, and 76 of the 102 finished within a second. The photos it couldn't solve were concentrated in conditions it can't handle yet, such as star trail photos where stars are lines rather than points (0 of 10) and fisheye photos that capture the whole sky in a circle.

200 night sky photos from the internet, never seen during development. It was confident on 102 within 8 seconds, and 0 photos got a confident wrong answer.

With a success rate around 50%, there's still room to improve, but since it says it doesn't know when it doesn't and never gives a wrong answer, we confirmed it was a tool good enough to release for now.

Results

The Saipan photo from 2019 that started it all can now be solved.

Alpha Centauri, which I found by hand in 2019, is drawn on it.

This is the moment AI finally made a seven-year dream come true.

What I learned building it together

AI is better at development than I am. Problems I had wrestled with for the past three years without finding the cause, the AI worked through one by one this time, measuring on many photos and keeping records.

I didn't write each line of code myself, but making something was still fun. Ideas turned into something I could see quickly, so I could build it in a short, focused stretch, and it was deeply absorbing and satisfying.

It was also nice that I could leave the work to it and do other things. While the AI was building, I could take care of work or personal things, sometimes play a game, and just check the results. So I could keep making steady progress alongside other things without setting aside big blocks of time.

That said, since I didn't build it directly, I seem to feel a little less attached to the code than before. And now that anyone can build something like this easily with AI, I don't feel I've done anything special.

On the other hand, communicating with people, like the presentation slides and this post, still had to be reviewed and revised piece by piece.

7. Try it yourself

You can try plate solving below, or at star.byb.kr.

More to explore

Along the way, the code for computing the sky piled up, so I also made things to learn and things to play.

Things to learn

• How it works: The solving process covered in this post (preparing the star map, finding stars, matching shapes, checking, hard photos), laid out step by step with demos you can play with.
• Sky reading basics: Nine concepts, including sky coordinates, why the day shifts by 4 minutes, star hopping (finding stars by jumping across them, using the fact that a fist is 10 degrees), and how sailors of old found their position by the stars.
• Learning constellations: Collect a constellation guide, learn constellation shapes with cards, and find your way between stars in the real sky.

Things to play

The Star Playground has 12 games and 7 things to explore.

• Sky Match: Turn the latitude, date and time dials to recreate the sky in a photo. It's plate solving done by a person.
• SkyGuessr: Look around the night sky and drop a pin on a world map where you think it was taken.
• Human Calculator vs Computer: Who finds the constellation in a photo first? You use your eyes, the computer uses the plate solver.
• Constellation Hide-and-Seek: Find and tap the given constellation in the night sky. Try it right below.

Find more in the Star Playground.

8. Homework left

The goals I set in 2022 were modest: a PoC, bright stars only, working even with the ground or clouds in the shot, and if not, at least one or two photos. It took four years to reach them all, and instead of one or two photos, 15 of the 16 samples I used during development were solved.

There's still work left: fisheye lenses and star trail photos, solving photos with only ten or so stars without the time and place, and measuring how long it takes on a phone.

If you have night sky photos on your phone, try uploading them to star.byb.kr. If one doesn't get solved, let me know. It becomes material for the next improvement.

Code: github.com/ybbarng/star.byb.kr

9. What Buzzvil made possible

Plate Solver is a personal project, but I couldn't have come this far without Buzzvil.

• A company that uses AI actively: Buzzvil is turning into an AI-native organization. Building with AI instead of writing code directly, as in the third attempt, has taken root in our day-to-day work too, and even colleagues who aren't developers built things with AI themselves at our internal AI hackathon. (그놈의 AI, 결국 같이 일하게 됐습니다 — "That AI, we ended up working with it after all," in Korean)
• Buzz Challenge, which supports growth and dreams: Every quarter, each of us sets a goal and takes on a challenge, and the company cheers it on. The first attempt of this project also started as a Dream Challenge.
• Buzz Holiday, for recharging: After three years with the company, you get two weeks of paid leave. I used it to go see the aurora in Yellowknife, Canada, and the aurora photo at the start of this post was taken on that trip. (버즈빌리언이 리프레시 하는 방법, 버즈홀리데이 — "How Buzzvillians recharge: Buzz Holiday," in Korean)

🚀 Want to find stars and build your career with us at Buzzvil, an AI-native company?
>> See our open positions now (click)

References

Papers

• Lang, D., Hogg, D. W., Mierle, K., Blanton, M., & Roweis, S. (2010). Astrometry.net: Blind astrometric calibration of arbitrary astronomical images. The Astronomical Journal, 139(5), 1782–1800. arXiv:0910.2233 — quad fingerprints, judging by the probability of chance
• Górski, K. M., et al. (2005). HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed on the Sphere. The Astrophysical Journal, 622(2), 759–771. arXiv:astro-ph/0409513 — dividing the sky into cells of equal area
• Bentley, J. L. (1975). Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9), 509–517. — KD-tree

Data

• Hoffleit, D., & Warren, W. H. Jr. (1991). The Bright Star Catalogue, 5th Revised Edition. VizieR V/50 — star catalog
• Olaf Frohn, d3-celestial (BSD license) — constellation lines and constellation boundaries (IAU)

Sun, Moon and planet calculations

• Meeus, J. (1998). Astronomical Algorithms (2nd ed.). Willmann-Bell. — checking sidereal time and other calculations
• Low-precision Sun and Moon formulas from The Astronomical Almanac
• Standish, E. M. Keplerian Elements for Approximate Positions of the Major Planets. JPL. — planet positions
• JPL Horizons — checking the results

Photos

• Night sky photos taken by colleagues: Saipan (Jarrod), the pavilion (Berry), aurora (Carl), Australia and Jeongseon (Nike)

• The aurora photo in the introduction and the featured image, and the stargazing app (Constellation Map) screen: the author

• Photographic plate: the part of a Harvard College Observatory photographic plate where Williamina Fleming discovered the Horsehead Nebula in 1888

• Horsehead Nebula: ESO (eso1839i), CC BY 4.0 — Wikimedia Commons File:Nebulosa testa di cavallo con SPECULOOS.jpg

• Some of the early development sample photos were ones I picked because I liked them from the Naver café Byeolhaneul Jigi. They are blurred in this post.

• 12 of the site's current samples are freely licensed photos (public domain, CC0, CC BY) from Wikimedia Commons. Authors and licenses are listed on the site's Sources and references page.