Migrating from PhantomJS to Puppeteer

Buzzvil operates several PhantomJS rendering servers to generate advertising and content images for mobile lockscreens. PhantomJS is often used for webpage screenshots, but its ability to render and capture pages headlessly also makes it useful for dynamic image generation. Our servers generate lockscreen images in real time from more than 200 content providers, so they must reliably produce hundreds of images per minute.

To scale, we distributed work across multiple rendering servers. Within each server, we also built a PhantomJS process pool using our Ghost Town library to improve resource utilization; see Scaling PhantomJS With Ghost Town.

Over time, our image templates diversified. Supporting emoji and special characters required installing more fonts, notably Noto Sans CJK. We then encountered inconsistent font rendering in PhantomJS.

We never established the exact cause. It may have been a PhantomJS issue, or inconsistent font caches across servers as fonts were added over time.

Like our other workloads, the rendering servers were initially intended to be built and updated consistently with Packer. But the service rarely received new features or deployments. Long-lived servers accumulated manual maintenance, until keeping systems and fonts current through Packer became difficult. They had become snowflake servers: like individual snowflakes, each deployed server gradually differed from the others. Everything appeared fine in normal operation, but scaling out or deploying a new template exposed problems.

A more serious issue was that PhantomJS was no longer maintained. Headless Chrome arrived with Chrome 59 in 2017, followed by the Puppeteer Node API, making a widely used modern rendering engine easy to run headlessly. PhantomJS’s maintainer effectively announced the end of development, and the original developer archived the project in 2018. Without updates, templates could not use newer CSS features and bugs would remain unfixed, making application maintenance increasingly difficult.

The problems were therefore:

1. Infrequent deployments had allowed servers—especially their fonts—to drift into snowflakes.
2. PhantomJS’s discontinued development prevented bug fixes and modern CSS support, complicating maintenance and new template development.

The solutions were clear. For the first issue, we could package the application and its system dependencies, including fonts, in a container and continually rebuild it through CI/CD. Rebuilding AMIs with Packer on every deployment could also have prevented drift. But AMI creation was not integrated into CI/CD: only the application was deployed continuously. With infrequent releases, keeping servers running and maintaining them manually had been convenient, until rebuilding a fresh AMI became difficult. We therefore chose immutable Docker images deployed to our existing Kubernetes cluster instead.

For the second issue, we replaced PhantomJS with Puppeteer. This was easier than expected. Whether intentionally or not, Puppeteer’s API resembles PhantomJS’s. It wasn’t a drop-in replacement, but modest changes around the API calls were enough.

Replacing the engine did not, however, guarantee that existing templates rendered as intended. We had to render our many templates with both engines and compare them individually. There were no retained definitions of each template’s arguments or intended output, so we first created sample cases. Comparing outputs remained manual, but having something concrete to inspect was a major improvement. We plan to add automated test cases to make future development easier.

The results were encouraging. Many outputs differed from the old ones, but the modern Chrome rendering engine generally interpreted CSS more accurately and produced what the template’s author originally intended.

FROM node:10-slim

RUN apt-get update && \
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget unzip && \
wget https://github.com/Yelp/dumb-init/releases/download/v1.2.1/dumb-init_1.2.1_amd64.deb && \
dpkg -i dumb-init_*.deb && rm -f dumb-init_*.deb && \
apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*

RUN yarn global add puppeteer@1.10.0 && yarn cache clean

ENV NODE_PATH="/usr/local/share/.config/yarn/global/node_modules:${NODE_PATH}"

RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser

# Set language to UTF8
ENV LANG="C.UTF-8"

RUN wget -P ~/fonttmp \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoSans-unhinted.zip \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoSansCJKjp-hinted.zip \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoSansCJKkr-hinted.zip \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoSansCJKtc-hinted.zip \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoSansCJKsc-hinted.zip \
    https://noto-website-2.storage.googleapis.com/pkgs/NotoColorEmoji-unhinted.zip \
 && cd ~/fonttmp \
 && unzip -o '*.zip' \
 && mv *.*tf /usr/share/fonts \
 && cd ~/ \
 && rm -rf ~/fonttmp

WORKDIR /app
# Add user so we don't need --no-sandbox.
RUN mkdir /screenshots && \
    mkdir -p /home/pptruser/Downloads && \
    mkdir -p /app/node_modules && \
    chown -R pptruser:pptruser /home/pptruser && \
    chown -R pptruser:pptruser /usr/local/share/.config/yarn/global/node_modules && \
    chown -R pptruser:pptruser /screenshots && \
    chown -R pptruser:pptruser /usr/share/fonts && \
    chown -R pptruser:pptruser /app

# Run everything after as non-privileged user.
USER pptruser

RUN fc-cache -f -v

COPY --chown=pptruser:pptruser package*.json /app/
RUN npm install && \
    npm cache clean --force
COPY --chown=pptruser:pptruser . /app/

ENTRYPOINT ["dumb-init", "--"]
CMD ["npm", "start"]

Some permissions issues with Puppeteer led to the Dockerfile above. For up-to-date advice on Puppeteer Docker images, see the troubleshooting documentation.

Kubernetes makes horizontal scaling and load balancing across containers straightforward. To improve each container’s throughput, we also created a Chrome process pool, much as Ghost Town had done for PhantomJS. We used generic-pool:

const puppeteer = require("puppeteer");
const genericPool = require("generic-pool");

const puppeteerArgs = [
  "--no-sandbox",
  "--disable-setuid-sandbox",
  "--disable-dev-shm-usage",
];

const createPuppeteerPool = ({
  max = 5,
  min = 2,
  maxUses = 50,
  initialUseCountRand = 5,
  testOnBorrow = true,
  validator = () => Promise.resolve(true),
  idleTimeoutMillis = 30000,
  ...otherConfig
} = {}) => {
  const factory = {
    create: async () => {
      const browser = await puppeteer.launch({
        headless: true,
        args: puppeteerArgs,
      });
      browser.useCount = parseInt(Math.random() * initialUseCountRand);
      return browser;
    },
    destroy: (browser) => {
      browser.close();
    },
    validate: (browser) => {
      return validator(browser).then((valid) =>
        Promise.resolve(valid && (maxUses <= 0 || browser.useCount < maxUses))
      );
    },
  };

  const pool = genericPool.createPool(factory, {
    max,
    min,
    testOnBorrow,
    idleTimeoutMillis,
    ...otherConfig,
  });
  const genericAcquire = pool.acquire.bind(pool);
  pool.acquire = () =>
    genericAcquire().then((browser) => {
      browser.useCount += 1;
      return browser;
    });
  pool.use = (fn) => {
    let resource;
    return pool
      .acquire()
      .then((r) => {
        resource = r;
        return resource;
      })
      .then(fn)
      .then(
        (result) => {
          pool.release(resource);
          return result;
        },
        (err) => {
          pool.release(resource);
          throw err;
        }
      );
  };
  return pool;
};

module.exports = createPuppeteerPool;

Caveats

A few details needed attention during the migration. Some existing templates loaded images through file:// URLs. PhantomJS allowed this, but Headless Chrome’s security policy blocked local file loading; see the related issue. We changed those templates to use http:// and served the local images as static files through Express.

Another issue involved loading HTML. Our PhantomJS implementation compiled Jade templates and passed the result to the page’s setContent method. Puppeteer’s page#setContent, however, did not wait for external images to load. A Puppeteer issue recommended representing the HTML as a data URI and using page#goto with a waitUntil option instead:

await page.goto(`data:text/html,${html}`, { waitUntil: "networkidle0" });

Be careful with networkidle0 and networkidle2. They do wait long enough for external images, but they also require a 500 ms interval without the relevant additional network connections. That adds a delay even after images have loaded. For static HTML rather than a single-page application, waiting for the load event is sufficient.

We also added a metrics endpoint for Prometheus and built a Grafana dashboard to support operations and future maintenance. It shows which templates are actually used and how long they take to render, helping identify unused templates and monitor the service.

Closing thoughts

Many teams have recently moved from PhantomJS to Puppeteer, so this may not be an entirely new story. But Buzzvil’s rendering service had already been heavily optimized around PhantomJS and needed high concurrent throughput. Replacing the engine involved real uncertainty.

We judged that allowing a core lockscreen image service to become harder to change because of an unmaintained dependency would create substantial technical debt. With that in mind, we built a container-based CI/CD pipeline that continually produces fresh images. The results were satisfying: after migration, previously delayed template development and new content provider integrations became easier.

Technical debt inevitably accumulates as business requirements change quickly. Engineers naturally want to clear every debt they see, but we cannot spend all our time doing so; resources are limited. If you’re unsure which debt to address now, start by measuring. Quantitative metrics can help persuade decision-makers and teammates. More broadly, monitoring a service’s key indicators improves visibility and makes it easier to identify the bottlenecks that will truly matter later.

References

• https://docs.browserless.io/blog/2018/06/04/puppeteer-best-practices.html
• https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md
• Icons made by Freepik from Flaticon are licensed under Creative Commons BY 3.0.