Ad Management 파트 서버 개발자의 지역 타게팅 개선기
안녕하세요. Demand Product 팀의 Ad Management 파트에서 서버 개발자로 일하고 있는 Kay입니다. 제가 팀에 합류한 지도 어느덧 2년이 되어가는데요, Ad Management 파트(이하 AdM)은 무슨 일을 하는지 간단하게 소개하고 재미있게 했던 …
Read ArticleMy first project at Buzzvil was to develop a system to reliably render images at scale with PhantomJS. The result of my efforts was Ghost Town: simple queued & clustered PhantomJS processing in a tiny Node.js module.
PhantomJS is not an easy library to work with. Crashing and freezing is common. High memory usage and slow startup time is expected, and scaling must be done manually. Ultimately, PhantomJS and phantomjs-node are neglected projects, so these issues can be expected to remain indefinitely. At Buzzvil, we needed the ability to reliably render a wide variety of images, with low latency and therefore high concurrency. No existing project effectively managed this, so I researched and designed a module that could both gracefully recover from crashes and scale automatically.
Enter Ghost Town. It does the heavy lifting of launching and managing PhantomJS and queuing and processing tasks, and then it gets out of your way to let you do the rest. Each item is stored in the master’s queue until a worker is ready, and then assigned for processing. If the processing times out or the assigned worker fails, Ghost Town will requeue it. Reliability guaranteed! To prevent memory leaks, Ghost Town creates separate PhantomJS processes for each worker, and periodically relaunches them based on their number of pages created.
Initializing Ghost Town is easy. (Several configuration options are available for tweaking the runtime and efficiency settings; see the documentation for details.) With no configuration:
var town = require("ghost-town")();
Next, implement the master and worker as when using the normal cluster module. Usually a simple if (town.isMaster)
check will suffice.
if (town.isMaster) {
// master code
// queue items here
} else {
// worker code
// process items here
}
The main master method is town.queue(data, next)
, which queues an item with data
, and calls next(err, data)
when the item has been processed by the worker. You’re free to implement the master queuer however you like, probably using some sort of messaging system. At Buzzvil, we use Thrift. Our master starts a server, and queues render requests as it receives them:
thrift.createServer(Renderer, {
render: function (html, width, height, next) {
town.queue({
html: html,
width: width,
height: height
}, function (err, data) {
next(err, !err && new Buffer(data, "base64"));
});
}
}).listen(1337);
The only worker hook is the town!queue(page, data, next)
event. Ghost Town automatically manages everything, so page
will always be a brand new PhantomJS page. All you need to do is configure the page, process the data
passed from the master, and call next(err, data)
to pass it back. At Buzzvil, we render HTML documents. Our worker configures the page (size, headers, content) and renders it to an image:
town.on("queue", function (page, data, next) {
// setup code
// sequentially configure the page here
// page.set("viewportSize", ...)
// page.set("customHeaders", ...)
// page.set("onLoadFinished", ...)
// page.set("content", ...)
page.renderBase64("jpeg", function (data) {
next(null, data);
});
});
Ghost Town has been successfully running at scale in production for over a month now, ultimately a significant improvement over our previous manual concurrency solution. Further improvements can be expected with the release of Node.js v0.12, and we hope the community benefits from our release of Ghost Town under the MIT license. Happy coding!
안녕하세요. Demand Product 팀의 Ad Management 파트에서 서버 개발자로 일하고 있는 Kay입니다. 제가 팀에 합류한 지도 어느덧 2년이 되어가는데요, Ad Management 파트(이하 AdM)은 무슨 일을 하는지 간단하게 소개하고 재미있게 했던 …
Read Article들어가며 안녕하세요, 버즈빌에서 MLOps 엔지니어로 일하고 있는 BK입니다. 버즈빌 광고 플랫폼 팀은 광고 성과 최적화를 위해서 다양한 머신러닝 모델을 활용하고 있습니다. 앞으로 두 편의 글을 통해 모델을 개발, 관리, …
Read Article