HP Become Makeup Consultant

Snap, match, shop
Color-matching made simple
» HP Labs


» News release
» Consistent Image-Based Measurement and Classification of Skin Color (tech report)
» Nina Bhatti's bio

By Simon Firth, July 2007

Identifying the right shade of makeup may not be the first thing that comes to mind when you think about using your mobile phone.

It wasn't the first thing researchers at HP Labs imagined, either.
But thanks to their expertise in color science, imaging and mobile communications, they created a novel color-matching technology
that turns any camera-equipped phone into a mobile beauty advisor.
The technology, which helps consumers select makeup shades
best suited for their skin color, may also have applications for retailers, consumer goods companies, healthcare providers and others needing a high level of color or image coordination. These companies could potentially offer services that reach people where and when they need assistance.

(For information about partnering with HP to make the technology available to customers, please contact HP's intellectual property licensing team.)

How it works

Instead of sitting down with a consultant at a beauty counter – or puzzling over similar-looking shades in the store – consumers take their own photos and send these via MMS (multimedia message)
to a backend computer system. The system locates the consumer's face within the image and color-corrects the image for camera and lighting discrepancies.

The image is then compared to an existing database of previously captured and analyzed images of skin tones of real people. In a matter of seconds, the consumer receives a text message response, with a recommendation on the shade of foundation that best matches her complexion.

Why foundation? Because getting your foundation right is the hardest part of the whole makeup conundrum. And it's the most unforgiving when you get it wrong.


Color-matching challenge


Consumers face "a mind-boggling array of colors," says Nina Bhatti, who lead the research team. Most manufacturers offer between 14 and 24 shades, many of which are almost impossible to tell apart.

Even colors that look similar may dry differently. Then there are the confusing ways in which the colors are described; the same shade may even be available in three different formulas – 'warm,' 'cool,' and 'neutral.'

A good cosmetic consultant can reliably make these judgments, despite the stark lighting often available in stores. But this service is rarely available to shoppers, and then is usually limited to high-end products.

Mobile beauty advisor

How, then, to provide cosmetics consulting via mobile phone?

The somewhat daunting problem, Bhatti says, was to determine
how to make a precise color judgment from a digital picture of a face taken by any kind of camera under any kind of illumination — especially when both the type of camera and the lighting used can dramatically alter face color in a photograph.

Researchers solution: include in each photograph a small reference chart containing blocks of color with known color values. This chart is designed to be very sensitive to skin colors. (Shoppers hold the card next their faces when taking the photo.) When HP receives each image, it scans the picture to automatically find the chart.

"We know what colors the chart is supposed to be," explains Bhatti, "and so we can mathematically transform the image to correct it. When we do that, we also correct the skin tone."

Consulting the experts

The final stage is to match the sender's true skin color with a foundation choice.

That's not completely straightforward, says Bhatti, because makeup is as much an art as a science. "Cosmetologists have all kinds of secrets," she reports. "For example, if you have a lot of red in your skin, they tend to put a slightly more yellow foundation on you, rather than match that red."

To allow for this, researchers asked makeup experts to recommend foundation colors for a large group of women they brought into HP Labs as reference models.

The team used the data as the basis for a statistical processing program that compares each new user's face color with the colors of all the women in the study.

"We find the reference model whose color best matches yours and then with very high confidence we can say you wear the same foundation that she does," says Bhatti.

Imaging, color science and more

Behind mobile color-matching is a combination of both old and new HP technology. The application uses existing HP face-detection software and draws on HP's long experience with color science.

But researchers also pushed the frontiers of color science to make color calibration much more sensitive to skin colors. And they developed the statistical training and processing system that tells you what skin pixels constitute what foundation color from scratch.

"We combined the power of multiple laboratories," says Bhatti, noting especially contributions by researchers from HP's Digital Printing and Imaging Lab, and its Mobile and Media Systems Lab.

The research team: Top row (left to right) Harlyn Baker, Sabine Susstrunk, Nina Bhatti, Nic Lyons. Bottom row: Mike Harville, John Schettino, Scott Clearwater.


"It takes expertise in imaging, backend processing and mobility to make this work, all of which we have here," she adds. "This is one of those things that really only HP is equipped to do on a truly commercial scale."
http://www.hpl.hp.com/news/2007/jul-sep/color_match.html

Printable version