Face Recognition: Clever or Just Plain Creepy?

Mon Mar 2 03:18:27 PST 2009

http://www.technologyreview.com/printer_friendly_article.aspx?id=22234&channel=computing&section=

Friday, February 27, 2009

New photo programs from Apple and Google include revolutionary face-spotting
technology.

By Simson Garfinkel and Beth Rosenberg

We have more than 25,000 digital photographs stored on our computer hard
drives--most of them of people. Until now, our sole means of tracking down a
familiar face was to search manually: by date, EXIF data, "tags," or the
brute force of our own memory. Now computers can do the searching, thanks to
the nifty face-recognition feature that Apple and Google have put into the
latest versions of their photo-management systems.

Face recognition was one of those brilliant but technically iffy and
ethically tricky counterterrorism technologies deployed as a result of the
September 11 attacks. The idea was to automatically screen out terrorists as
they walked through security checkpoints--only it didn't work out that way:
at a test in Tampa, for example, airport employees were correctly identified
just 53 percent of the time. Civil-liberties groups also raised concerns
about false positives--people being mistakenly identified as terrorists, and
possibly arrested, just because of their looks. And so, without a
demonstratable benefit, face recognition largely dropped off the public's
radar.

That's the public's radar, mind you. Many countries, including the United
States, quietly revised their requirements for passport photos to make them
friendlier to face-recognition software. The National Institute of Standards
and Technology, which had been testing the technology since 1994, conducted
large-scale face-recognition tests in 2002 and 2006. Oregon and some other
states began using face recognition to detect when one person tries to obtain
a license under different names. And all the time, the technology kept
getting better. Much better.

In order to have a functioning face-recognition system, a computer must first
be able to detect the face--that is, given a photograph, it must be able to
find the faces in it. Technically, this is easier and more reliable than
identifying a particular person. This technology was pretty much perfected
just after September 11, 2001. The result: face-detection systems began
appearing in digital cameras and camcorders a few years ago. These algorithms
generally work by searching for objects that look like eyes, a nose, and
perhaps something that's kind of round. They identify boxes where faces are
likely to be, and then tell the autofocus system what part of the photo needs
to be in focus. After all, everybody hates it when Grandma's eyes are blurry,
right?

So face recognition starts with face detection. The face is then rotated so
that the eyes are level and scaled to a uniform size. Next, one of three
different technical approaches kicks in. Each of these approaches is, of
course, covered by its own set of patents and bundled into various vendor
offerings. One approach transforms the face into a mathematical template that
can be stored and searched; a second uses the entire face as a template and
performs image matching. And a third approach attempts to create a 3-D model
based on the face, and then performs some kind of geometric matching. Based
on our experience with the software, we believe that Apple's system is using
a landmarks approach, while the Google system is doing some kind of image
matching. But we could be wrong. Neither company has publicized which
algorithms it is using.

We tested the face recognition in Apple's iPhoto '09 by applying it to two
different databases of 17,000 and 10,000 photos, stored on our own hard
drives. Google's Picasa only works with previously uploaded Web albums; we
tested it on roughly 500 photos there. The verdict: both of these systems
mostly work, are extremely cool, and are also kind of creepy.

iPhoto '09 is certainly the friendlier of the two. The first time you run
iPhoto, it searches out all the faces in your photo library; this took about
four hours on a dual-core iMac. Next, you click on a photo of someone you
know; click "Name" and fill in the text box beneath your subject's face.
iPhoto will run through your photo library looking for other photos of the
same person. (The recognition seems to be based on features inside a
recognition box that is bounded by the left and right temples, eyebrows, and
chin.)

Overall, iPhoto does a surprisingly good job finding a bunch of photos of the
person you've selected and "named." But in the process, it finds photos of
other people as well. So your next task is to tell iPhoto which photos it got
right and which are wrong. iPhoto uses this information to update its
mathematical models. It then looks back through your photo library for other
photos of the same person. If it can't find any, you can manually point one
out to give iPhoto another starting point; it will then seek out more. You
can also click on a photo and ask iPhoto to try to figure out who is in the
picture; if you confirm iPhoto's guess, the model gets better still.

We were astonished at how good iPhoto was at finding photos of our kids.
Amazingly, iPhoto could even distinguish between our identical twins. (The
trick is that one of them has a face that's a bit thinner and taller than the
other's.) We were disappointed, however, that it found many more photos of
one twin than the other, although we photograph both in equal numbers--and
often in the same shot. A study of their photographs revealed something that
we hadn't noticed, but iPhoto had: one twin always looks directly at the
camera, but the other tends to tilt his head away, and iPhoto's face
recognition doesn't work if the program only sees one eye. We also have lots
of photos of kids in face paint. iPhoto found practically none of those,
except for when the paint was confined to the middle of the child's
forehead--which is outside its recognition box.

It's tempting to read a lot into iPhoto's recognition system. Searching for
photos of Beth produced lots of photos of Simson's ex-girlfriends. It's
tempting to say that iPhoto knows what Simson likes, but this could also be a
bias in our test corpus: pick random photos out of Simson's library, and
you're sure to find a bunch of his ex-girlfriends.

iPhoto was also surprisingly good at finding photos of our cats, especially
the ones with white or orange fur. Unfortunately, it failed to find the
tabbies--presumably, facial features are harder to distinguish when the eyes
are the same color as the cheeks. And iPhoto does a startling job at finding
and recognizing faces in shadows and other low-contrast situations. That's
because iPhoto cranks up the contrast between face and background, presumably
to make it easier to get out the features.

Since we installed iPhoto '09, our family has spent hours sitting around the
computer, searching for photos of the kids, and teaching the computer what
each of us looks like. We found a lot of old photos we had forgotten. We
laugh at the mismatches. We try to understand the algorithms. This is one of
the most entertaining programs that Apple has ever created.

Google's Picasa technology is far creepier. Instead of starting with a photo
of someone you know and searching for all the similar matches, Google takes
every photo that you've uploaded to Picasa, searches them all for faces, then
"clusters"these faces into groups of, supposedly, the same people. You then
go through each group and tell Google who a person is--including his or her
full name, nickname, and e-mail address.

In fact, Google's clustering isn't all that great. It frequently puts
different people in the same cluster, and it will make lots of different
clusters for the same person. And unlike iPhoto, which could easily match
photos of our 12-year-old daughter with her photos as a toddler, Google
thought that the children were different people. But Google's user interface
is pretty easy to employ, the matching task is strangely compelling, and
before you know it, you'll have every one of your photos tagged with all the
real names and e-mail addresses of each person that the photo features.

This real-name tagging is what makes Google's face recognition so creeosters.
This is certainly keeping with Google's corporate mission "to organize the
world's information and make it universally accessible and useful." But is
that what we really want from a photo-sharing website?

Our iPhoto experiences have been a delight: we've been excited and pleased to
find so many pictures of our kids, family, and friends--and even ourselves.
On the other hand, when we used the advanced tagging feature in Google's
Picasa, we felt as though we were intelligence analysts working in the
windowless lab of some totalitarian government.

We believe that consumer-driven face-recognition technology will
fundamentally change public-policy debates about biometrics and mass video
surveillance. After September 11, nobody really understood how this
technology worked, what it got right, and what it got wrong. But before the
end of this year, millions of Americans will have first-hand experience with
some of the very best face-recognition systems ever deployed. Once the
family-photo novelty wears off, we'll be watching to see if iPhoto and Picasa
users ask their government to regulate this technology--or accelerate its
deployment.