AOL Releases Search Logs from 500,000 Users

Mon Aug 7 01:05:50 PDT 2006

AOL Releases Search Logs from 500,000 Users
[1]Adam D'Angelo - 8/5/2006

AOL just released the logs of all searches done by 500,000 of their
users over the course of three months earlier this year. That means
that if you happened to be randomly chosen as one of these users,
everything you searched for from March to May (2006) is now public
information on the internet.

This was not a leak - it was intentional. In their desperation to gain
recognition from the research community, AOL decided they would
compromise their integrity to provide a data set that might become
often-cited in research papers: "Please reference the following
publication when using this collection..." is the message before the
download.

This is a blatant violation of users' privacy. The data is
"anonymized", which to AOL means that each screenname was replaced
with a unique number. "It is still a research question how much
information needs to be anonymized to protect users," [9]says Abdur
from AOL. Here are some examples of what you can find in the data:

User 491577 searches for "florida cna pca lakeland tampa", "emt school
training florida", "low calorie meals", "infant seat", and "fisher
price roller blades". Among user 39509's hundreds of searches are:
"ford 352", "oklahoma disciplined pastors", "oklahoma disciplined
doctors", "home loans", and some other personally identifying and
illegal stuff I'm going to leave out of here. Among user 545605's
searches are "shore hills park mays landing nj", "frank william
sindoni md", "ceramic ashtrays", "transfer money to china", and
"capital gains on sale of house". Compared to some of the data, these
examples are on the safe side. I'm leaving out the worst of it -
searches for names of specific people, addresses, telephone numbers,
illegal drugs, and more. There is no question that law enforcement,
employers, or friends could figure out who some of these people are.

I hope others can find more examples in the data, which is up for
[10]download over here. The data set is very large when uncompressed
which makes it pretty hard to work with, but someone should set up a
web interface so people can browse it (or even 10% of it) without
having to download the 400mb file. If you make a mirror or better
interface to the data, or find other examples, let me know and I'll
put a link up here.

This is the same data that the DOJ wanted from Google back in March.
[11]This ruling allowed Google to keep all query logs secret. Now any
government can just go download the data from AOL.

It's unclear if this is the type of data AOL released to the
government [12]back when Google refused to comply. If nothing else,
this should be a good example of why search history needs strong
privacy protection.

Thanks to Greg Linden for pointing this out [13]here.

Update 2: The md5 of the file AOL posted (and now removed) is
31cd27ce12c3a3f2df62a38050ce4c0a. I'm posting it so you can make sure
you have a valid copy, but so far none of the copies I've seen are
fake.

Update: Seems like AOL took it down. There are some mirrors of the
data in the comments of the digg story, linked below. I estimate about
1000 people have the file, so it's definitely going to be circulated
around. The [2]main AOL research page is still up, with some other
data collections. The [3]google cache of the download page is still
up, but you can't get the data. Here's discussion at other sites:
  * [4]siliconbeat
  * [5]techcrunch
  * [6]digg
  * [7]reddit
  * [8]zoli's blog

References

1. http://www.ugcs.caltech.edu/~dangelo/
2. http://research.aol.com/pmwiki/pmwiki.php?n=Main.Home
3. http://72.14.207.104/search?q=cache:2Qvd2z9VbuIJ:research.aol.com/
pmwiki/pmwiki.php%3Fn%3DResearch.500kUserQueriesSampledOver3Months
+&hl=en&gl=us&ct=clnk&cd=1
4. http://www.siliconbeat.com/entries/2006/08/06/
aol_research_exposes_data_weve_got_a_little_sick_feeling.html
5. http://www.techcrunch.com/2006/08/06/aol-proudly-releases-massive-
amounts-of-user-search-data/
6. http://digg.com/tech_news/AOL_Releases_Search_Logs_from_500_000_Users
7. http://reddit.com/info/cfvt/comments
8. http://www.zoliblog.com/blog/_archives/2006/8/6/2204969.html
9. http://research.aol.com/pmwiki/pmwiki.php?n=Research.
500kUserQueriesSampledOver3Months
10. http://research.aol.com/pmwiki/pmwiki.php?n=Research.
500kUserQueriesSampledOver3Months
11. http://googleblog.blogspot.com/2006/03/judge-tells-doj-no-on-
search-queries.html
12. http://www.boingboing.net/2006/01/20/aol_we_did_not_compl.html
13. http://glinden.blogspot.com/2006/08/chance-to-play-with-big-
data.html

--
Seth Finkelstein  Consulting Programmer  http://sethf.com
Infothought blog - http://sethf.com/infothought/blog/
Interview: http://sethf.com/essays/major/greplaw-interview.php

-------------------------------------
You are subscribed as eugen at leitl.org
To manage your subscription, go to
 http://v2.listbox.com/member/?listname=ip

Archives at: http://www.interesting-people.org/archives/interesting-people/

----- End forwarded message -----
--
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820            http://www.ativel.com
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]