Bingo! Why Bearer-Settled Recursive Auction Markets will work (was re:[NEC] 2.3: Power Laws, Weblogs, and Inequality)

[R. A. Hettinga]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Remember Eric Hughes' "institutionalized" digital piracy market, and,
before him, the Agoric guys, and their Digital Silk Road stuff?

Remember what I said about how, in such a market, the people who made
the *new* stuff first would make the most money, about the first copy
being the most valuable?

Read this, and then go look at the charts on the web. It's important.

Now, think about a bearer-settled cash auction of the first copies of
new content into a napster-like network.

Remember that each cryptographically-authenticated (I dislike the
word "signed", I agree with Perry Metzger and Carl Ellison;
"signature" doesn't mean what it says) copy is fungible. It's the
same as any other. Thus,  the market, like all commodity markets, is
operating, in an economic terms of art, under perfect competition.


I'd always thought that the price curve in hops from the original
source and, in time from a given product's introduction to the net,
would look like power curves, or maybe a gamma distribution, but, in
Clay's article below, here's proof, or at least as much proof as
we're going to get until we actually do it.

I expect that the propagation rates of these markets are going to be
*very* fast, which follows from the speed with which information
flows through the net. Seen in three dimensions, with the charts laid
on top of each other we get something that looks like what happens
after a water drop hits a smooth pond.


This idea of recursive auction markets across a geodesic network is,
for the most part, a fundamental indictment of the very economic
efficacy of copyright on a geodesic network. In getting what people
want for what they're most willing to pay for it, for the lowest cost
of production and distribution, this kind of market will make more
money, faster, for the *producers* of content, and will adequately
compensate the *distributors* of content (people who continually
resell their copies to the highest bidders, of course :-)), while, at
the same time, not requiring monitoring the entire system by a single
entity.

Like Gilmore said about censorship, if there's cash and an auction
mechanism, the network will finally see copyright as damage and route
around it.

Cheers,
RAH

- --- begin forwarded text


User-Agent: Microsoft-Entourage/10.1.0.2006
To: <nec at shirky.com>
From: nec-admin at shirky.com
Subject: [NEC] 2.3: Power Laws, Weblogs, and Inequality
Sender: nec-admin at shirky.com
Date: Sat, 08 Feb 2003 12:54:00 -0500
Status: R

NEC @ Shirky.com, a mailing list about Networks, Economics, and
Culture

           Published periodically / # 2.3 / February 8, 2003
        Licensed under the Creative Commons Attribution License
               Subscribe at http://shirky.com/nec.html

In this issue:

 - Introduction
 - Essay: Power Laws, Weblogs, and Inequality
    (Also at http://www.shirky.com/writings/powerlaw_weblog.html)
 - Reader feedback on The Big Flip: I was wrong
 - The O'Reilly Emerging Technology Conference

* Introduction
=======================================================

This issue's essay is on the inevitability of power laws in social
systems, and in particular in the weblog world. I have lived through
a decade of seeing social systems on the internet start small and
egalitarian and grow large and unequal, and every time it has
happened, some of the users of the system have taken it upon
themselves to complain that the the Old Guard had gotten cliquish, or
that the newbies were not living up to what was expected of them, or
[insert psychological explanation here.]

These explanations, focussed as they were on individual behaviors,
never seemed to me to be adequate to describe what was obviously
structural change that happened to all kinds of systems. Recently,
work by Barabasi, Huberman, and Watts have all pointed to ways in
which power law distributions, where the rank of the Nth item is
1/Nth that of the first item, arise in social systems. The structural
inevitability of power laws explain these kinds of inequalities in
social systems far better than any explanation focussed on the
actions of individual members of the system.

One note: the essay includes three figures, accompanying this mail as
attachments, but it may be easier to read on the web, at
http://www.shirky.com/writings/powerlaw_weblog.html.

Also, because inbound spam to the list is now so extreme, I have a
choice between working from home enlarging my genitals with HGH while
waiting for my check from Nigeria to clear, or auto-flushing inbound
mail. Much to the disappointment of Madame Abacha, I am taking the
latter course of action, so if you have a response to something you
read here, please respond to me directly, at clay at shirky dot com.

- -clay

* Essay
==============================================================

Weblogs, Power Laws, and Inequality
(http://www.shirky.com/writings/powerlaw_weblog.html)

A persistent theme among people writing about the social aspects of
weblogging is to note [http://www.fawny.org/decon-blog.html] (and
usually lament [http://onepotmeal.com/blog/archives/001178.html]) the
rise of an A-list, a small set of webloggers who account for a
majority of the traffic in the weblog world. This complaint follows a
common pattern we've seen with MUDs, BBSes, and online communities
like Echo and the WELL. A new social system starts, and seems
delightfully free of the elitism and cliquishness of the existing
systems. Then, as the new system grows, problems of scale set in. Not
everyone can participate in every conversation. Not everyone gets to
be heard. Some core group seems more connected than the rest of us,
and so on.

Prior to recent theoretical work on social networks, the usual
explanations invoked individual behaviors: some members of the
community had sold out, the spirit of the early days was being
diluted by the newcomers, et cetera. We now know that these
explanations are wrong, or at least beside the point. What matters is
this: Diversity plus freedom of choice creates inequality, and the
greater the diversity, the more extreme the inequality.

In systems where many people are free to choose between many options,
a small subset of the whole will get a disproportionate amount of
traffic (or attention, or income), even if no members of the system
actively work towards such an outcome. This has nothing to do with
moral weakness, selling out, or any other psychological explanation.
The very act of choosing, spread widely enough and freely enough,
creates a power law distribution.

- - A Predictable Imbalance

Power law distributions, the shape that has spawned a number of
catch-phrases like the 80/20 Rule and the Winner-Take-All Society,
are finally being understood clearly enough to be useful. For much of
the last century, investigators have been finding power law
distributions in human systems. The economist Vilfredo Pareto
observed that wealth follows a "predictable imbalance", with 20% of
the population holding 80% of the wealth
[http://bently.com/articles/999pareto.asp]. The linguist George Zipf
observed that word frequency falls in a power law pattern
[http://linkage.rockefeller.edu/wli/zipf/], with a small number of
high frequency words (I, of, the), a moderate number of common words
(book, cat cup), and a huge number of low frequency words
(peripatetic, hypognathous). Jacob Nielsen observed power law
distributions in web site page views, and so on.
[http://www.useit.com/alertbox/zipf.html]

We are all so used to bell curve distributions that power law
distributions can seem odd. The shape of Figure #1, several hundred
blogs ranked by number of inbound links, is roughly a power law
distribution. Of the 433 listed blogs, the top two sites accounted
for fully 5% of the inbound links between them. (They were
InstaPundit and Andrew Sullivan, unsurprisingly.) The top dozen (less
than 3% of the total) accounted for 20% of the inbound links, and the
top 50 blogs (not quite 12%) accounted for 50% of such links.

http://www.shirky.com/writings/powerlaw/figure1.gif

Figure #1: 433 weblogs arranged in rank order by number of inbound
links. The data is drawn from N.Z Bear's 2002 work on the blogosphere
ecosystem [http://www.truthlaidbear.com/ecosystem.shtml], a project
that is now sadly defunct.

The inbound link data is just an example: power law distributions are
ubiquitous. Yahoo Groups mailing lists ranked by subscribers is a
power law distribution. (Figure #2) LiveJournal users ranked by
friends is a power law. (Figure #3) The traffic to this article will
be a power law, with a tiny percentage of the sites sending most of
the traffic. If you run a website with more than a couple dozen
pages, pick any time period where the traffic amounted to at least
1000 page views, and you will find that both the page views
themselves and the traffic from the referring sites will follow power
laws.

http://www.shirky.com/writings/powerlaw/figure2.gif

Figure #2: All mailing lists in the Yahoo Groups Television category,
ranked by number of subscribers (Data from September 2002.)

http://www.shirky.com/writings/powerlaw/figure3.gif

Figure #3: LiveJournal users ranked by number of friends listed.
(Data from March 2002)

- - Rank Hath Its Privileges

The basic shape is simple - in any system sorted by rank, the value
for the Nth position will be 1/N. For whatever is being ranked --
income, links, traffic -- the value of second place will be half that
of first place, and tenth place will be one-tenth of first place.
(There are other, more complex formulae that make the slope more or
less extreme, but they all relate to this curve.) We've seen this
shape in many systems. What've we've been lacking, until recently, is
a theory to go with these observed patterns.

Now, thanks to a series of breakthroughs in network theory by
researchers like Albert-Lazlo Barabasi [http://www.nd.edu/~alb/],
Duncan Watts [http://smallworld.sociology.columbia.edu/watts.html],
and Bernardo Huberman [http://www.hpl.hp.com/shl/people/huberman/],
among others, breakthroughs being described in books like Linked
[http://isbn.nu/0738206679], Six Degrees [http://isbn.nu/0393041425],
and The Laws of the Web [http://isbn.nu/0262083035], we know that
power law distributions tend to arise in social systems where many
people express their preferences among many options. We also know
that as the number of options rise, the curve becomes more extreme.
This is a counter-intuitive finding -- most of us would expect a
rising number of choices to flatten the curve, but in fact,
increasing the size of the system increases the gap between the #1
spot and the median spot.

A second counter-intuitive aspect of power laws is that most elements
in a power law system are below average, because the curve is so
heavily weighted towards the top performers. In Figure #1, the
average number of inbound links (cumulative links divided by the
number of blogs) is 31. The first blog below 31 links is 142nd on the
list, meaning two-thirds of the listed blogs have a below average
number of inbound links. We are so used to the evenness of the bell
curve, where the median position has the average value, that the idea
of two-thirds of a population being below average sounds strange.
(The actual median, 217th of 433, has only 15 inbound links.)

- - Freedom of Choice Makes Stars Inevitable

To see how freedom of choice could create such unequal distributions,
consider a hypothetical population of a thousand people, each picking
their 10 favorite blogs. One way to model such a system is simply to
assume that each person has an equal chance of liking each blog. This
distribution would be basically flat -- most blogs will have the same
number of people listing it as a favorite. A few blogs will be more
popular than average and a few less, of course, but that will be
statistical noise. The bulk of the blogs will be of average
popularity, and the highs and lows will not be too far different from
this average. In this model, neither the quality of the writing nor
other people's choices have any effect. In this model, there are no
shared tastes, no preferred genres, no effects from marketing or
recommendations from friends.

But people's choices do affect one another. If we assume that any
blog chosen by one user is more likely, by even a fractional amount,
to be chosen by another user, the system changes dramatically. Alice,
the first user, chooses her blogs unaffected by anyone else, but Bob
has a slightly higher chance of liking Alice's blogs than the others.
When Bob is done, any blog that both he and Alice like has a higher
chance of being picked by Carmen, and so on, with a small number of
blogs becoming increasingly likely to be chosen in the future because
they were chosen in the past.

Think of this positive feedback as a preference premium. The system
assumes that later users come into an environment shaped by earlier
users; the thousand-and-first user will not be selecting blogs at
random, but will rather be affected, even if unconsciously, by the
preference premiums built up in the system previously.

Note that this model is absolutely mute as to why one blog might be
preferred over another. Perhaps some writing is simply better than
average (a preference for quality), perhaps people want the recom-
mendations of others (a preference for marketing), perhaps there is
value in reading the same blogs as your friends (a preference for
"solidarity goods", things best enjoyed by a group). It could be all
three, or some other effect entirely, and it could be different for
different readers and different writers. What matters is that any
tendency towards agreement in diverse and free systems, however small
and for whatever reason, can create power law distributions.

Because it arises naturally, changing this distribution would mean
forcing hundreds of thousands of bloggers to link to certain blogs
and to de-link others, which would require both global oversight and
the application of force. Reversing the star system would mean
destroying the village in order to save it.

- - Inequality and Fairness

Given the ubiquity of power law distributions, asking whether there
inequality in the weblog world (or indeed almost any social system)
is the wrong question, since the answer will always be yes. The
question to ask is "Is the inequality fair?" Four things suggest that
the current inequality is mostly fair.

The first, of course, is the freedom in the weblog world in general.
It costs nothing to launch a weblog, and there is no vetting process,
so the threshold for having a weblog is only infinitesimally larger
than the threshold for getting online in the first place.

The second is that blogging is a daily activity. As beloved as Josh
Marshall (www.TalkingPointsMemo.com) or Mark Pilgrim
(www.DiveIntoMark.com) are, they would disappear if they stopped
writing, or even cut back significantly. Blogs are not a good place
to rest on your laurels.

Third, the stars exist not because of some cliquish preference for
one another, but because of the preference of hundreds of others
pointing to them. Their popularity is a result of the kind of
distributed approval it would be hard to fake.

Finally, there is no real A-list, because there is no discontinuity.
Though explanations of power laws (including the ones here) often
focus on numbers like "12% of blogs account for 50% of the links",
these are arbitrary markers. The largest step function in a power law
is between the #1 and #2 positions, by definition. There is no A-list
that is qualitatively different from their nearest neighbors, so any
line separating more and less trafficked blogs is arbitrary.

- - The Median Cannot Hold

However, though the inequality is mostly fair now, the system is
still young. Once a power law distribution exists, it can take on a
certain amount of homeostasis, the tendency of a system to retain its
form even against external pressures. Is the weblog world such a
system? Are there people who are as talented or deserving as the
current stars, but who are not getting anything like the traffic?
Doubtless. Will this problem get worse in the future? Yes.

Though there are more new bloggers and more new readers every day,
most of the new readers are adding to the traffic of the top few
blogs, while most new blogs are getting below average traffic, a gap
that will grow as the weblog world. It's not impossible to launch a
good new blog and become widely read, but it's harder than it was
last year, and it will be harder still next year. At some point
(probably one we've already passed), weblog technology will be seen
as a platform for so many forms of publishing, filtering,
aggregation, and syndication that blogging will stop referring to any
particularly coherent activity. The term 'blog' will fall into the
middle distance, as 'home page' and 'portal' have, words that used to
mean some concrete thing, but which were stretched by use past the
point of meaning. This will happen when head and tail of the power
law distribution become so different that we can't think of J. Random
Blogger and Glenn Reynolds of InstaPundit.com as doing the same
thing.

At the head will be webloggers who join the mainstream media (a
phrase which seems to mean "media we've gotten used to.") The
transformation here is simple - as a blogger's audience grows large,
more people read her work than she can possibly read, she can't link
to everyone who wants her attention, and she can't answer all her
incoming mail or follow up to the comments on her site. The result of
these pressures is that she becomes a broadcast outlet, distributing
material without participating in conversations about it.

Meanwhile, the long tail of weblogs with few readers will become
conversational. In a world where most bloggers get below average
traffic, audience size can't be the only metric for success.
LiveJournal had this figured out years ago, by assuming that people
would be writing for their friends, rather than some impersonal
audience. Publishing an essay and having 3 random people read it is a
recipe for disappointment, but publishing an account of your Saturday
night and having your 3 closest friends read it feels like a
conversation, especially if they follow up with their own accounts.
LiveJournal has an edge on most other blogging platforms because it
can keep far better track of friend and group relationships, but the
rise of general blog tools like Trackback may enable this
conversational mode for most blogs.

In between blogs-as-mainstream-media and blogs-as-dinner-conversation
will be Blogging Classic, blogs published by one or a few people, for
a moderately-sized audience, with whom the authors have a relatively
engaged relationship. Because of the continuing growth of the weblog
world, more blogs in the future will follow this pattern than today.
However, these blogs will be in the minority for both traffic
(dwarfed by the mainstream media blogs) and overall number of blogs
(outnum- bered by the conversational blogs.)

Inequality occurs in large and unconstrained social systems for the
same reasons stop-and-go traffic occurs on busy roads, not because it
is anyone's goal, but because it is a reliable property that emerges
from the normal functioning of the system. The relatively egalitarian
distribution of readers in the early years had nothing to do with the
nature of weblogs or webloggers. There just weren't enough blogs to
have really unequal distributions. Now there are.

- -=-

<snippage...>

* End
====================================================================

This work is licensed under the Creative Commons Attribution License.
The licensor permits others to copy, distribute, display, and perform
the work. In return, licensees must give the original author credit.

To view a copy of this license, visit
http://creativecommons.org/licenses/by/1.0

or send a letter to
Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305,
USA.

2003, Clay Shirky






- --- end forwarded text


-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0 - not licensed for commercial use: www.pgp.com

iQA/AwUBPkfBBcPxH8jf3ohaEQIdIgCg3RMXI9B1ru5YcEpXU8iPEUwghuIAnixX
do40shJI2PSt5bD/KS4zINGq
=1gs6
-----END PGP SIGNATURE-----

-- 
-----------------
R. A. Hettinga <mailto: rah at ibuc.com>
The Internet Bearer Underwriting Corporation <http://www.ibuc.com/>
44 Farquhar Street, Boston, MA 02131 USA
"... however it may deserve respect for its usefulness and antiquity,
[predicting the end of the world] has not been found agreeable to
experience." -- Edward Gibbon, 'Decline and Fall of the Roman Empire'