Details of the study and the new model will appear in the Proceedings of the National Academy of Sciences, Volume 99, Issue 8, pp. 5207-5211, April 2002.
Previous studies have shown that the distribution of links to web sites approximates a "power law" (Science, 286, p. 509; Nature, 401, p. 130), where a tiny fraction of sites receive a hugely disproportionate share of links, and the vast majority of sites are essentially ignored. Since the number of links to a web site closely relates to its popularity, traffic, and search-engine ranking, these studies imply a very bleak state of competition on the web, where the "winners" dominate (e.g., sites like Yahoo and Amazon), while new entrants simply cannot compete. In other words, "winners take all".
The NEC scientists took a closer look, examining the distributions of links within various web communities, or topically-related collections of pages. They discovered that the distribution of links can deviate greatly from the power law distribution found in previous studies of the entire web, although a very clear pattern still emerges which is common across different communities (in scientific terms, the distribution has "a lognormal body with a power law tail"). Moreover, the degree of deviation from the power law, and thus the extent to which "winners take all", varies widely from group to group. The scientists developed a model of how individual web communities "grow" by incorporating new pages and new links according to certain rules. The model predicts the same distributions that were observed on the real web. The model parameters quantify the degree to which rich (highly-linked) sites grow richer, and how new sites can compete. The model may be used, for example, by a new e-commerce site to estimate how much competition it will face within its niche.
"We were suprised to find such drastic differences for competition within individual communities, as compared to competition viewed across the entire web," said Dr. David Pennock of NEC Research Institute, the study's lead author. Pennock also noted that the dynamics of information dissemination online have the potential to alter competition and diversity in commerce and society, and that an increasing percentage of commerce and communication is occurring on the web over time.
The study's results may have implications for e-tailers trying to gain a foothold online. The process of finding stores on the web differs from the process of finding bricks-and-mortar establishments. For example, a consumer's decision to visit a physical store may be based on having driven past the store, receiving a recommendation from a local friend, or via local advertising. In contrast, the decision to visit an online store is strongly influenced by links to the store on the web, and by the presence of the store in search results (which is in turn influenced by links). The nature of link growth on the web may result in a substantial "first mover" advantage where it is more difficult for new entrants to gain links and traffic when competing with established and well-linked stores.
The researchers looked specifically at links within the following communities on the web: companies, newspapers, universities, and the personal homepages of scientists, as well as several e-commerce categories such as publications, consumer electronics, entertainment, sports, and photographers. They found that the degree of "winners take all" behavior varied greatly. The distribution was closest to a pure power law ("winners take all") for companies, newspapers, and publications. In contrast, the distribution for universities, scientists, and photographers was much less biased -- individual members of these communities fared much better.
Beyond the web, the new research may also be used to model and analyze other social networks such as the network of research paper citations or the network of US power grid connections.
More information and figures can be found at http://modelingtheweb.com/
About NEC Research Institute
NEC Research Institute, founded in 1988 and based in Princeton, conducts basic research in the areas of computer and physical sciences. Its major research elements include Web computing; robust computing; intelligence; vision and language; devices; materials; optics; nano physics; biophysics, theoretical computer sciences and physics. For more information about the Institute, please visit its Web site at www.neci.nec.com.
About NEC Corporation
NEC Corporation (NASDAQ: NIPNY; FTSE: 6701q.l) is a leading provider of Internet solutions, dedicated to meeting the specialized needs of its customers in the key computer, network and electron device fields through its three market-focused in-house companies: NEC Solutions, NEC Networks and NEC Electron Devices. NEC Corporation, with its in-house companies, employs more than 150,000 people worldwide and saw net sales of 4,991 billion Yen (approx. US$48 billion) in fiscal year 1999-2000. For further information, please visit the NEC home page at: www.nec-global.com.
Journal
Proceedings of the National Academy of Sciences