The effectiveness of federally funded research programs -- both basic and applied -- can be assessed meaningfully on an annual basis as required by law, says a new report from the Committee on Science, Engineering, and Public Policy, a joint committee of the National Academies of Sciences and Engineering and the Institute of Medicine. But different criteria should be used for different types of research to ensure that assessments fairly gauge progress.
"Measuring the performance of basic research is particularly challenging because major breakthroughs can be unpredictable and difficult to assess in the short term," said committee chair Phillip A. Griffiths, director, Institute for Advanced Study, Princeton, N.J. "Federal agencies should use a method we call 'expert review' to assess the quality of research they support, the relevance of that research to their mission, and the leadership of the research. This will ensure that funds spent on the research will generate the kinds of knowledge that in the past have brought great practical benefits."
During the course of its study, the committee heard two conflicting viewpoints on approaches to measuring basic research. One held that it is possible to measure research annually and provide quantitative measures of the useful outcomes of both basic and applied research. The other viewpoint was that given the long-range nature of basic research, no sensible way exists to respond to the annual measurement requirement. Therefore, some agencies may resort to using measures that seem to respond to federal law -- such as a list of the agency's top 100 discoveries of the preceding year -- but are actually meaningless.
"We concluded that both basic and applied research can be evaluated meaningfully on a regular basis," said Griffiths. "But it is important that agencies evaluate their research programs using measurements that match the character of the research." Differences in character will lead to differences in the appropriate time scale for measurement; in what is measurable and what is not; and in the expertise needed by those who contribute to the measurement process, the report says.
All federal agencies are mandated by law, under the Government Performance and Results Act, to set goals and use performance measures to encourage greater efficiency, effectiveness, and accountability. The Act requires that agencies write strategic plans with annual performance targets and produce an annual report that demonstrates whether these targets are met. The first performance reports are due in March 2000.
USE OF EXPERT REVIEW
The most effective means of evaluating federally funded research programs, the
committee said, is expert review, which should be used to assess both basic and
applied research. The committee outlined three forms of expert review and their
applications: quality review, relevance review, and benchmarking.
To assess quality, peer review should be used, the committee said. Peer reviewers should include scientists at agency, university, and industrial laboratories who have participated in and who best understand federally funded research programs. However, peer review as it currently takes place in most federal agencies varies greatly and should be analyzed and modified as necessary. Relevance review and international benchmarking also are needed.
Relevance review draws on not only the views of experts in the field but also potential users and experts in related fields, to evaluate the relevance of the research to an agency's mission. Benchmarking reviews use panels of experts from the United States and elsewhere to judge the international leadership status of the United States in a program.
BASIC VS. APPLIED RESEARCH
Basic research involves theoretical or experimental investigation to advance
scientific knowledge, without immediate practical application as a direct
objective. For example, basic research by physicists on atomic structure more
than 60 years ago led to today's Global Positioning System. Scientists could
not know at the time that their work decades later would help create a network
of satellites that could send signals to a receiver, costing a few hundred
dollars, and instantly pinpoint someone's location on the planet to within 100
feet.
Because many years can pass before an advancement is achieved, the outcomes or results of basic research cannot be measured on an annual basis but only in retrospect, the committee concluded. However, agencies can regularly assess the progress of basic research in terms of quality and relevance to agency goals and intended users. Another proposed measure is leadership -- that is, whether the research is being performed at the forefront of scientific and technological knowledge, and leads the world in that particular field.
Applied research uses knowledge gained through theoretical or experimental investigation to make things or create situations that will serve a practical purpose. Programs in applied research usually include a series of milestones to be reached by particular times, and a description of the intended outcomes as well as their significance to society. Progress toward these milestones can be measured annually, the committee found. For example, if the U.S. Department of Energy adopted the goal of producing cheaper solar energy, it could measure the results of research designed to decrease the costs.
To produce and benefit from advances in science and technology, the nation also must have a continuing supply of well-educated and highly trained scientists and engineers, the committee said. Strategic and performance plans should focus more attention on the goal of developing and maintaining human talent in fields critical to the agencies' missions. To ensure this, agencies should require and evaluate education and training components in research programs.
The science and engineering communities also should play an important role in the implementation of the Government Performance and Results Act. As a first step, they should become familiar with agency strategic and performance plans, which are available on agency web sites. Many researchers contribute much time and effort to review papers submitted for publication, grant applications, and program proposals that are supported by federal funds, but few of them are aware of the Act, the committee said.
In conducting its study, the committee reviewed and assessed the strategic and performance plans of 10 federal agencies -- the U.S. departments of Agriculture, Defense, Energy, and Transportation as well as the National Institutes of Health (NIH), the National Science Foundation (NSF), NASA, the Environmental Protection Agency, the National Institute of Standards and Technology, and the National Oceanic and Atmospheric Administration. Over the course of the 12-month study, workshops were held to gather information, including industry methods to evaluate research performance.
The study was sponsored by the National Research Council, NSF, NIH, NASA, and the U.S. departments of Agriculture, Transportation, and Defense. The National Academy of Sciences, National Academy of Engineering, and Institute of Medicine are private, non-profit institutions that provide science, technology, and health policy advice under a congressional charter. A committee roster follows.
*Copies of EVALUATING FEDERAL RESEARCH PROGRAMS: RESEARCH AND THE GOVERNMENT PERFORMANCE AND RESULTS ACT are available from the National Academy Press at the mailing address in the letterhead; tel. 202-334-3313 or 1-800-624-6242. The cost of the report is $18.00 (prepaid) plus shipping charges of $4.00 for the first copy and $.50 for each additional copy. Reporters may obtain a copy from the Office of News and Public Information. More information about the report and the committee's work is available at HTTP://WWW2.NAS.EDU/COSEPUP.
NATIONAL ACADEMY OF SCIENCES
NATIONAL ACADEMY OF ENGINEERING
INSTITUTE OF MEDICINE
Policy Division
COMMITTEE ON SCIENCE, ENGINEERING, AND PUBLIC POLICY
PHILLIP A. GRIFFITHS(1) (CHAIR)
Director
Institute for Advanced Study
Princeton, N.J.
BRUCE M. ALBERTS(1)
President
National Academy of Sciences
Washington, D.C.
PETER DIAMOND(1)
Professor of Economics
Massachusetts Institute of Technology
Cambridge
GERALD DINNEEN(2)
Vice President
Science and Technology
Honeywell Inc. (retired)
Edina, Minn.
MILDRED S. DRESSELHAUS(1,2)
Institute Professor of Electrical Engineering and Physics
Massachusetts Institute of Technology
Cambridge
JAMES J. DUDERSTADT(2)
President Emeritus and
University Professor of Science and Engineering
Millennium Project
University of Michigan
Ann Arbor
MARYE ANNE FOX(1)
Chancellor
North Carolina State University
Raleigh
RALPH E. GOMORY(1,2)
President
Alfred P. Sloan Foundation
NEW YORK CITY
RUBY P. HEARN(3)
Vice President
Robert Wood Johnson Foundation
Princeton, N.J.
PHILIP W. MAJERUS(1,3)
Professor of Medicine, Biochemistry, and Molecular Biophysics and Director,
Division of Hematology-Oncology
Washington University School of Medicine
St. Louis
JUNE E. OSBORN(3)
President
Josiah Macy Jr. Foundation
New York City
KENNETH I. SHINE(3)
President
Institute of Medicine
Washington, D.C.
MORRIS TANENBAUM(2)
Vice Chairman and Chief Financial Officer
AT&T (retired)
Short Hills, N.J.
WILLIAM JULIUS WILSON(1)
Malcolm Wiener Professor
Center for Social Policy
John F. Kennedy School of Government
Harvard University
Cambridge, Mass.
WILLIAM A. WULF(2)
President
National Academy of Engineering
Washington, D.C.
STAFF
RICHARD BISSELL
Executive Director, Policy Division
DEBORAH STINE
Study Director
ANNE-MARIE MAZZA
Senior Program Officer
_________________________________________
1 Member, National Academy of Sciences
2 Member, National Academy of Engineering
3 Member, Institute of Medicine