ARPA-E making progress toward achieving mission, says new assessment

WASHINGTON -- The Advanced Research Projects Agency-Energy (ARPA-E) is making progress toward achieving its statutory mission and goals, says a new congressionally mandated report by the National Academies of Sciences, Engineering, and Medicine. ARPA-E has funded research that no other funding source was supporting at the time, and the results of some of these projects have received follow-on funding from private and other public sources for various technologies, the report says.

The agency cannot reasonably be expected to have completely fulfilled its goals given so few years of operation and the size of its budget, the report says. The development of new energy technologies usually requires decades of effort, and decades also must usually elapse before there is evidence that innovations are truly transformative.

The report offers recommendations to guide ARPA-E in building upon its early success, urging the agency to continue its flexible management approach and risk-taking culture, to develop a framework for measuring its impacts, and to streamline its reporting processes.

"ARPA-E has made significant contributions to energy R&D that likely would not take place absent the agency's activities," said Pradeep Khosla, chair of the committee that wrote the report and chancellor of the University of California, San Diego. "We hope our report's recommendations will benefit the agency and the nation as ARPA-E continues to evolve and improve its operations."

Congress created ARPA-E within the U.S. Department of Energy with the mission "to overcome the long-term and high-risk technological barriers in the development of energy technologies." When Congress authorized ARPA-E in the 2007 America COMPETES Act, it requested an early assessment following 6 years of operation to examine the agency's progress toward achieving its statutory mission and goals. The Academies' new report documents the results of that assessment.

While the full market impacts of ARPA-E-funded technologies will not be seen for years, some intermediate outcomes are evident now, the report says. One quarter of the supported project teams or technologies have received follow-on funding for continued work. Roughly half of supported projects have published the results of their research in peer-reviewed journals, and about 13 percent have obtained patents. All of these are positive indicators for technologies on a trajectory toward commercialized products.

ARPA-E is in many cases successfully enhancing the economic and energy security of the United States by funding transformational activities, white space (technology areas that are novel or underexplored and unlikely to be addressed by the private sector or by other federal research programs), and feasibility studies to open up new technological directions and evaluate the technical merit of potential directions, the report says.

Importantly at this early stage, the committee found no signs that ARPA-E is failing to deliver on its mission and goals, on a path to failing, or in need of reform. In fact, attempts to reform the agency - such as applying pressure for ARPA-E to show short-term successes rather than focusing on its long-term mission and goals - would pose a significant risk of harming its efforts and chances of achieving its mission and goals, the report says.

The report also recommends that the ARPA-E director and program directors develop and implement a framework for measuring and assessing the agency's impact in achieving its mission and goals as soon as practicable, providing the agency with greater ability to demonstrate its value and impact. While the agency has in place an extensive data gathering and record keeping system at the project level, its system for collecting, tracking, and reporting publicly available high-level innovation metrics -- publications, funding from other sources, and disclosures and patents over time - is less extensive. The agency could link data from its robust internal database of project-level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to standard, observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals.

ARPA-E's program directors have been empowered to take risks in selecting projects, in line with the agency's mission, and to actively manage ongoing projects, which includes altering project milestones, budgets, and timelines. ARPA-E should strive to preserve this management approach, the report says. In addition, one of ARPA-E's strengths is its focus on funding high-risk, potentially transformative technologies and overlooked "off-roadmap" opportunities pursued by neither private firms nor other funding agencies, including other programs and offices within DOE. ARPA-E leadership and the secretary of energy should actively work to sustain this culture.

ARPA-E is a positive agent of change in DOE and the federal government, and its best practices are being adopted in some DOE offices, the report says. For example, the Office of Energy Efficiency and Renewable Energy has incorporated several elements of ARPA-E's approach into management of its own programs, including use of a workshop to define a program, use of concept papers to screen funding applicants, and early termination of underperforming projects. The secretary of energy should ensure that other offices and programs within DOE continue to explore and adopt elements of ARPA-E's practices that can improve the department's operations.

In addition, the report recommends that ARPA-E reconceptualize its "tech-to-market" program, which aims to provide researchers with business and other non-technical support and advice to help ensure that projects develop with market needs in mind. The agency considers this program to be an ongoing experiment, and the challenge of developing such a program may be greater than originally thought. The roughly three-year time frame of an ARPA-E project is too short to expect a technology to move from concept to market. ARPA-E should reconfigure the program to account for the wide variation in support needed across programs and performers with respect to prospective funding, commercialization, and deployment pathways. For example, ARPA-E should consider making full tech-to-market plans optional, but requiring performers to describe potential product applications if they can prove technological feasibility.

The report also recommends that the agency streamline its reporting requirements. The "high touch" nature of ARPA-E, where program directors review progress of those performing ARPA-E projects quarterly, is a hallmark of the agency. However, since the quarterly written reporting required can be challenging, ARPA-E should consider changing the process so that it consists only of quarterly presentations to the program directors and their teams, with only fourth-quarter or annual reports providing full written details.

###

The study was sponsored by the Advanced Research Projects Agency-Energy (ARPA-E) and the Alfred P. Sloan Foundation. The National Academies of Sciences, Engineering, and Medicine are private, nonprofit institutions that provide independent, objective analysis and advice to the nation to solve complex problems and inform public policy decisions related to science, technology, and medicine. They operate under an 1863 congressional charter to the National Academy of Sciences, signed by President Lincoln. For more information, visit http://national-academies.org. A committee roster follows.

Contacts:

Sara Frueh, Media Relations Officer
Joshua Blatt, Media Relations Assistant
Office of News and Public Information
202-334-2138; e-mail news@nas.edu
national-academies.org/newsroom/
Follow us on Twitter @theNASEM

Copies of An Assessment of ARPA-E are available from the National Academies Press on the Internet at http://www.nap.edu or by calling 202-334-3313 or 1-800-624-6242. Reporters may obtain a copy from the Office of News and Public Information (contacts listed above).

Policy and Global Affairs Division
Board on Science, Technology, and Economic Policy
and
Division on Engineering and Physical Sciences

Board on Energy and Environmental Systems

Pradeep K. Khosla1 (chair)
Chancellor
University of California
San Diego

Maxine L. Savitz1 (vice chair)
General Manager of Technology/Partnerships (retired)
Honeywell Inc.
Los Angeles

Pierre Azoulay
International Programs Professor of Management and Professor of Technological Innovation, Entrepreneurship, and Strategic Management
Sloan School of Management
Massachusetts Institute of Technology
Cambridge

Erica R.H. Fuchs
Professor
Department of Engineering and Public Policy
Carnegie Mellon University
Pittsburgh

Mark E. Jones
Executive External Strategy and Communications Fellow
The DOW Chemical Co.
Midland, Mich.

Eric Landree
Senior Engineer
The RAND Corp.
Arlington, Va.

Gilbert E. Metcalf
Professor of Economics
Department of Economics
Tufts University
Medford, Mass.

John Plaster
Managing Director of Global Power and Utilities Group, and
Head of Alternative Energy
Barclays Capital
New York City

Louis Schick
Founding Partner and Chief Technology Officer
NewWorld Capital Group LLC
New York City

Charles V. Shank1,2
Senior Fellow
Janelia Farm Research Campus
Howard Hughes Medical Institute
Ashburn, Va.

Stephanie S. Shipp
Deputy Director and Research Professor
Social and Decision Analytics Laboratory at the Virginia Bioinformatics Laboratory
Virginia Polytechnic Institute and State University
Alexandria

Scott Stern
David Sarnoff Professor of Management of Technology and Chair of the Technological Innovation, Entrepreneurship, and Strategic Management Group
Sloan School of Management
Massachusetts Institute of Technology
Cambridge

John C. Wall1
Vice President and Chief Technology Officer (retired)
Cummins Inc.
Belvedere, Calif.

Valerie Williams
Senior Program Evaluator of Community Programs
University Corporation for Atmospheric Research
Boulder, Colo.

Peidong Yang2
S.K. and Angela Chan Distinguished Professor of Energy and Professor of Chemistry
Department of Chemistry
University of California
Berkeley

STAFF

Paul Beaton
Staff Officer

1 Member, National Academy of Engineering
2 Member, National Academy of Sciences