Improving Defense Acquisition Processes with Evidence-based Analysis: An Illustrative Case Using the DoD SBIR Program

Report Number: NPS-AM-10-056

Series: Acquisition Management

Category: Small Business

Report Series: Proceedings Paper

Authors: Toby Edison

Title: Improving Defense Acquisition Processes with Evidence-based Analysis: An Illustrative Case Using the DoD SBIR Program

Published: 2010-04-01

Sponsored By: Acquisition Research Program

Status: Published--Unlimited Distribution

Research Type: Other Research Faculty

Full Text URL: http://acquisitionresearch.net/files/FY2010/NPS-AM-10-056.pdf

Keywords: Small Business Innovation Research (DoD SBIR)

Abstract:

This paper proposes and demonstrates that experimental and quasi-experimental program evaluation methods can be applied to some parts of the defense acquisition system to provide evidence of program effectiveness. The specific example presented is a quasi-experimental evaluation of the Department of Defense Small Business Innovation Research (DoD SBIR) program. Quasi-experimental methods are a set of program evaluation techniques that allow researchers to approximate the results of an experimental study, such as a randomized controlled trial, without performing the experiment. The paper performs a quasi-experimental evaluation of the DoD SBIR program, which provides evidence that the program is effective at transitioning SBIR-funded technologies into other DoD programs. This demonstration that quasi-experimental methods can be used to evaluate certain aspects of the DoD acquisition system provides policy analysts with new tools to meet Congressional requirements for acquisition system evaluation. The paper recommends that more quasi-experimental studies be conducted and actual experimental studies be executed. These methods can help the DoD overcome the well-documented deficiency in evaluating the effectiveness of its acquisition systems. The Office of Management and Budget, the Government Accountability Office and the House Armed Services Committee unanimously agree that the DoD does not objectively measure the performance of its acquisition system.