Skip to main content

Defense Acquisitions: DOD Needs to Develop Performance Criteria to Gauge Impact of Reform Act Changes and Address Workforce Issues

GAO-10-774 Published: Jul 29, 2010. Publicly Released: Jul 29, 2010.
Jump To:
Skip to Highlights

Highlights

In May 2009, Congress passed the Weapon Systems Acquisition Reform Act of 2009 (Reform Act). The Reform Act contains a number of systems engineering and developmental testing requirements that are aimed at helping weapon programs establish a solid foundation from the start of development. GAO was asked to examine (1) DOD's progress in implementing the systems engineering and developmental testing requirements, (2) views on the alignment of the offices of the Directors of Systems Engineering and Developmental Test and Evaluation, and (3) challenges in strengthening systems engineering and developmental testing activities. In conducting this work, GAO analyzed implementation status documentation and obtained opinions from current and former DOD systems engineering and testing officials on the placement of the two offices as well as improvement challenges.

Recommendations

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Defense In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to ensure development and implementation of performance criteria for systems engineering plans and developmental test and evaluation master plans, such as those related to technology maturity, design stability, manufacturing readiness, concurrency of development and production activities, prototyping, and the adequacy of program resources.
Closed – Implemented
The Deputy Assistant Secretary for Systems Engineering issued guidance on the development of performance criteria in the new Systems Engineering Plan Outline Version 1.0, dated April 2011. Development and implementation of performance criteria must be documented in all plans requiring the Deputy Assistant Secretary for Systems Engineering's approval. The Director of Developmental Testing has identified an initial set of performance criteria that have been piloted on two programs. Based on results of the pilot, the office plans to update the metrics and begin using them throughout 2012 on programs that will be included in the 2012 annual report. The Acting Deputy Assistant Secretary for Developmental Test and Evaluation reported to Congress in March 2013 that the developmental testing office has taken several actions to establish measurable performance criteria and associated metrics. First, in fiscal year 2010, the office developed framework, which contained performance criteria and metrics, to assess individual program performance. Next, the office pilot tested the framework on two major defense acquisition programs in fiscal year 2011 and the results were used to refine the initial set of performance criteria and associated metrics. The effort produced a framework of 11 performance criteria: six performance criteria for program developmental test and evaluation performance and five performance criteria for overall developmental test and evaluation performance. Finally, the office applied the revised framework to the 46 programs it selected to report on in its fiscal year 2012 report to the Congress. The Deputy Assistant Secretary for Developmental Test and Evaluation plans to fully integrate the performance measures into assessment and decision support processes in the future and will review the framework on a periodic basis and make adjustments as necessary.
Department of Defense In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to track the extent to which program offices are adopting systems engineering and developmental testing recommendations.
Closed – Implemented
The systems engineering office has begun to track the extent to which recommendations included in program support reviews are being adopted. According to the Deputy Assistant Secretary for Systems Engineering, an average of 86 percent of actionable recommendations included in 17 program support reviews conducted in fiscal year 2010 were accepted by program managers. The developmental testing office has begun to track the extent to which recommendations to programs are being adopted. According to the Deputy Assistant Secretary for Developmental Test and Evaluation office, an average of 50 percent of recommendations in fiscal year 2011 were adopted. The office will report updated information in their 2012 annual report.
Department of Defense In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to work with the services to determine the appropriate number of government personnel needed to perform the scope of systems engineering and developmental testing activities.
Closed – Implemented
In its March 2012 joint annual report to Congress, the systems engineering office provided a complete analysis of the current and future workforce plans for each service. Specifically, the Deputy Assistant Secretary for Systems Engineering reported that the services had almost 38,000 civilian and military acquisition personnel for fiscal year 2011 to perform systems engineering activities and that existing authorities were sufficient to support high-priority and critical positions in the military department systems engineering workforce. In the March 2014, the Deputy Assistant Secretary for Systems Engineering reported that the military departments had about the same number of people, but that the services will need to prioritize resources including funding to recruit, train, and retain critical systems engineering skills sets. In its March 2014 annual report to the Congress, the Deputy Assistant Secretary for Developmental Test and Evaluation provided an assessment of Army, Navy, and Air Force Lead developmental test and evaluation organizations to determine whether the current resources were adequate for performing the scope of developmental testing activities. Specifically, the Deputy Assistant Secretary for Developmental Test and Evaluation reported the total number of civilian and military personnel within the acquisition test and evaluation workforce was approximately 8,200. Additionally, the Deputy Assistant Secretary assessed 32 acquisition programs, such as the Joint Strike Fighter and Littoral Combat Ship, in fiscal year 2013 and determined that they all had adequate test and evaluation resources to support their respected services' near-term priorities. However, six of these programs were found to have the potential for gaps in test and evaluation capabilities that need to be addressed to support longer terms needs.
Department of Defense In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to develop plans for addressing the training needs of the new hires and contractors who are expected to be converted to government personnel.
Closed – Implemented
Both the Deputy Assistant Secretary for Systems Engineering and Developmental Test and Evaluation sponsored Defense Acquisition University course development and class revisions in fiscal year 2010. The offices are identifying any skills gaps that may exist between the workforce's current capabilities and those needed to meet future mission requirements; and shape future workforce training, planning, and development. They will report on the status of their actions in the 2011 Joint Annual Report.
Department of Defense In order to determine the effectiveness of the newly established offices, the Secretary of Defense should direct the Directors of Systems Engineering and Developmental Test and Evaluation to report to Congress on the status of these efforts in future joint annual reports required by the Reform Act.
Closed – Implemented
The 2011 Joint Annual Report included information on military service efforts to grow their respective SE and DT workforces and train new employees. The directors plan to help the services address problems in these areas. The Deputy Assistant Secretaries did not provide information on the extent to which programs are implementing their recommendations. DOD stated in their response to our report that they plan to track implementation status, but would not report that information formally to the Congress in the joint annual report. We have obtained information that confirms that the Deputy Assistant Secretary for Systems Engineering is tracking implementation status. The Deputy Assistant Secretary for Developmental Test and Evaluation is also tracking implementation status. They plan to continue to report on actions in the 2012 Joint Annual Report.

Full Report

Office of Public Affairs

Topics

Defense procurementDevelopmental testingEngineeringEvaluation criteriaFederal agency reorganizationOperational testingPerformance measuresProcurement planningProgram evaluationResearch and developmentSystems designSystems evaluationWeapons research and developmentWeapons systemsPolicies and proceduresProgram goals or objectives