ETV Quality & Management Plan

EPA Report No: EPA/600/R-98/064

May 1998

ENVIRONMENTAL TECHNOLOGY VERIFICATION PROGRAM
QUALITY AND MANAGEMENT PLAN FOR THE PILOT PERIOD (1995-2000)  
National Risk Management Research Laboratory National Exposure Research Laboratory Office of Research and Development U.S. Environmental Protection Agency Cincinnati, Ohio 45268    

APPROVED BY

ACKNOWLEDGMENTS

The first draft of this document was developed by a team of writers consisting of the following quality assurance staff members from the U.S. Environmental Protection Agency Office of Research and Development National Risk Management Research Laboratory and National Exposure Research Laboratory: Sam Hayes, Lora Johnson, Ann Kern, and Jeff Worthington.

Subsequent revisions included significant input in the form of comments from members of the Environmental Technology Verification Team. Verification partners also provided comments. The team of writers were joined by the following EPA staff in development of this final document: Nancy Adams, Penelope Hansen, Linda Porter, and Shirley Wasson.

The HTML version of this document was prepared in July 1998 by Jeremiah McBurrows, an apprentice in the Minority Research Apprenticeship Program of the U. S. Environmental Protection Agency and the University of Cincinnati.

TABLE OF CONTENTS

ABBREVIATIONS AND ACRONYMS

INTRODUCTION

PART A

MANAGEMENT SYSTEMS

1.0 MANAGEMENT AND ORGANIZATION

1.1 ETV quality policy

1.2 Organization structure

1.3 ETV customer identification and ETV customer needs/expectations/work objectives

1.4 Potential verification partners

1.5 Management resolution for verification partner quality constraints

1.6 Resources

1.7 Authority to stop work for safety and quality considerations

2.0 QUALITY SYSTEM AND DESCRIPTION

2.1 Authorities and conformance to E4 quality standard

2.2 Quality system documents

2.3 Quality system scope

2.4 Quality expectation for products and services

2.5 Quality procedures documentation

2.6 Quality controls

2.7 Management system reviews (MSRs)

3.0 PERSONNEL QUALIFICATION AND TRAINING

3.1 Personnel training and qualification procedures

3.2 Formal qualifications and certifications

3.3 Technical management and training

3.4 Retraining

3.5 Personnel job proficiency

4.0 ETV VERIFICATION PARTNER SELECTION

4.1 Planning and control of selection process

4.2 Technical and quality requirements

4.3 Quality specification/conformance

4.4 Peer review of assistance agreements

4.5 Conformance of verification testing efforts

5.0 RECORDS

5.1 Scope

5.2 Preparation, review, approval, and distribution

5.3 Records storage and obsolete records

6.0 COMPUTER HARDWARE AND SOFTWARE

6.1 General procedures

6.2 Scope of ETV computer hardware/software procedures

6.3 Configuration testing

6.4 Measurement and testing equipment configurations

6.5 Change assessments - configurations, components, and requirements

6.6 ETV website roles and responsibilities

7.0 PLANNING

7.1 Systematic planning process

7.2 Planning document review

8.0 IMPLEMENTATION OF WORK PROCESSES

8.1 Implementation

8.2 Procedures

8.3 Oversight

9.0 ASSESSMENT AND RESPONSE

9.1 Numbers and types of assessments

9.2 Procedures

9.3 Personnel qualifications, responsibility, and authority

9.4 Response

10.0 QUALITY IMPROVEMENT

10.1 Annual review for quality improvement

10.2 Detecting and correcting quality system problems

10.3 Cause and effect relationship

10.4 Root cause

10.5 Quality improvement action

PART B

COLLECTION AND EVALUATION OF ENVIRONMENTAL DATA

1.0 PLANNING AND SCOPING

1.1 Systematic planning of the verification test

1.2 Systematic planning for verification testing

2.0 DESIGN OF TECHNOLOGY VERIFICATION TESTS

2.1 Design process

3.0 IMPLEMENTATION OF PLANNED OPERATIONS

3.1 Implementation of planning

3.2 Services and items

3.3 Field and laboratory samples

3.4 Data and information management

4.0 ASSESSMENT AND RESPONSE

4.1 Assessment types

4.2 Assessment frequency

4.3 Response to assessment

5.0 ASSESSMENT AND VERIFICATION OF DATA USABILITY

5.1 Data validation

5.2 Existing data

5.3 Reports reviewed

REFERENCES

APPENDIX A

APPENDIX B

APPENDIX C

DOCUMENTS AND GENERAL TERMS

Return to Table of Contents

Annual progress report
The report developed on an annual basis to report implementation of the ETV program.

Directors of Quality Assurance
Quality assurance directors for the EPA ORD laboratories, the National Exposure Research Laboratory and the National Risk Management Research Laboratory

E4
The ANSI/ASQC national consensus standard, which is the Agency standard, is applicable to assistance agreements. The standard is entitled, E4-1994, Specifications and Guidance for Quality Systems for Environmental Data Collection and Environmental Technology Programs.

ETV assistance agreement
The contractual record developed by the EPA and signed by the verification partner.

ETV coordinator
The EPA person designated by EPA ORD to lead the ETV team.

ETV verification statement
A summary statement, developed by the verification partner and approved by the EPA pilot manager, which reports individual technology performance.

ETV verification report
The report of the result of an individual technology test.

EPA line management
The management structure to whom each EPA pilot manager reports; i.e. branch chief, division director, laboratory director.

EPA pilot manager
The EPA person designated by EPA line management to serve as the lead for an individual ETV pilot.

EPA pilot quality managers
The EPA quality assurance person designated by EPA line management to manage quality assurance efforts on behalf of the pilot manager.

EPA review/audit reports
The “quality records” developed by EPA as a result of conducting assessments during ETV implementation.

ETV team
EPA employees actively working on the ETV program; the ETV coordinator, pilot managers, and the directors of quality assurance are core members

ETV test objective
The stated objective(s) of each technology test. Verification partners use the DQO process to establish test objectives and test measurement quality criteria.

ETV webmaster
The person designated by EPA line management with responsibility for establishing and maintaining the ETV website.

Evaluation contractor
The contractor selected to collect information on pilot performance.

Generic verification protocol
Those protocols developed, modified, or selected to promote uniform testing for a single pilot operated by the verification partner. Adequate documentation of a robust protocol may allow the development of abbreviated individual test/QA plans which incorporate the generic verification protocol by reference.

Laboratory director
The directors of EPA ORD laboratories, the National Exposure Research Laboratory and the National Risk Management Research Laboratory.

Management system review
The qualitative assessment of data collection operation and/or organization(s) to evaluate the adequacy of the prevailing quality management structure, policies, practices, and procedures for obtaining the type and quality of data needed are obtained.

Office of Research and Development Assistant Administrator
The administrative lead person directing the EPA’s Office of Research & Development.

Quality and Management Plan for the Pilot Period
The specific policies and procedures for managing quality related activities in the ETV program.

Raw data
All data and information recorded in support of analytical and process measurements made during planning, testing, and assessing environmental technology including support records such as: computer printouts, instrument run charts, standards preparation records, field log records, technology operation logs, and monitoring records. ETV test files (all records including raw data) and technical data and associated quality control data which support the data summarized and the conclusions made in each ETV verification report.

Records
All books, papers, maps, photographs, machine readable materials, or other documentary materials, regardless of physical form or characteristics, made or received by the EPA or a verification partner or their designated representative for the ETV program.

Stakeholder groups
Groups set up for each pilot area consisting of representatives of any or all of the following verification customer groups: buyers and users of technology, developers and vendors, the consulting engineering, the finance and export communities, and government permitters and regulators.

Standard operating procedures
Procedures describing routine verification activities including sample collection, analytical testing, and associated verification process.

Test/QA plan
The plan developed by a verification partner for each individual test of a technology or technology class. Therefore, the test/QA plan may include more than one technology. The test/QA plan provides the experimental approach with clearly stated test objectives and associated quality objectives for the related measurements. The test/QA plan may incorporate or reference generic verification protocols.

Test measurement
Those critical measurements that must be made during the course of a technology test to evaluate achievement of the ETV test objective.

Verification
Establishing or proving the truth of the performance of a technology under specific, predetermined criteria or protocols and adequate data quality assurance procedures.

Verification partners
The public and private sector organizations holding cooperative or interagency agreements to assist EPA in implementing the ETV program.

Verification partner manager
The person designated by the verification partner to manage the pilot and serve as the chief point of contact with the EPA.

Verification partner quality manager
The person designated by the verification partner to manage quality assurance for the pilot on behalf of the verification partner manager.

Verification partner quality management plan
The procedures for quality related activities developed and implemented by the verification partner to assure quality in the work processes and services developed for ETV. If the verification partner has a current quality system that accommodates ETV’s needs, additional quality system elements do not need to be developed.

VP review/audit reports
The “quality records” developed by the verification partner as a result of conducting assessments during ETV implementation.

Verification Strategy
The ETV Program Verification Strategy, first published in February 1997, outlines the pilot period goals, programmatic operating principles, pilot selection criteria, key definitions, budgets, and implementation activities that are molding the ETV program, as well as the challenges that are emerging and the decisions that need to be addressed in the future.

ABBREVIATIONS AND ACRONYMS

Return to Table of Contents
 
 

INTRODUCTION

Return to Table of Contents

Background

The Environmental Technology Verification Program (ETV) has been established by the Environmental Protection Agency (EPA) to evaluate the performance characteristics of innovative environmental technologies across all media and to report this objective information to the permitters, buyers, and users of environmental technology. ETV has evolved in response to the following mandates:
 

• A directive to EPA by the President in his 1995 environmental technology strategy, Bridge to a Sustainable Future, to “work with the private sector to establish a market-based verification process... which will be available nationally for all environmental technologies within three years.
• Goals articulated in the Vice-President's Reinventing Government; A Performance Review which directed EPA to begin a comprehensive environmental technology verification program no later than October 1995.
• Congressional appropriation language contained in the FY96 and FY97 budgets, that the Agency fund technology verification activities at the $10 million level in each year.

To comply with these directives, EPA's Office of Research and Development (ORD) has established a five year pilot program to evaluate alternative operating parameters and determine the overall feasibility of a technology verification program. ETV began in October 1995 and will be evaluated through October 2000, at which time the Agency will prepare a report to Congress containing the results of the pilot program and recommendations for its future operation.

Program Description
Developers of innovative environmental technology report numerous impediments to commercialization. Among those most frequently mentioned is the lack of acceptance of vendor performance claims. It is believed that objective, independently acquired, high quality performance data and operational information on new technologies will significantly facilitate the use, permitting, financing, export, purchase, and general marketplace acceptance of such technologies. The purpose of ETV is to provide such data and information to the customer groups that require them in order to accelerate the real world implementation of improved technology. Better technology will more thoroughly, rapidly, and efficiently protect the environment. It is important to stress that the product of ETV is high quality data and information, not technology approval or endorsement.

The thesis that independent performance verification more rapidly moves new technology into use will be tested by EPA’s five year pilot program. ETV funds and operates twelve pilot projects, each operated by third party organizations under the auspices of EPA. These "partner organizations" include private sector testing, evaluation and research companies, state technology evaluation programs, federal laboratories, and industry associations. For the most part, each pilot is focused on a different environmental, industry, or technology sector (e.g., air pollution control technology, drinking water systems, field monitoring devices, industrial coatings products). By design, all pilots are operated in a somewhat different manner in order to test various methods for both technical and operational efficiency and effectiveness in verifying performance. Management techniques are in place to assure that constant evaluation of alternative methods occurs and results in continuous improvement of processes throughout the pilot period.

Because credible information is the ultimate product of ETV, the highest appropriate quality assurance procedures will be used throughout the program. The EPA's Office of Research and Development implements an Agency-wide quality system to assure that activities conducted in EPA research laboratories, other EPA research facilities or locations, or at facilities being operated on behalf of or in cooperation with the EPA are supported by data of known and acceptable quality for their intended use. Individual research laboratories develop laboratory-specific quality management plans. The National Risk Management Laboratory (NRMRL) and the National Exposure Research Laboratory (NERL) are implementing ETV in conformance with such plans.

Program and Quality Management Documents
Several documents define the overall operation of the program during its pilot stage. The first to be published (February 1997) was the Environmental Technology Verification Strategy. This document lays out goals, customer and key word definitions, basic operating principles, pilot selection criteria, and the programmatic and budgetary vision of the pilot program. The Strategy is intended to evolve over time and is now (May 1998) undergoing review to evaluate the need for modification and amplification.

The second major program management document being used by ETV to guide its operation is the ETV Quality and Management Plan (QMP) which follows this introduction. Under development for over a year, the ETV QMP uses the structure, policies and standards of the American National Standard ANSI/ASQC E4-1994, "Specification and Guidelines for Quality Systems for Environmental Data Collection and Environmental Programs." This document,

As of February 1996, all cooperative agreements entered into by EPA concerning environmental technology must be in conformance with the provisions of E4. This requirement is expected to be extended to EPA contracts in 1998#.

The ETV Quality and Management Plan
Based upon the structure and standards of E4, the ETV QMP lays out the definitions, procedures, processes, inter-organizational relationships, and outputs that will assure the quality of both the data and the programmatic elements of ETV during the pilot period. Part A of the ETV Quality and Management Plan contains the specifications and guidelines that are applicable to common or routine quality management functions and activities necessary to support the ETV program. Part B of the ETV Quality and Management Plan contains the specifications and guidelines that apply to test-specific environmental activities involving the generation, collection, analysis, evaluation, and reporting of test data.

EPA's verification program is organizationally complex, involving numerous outside organizations through its extensive stakeholder process, partner organizations who bear most of the quality assurance responsibilities, and testing and consulting companies hired by partner organizations to conduct field and laboratory work. Within EPA, the program is coordinated through ORD's ETV Team consisting of staff from ten Branches located in six Divisions of two Laboratories, NRMRL and NERL, along with quality assurance staff in each of the laboratories' physical locations. Finally, EPA program offices and regions are increasingly involved in outreach activities, as are states and other Federal agencies through the White House Environment and Technology Working Group. The ETV QMP is designed to play a major role in clearly delineating the roles and responsibilities of all of these diverse and important players.

All partner organizations will use this document and its parent, the E4 standard, to create quality management plans that assure appropriate levels of data collection, quality outputs, and customer responsiveness. These plans will be submitted to EPA for review and approval by pilot managers and quality assurance staff. It is not the purpose of this document to require that partner organizations create wholly new operating procedures solely for use under ETV. Most organizations selected by EPA as cooperators already have many of the procedural and process elements required by E4 incorporated into their existing management systems. Other requirements found in this document will be new or different. Cooperators should address all appropriate elements of the ETV QMP either specifically in their ETV plan or include appropriate and adequately detailed references to existing documents.

The ETV QMP will be reviewed on an annual basis throughout the pilot period (and beyond if the program is extended) to incorporate lessons learned from the experiences of the pilots and feedback from customer groups. The addition of new policies and elimination or modification of ineffective procedures will be discussed with all participants and modifications to partner QMP's may be required.

The ETV QMP follows the general outline of the ANSI/ASQC E4-1994 document.

____________________

*American Society for Quality; American National Standard and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs; Milwaukee, Wisconsin; 1994.

#All ETV pilots are executed under cooperative agreements with the exception of the Site Characterization and Monitoring and the Industrial Coatings and Coatings Equipment Pilot which utilize Interagency Agreements with the Departments of Energy and Defense.

____________________

PART A
MANAGEMENT SYSTEMS

Return to Table of Contents

Part A of the ETV Quality and Management Plan contains the specifications and guidelines that are applicable to common or routine quality management functions and activities necessary to support the ETV program.

Part B of the ETV Quality and Management Plan contains the specifications and guidelines that apply to test-specific environmental activities involving the generation, collection, analysis, evaluation, and reporting of test data.

1.0 MANAGEMENT AND ORGANIZATION

1.1 ETV quality policy

Return to Table of Contents

The Office of Research and Development shall establish and implement a quality policy to ensure that the Environmental Technology Verification (ETV ) program produces the type and quality of program outputs needed and expected by ETV clients.

The EPA Office of Research and Development's (ORD) quality policy for the Environmental Technology Verification (ETV) program is established as follows:

The quality system for the overall ETV program seeks to be consistent with industry consensus standards. Each verification partner shall implement a valid and approved quality system. As of February 1996, the Agency’s required quality system for cooperative agreements is ANSI/ASQC E4. Each verification test will be performed according to planned and documented, pre-approved test/QA plans. All technical statements in ETV verification reports shall be supported by the appropriate data.

Return to Table of Contents

The relevant organizations, functional responsibilities, levels of accountability and authority, and lines of communication shall be formally defined in the quality system and approved by the EPA laboratory directors responsible for the quality of work performed by or on behalf of each EPA laboratory.

The overall organizational structure of the ETV program graphically presents lines of accountability, authority, and communication. The general functional responsibilities for the major organizational units are specified in the structure.

To view a larger and more legible organization chart, click here.

1.2.1 Assistant Administrator for ORD and the USEPA Administrator responsibilities:

• provides overall program direction
• serves in a program leadership role with Congress, other agencies of the Executive Branch, and the general public

1.2.2 ORD laboratory directors responsibilities:

• approve and implement annual program budgets and resource allocations
• allocate laboratory personnel and other resources to accomplish ETV’s goals, including appointment of the ETV coordinator
• approve all ETV verification statements
• review and approve ETV Verification Strategy and ETV Quality and Management Plan (QMP)
• ensure that appropriate assessments (i.e., see Table 9.1) are implemented

1.2.3 Division directors and branch chiefs responsibilities:

• allocate appropriate division and branch personnel and other necessary resources to support pilots located in the division/branch
• provide oversight of pilot outputs and products prior to public release

1.2.4 ETV coordinator responsibilities:

• leads the ETV team by providing communication opportunities, e.g., periodic conference calls, meetings, and training
• coordinates the overall ETV program, including design of multi-year strategies, operating principles, implementation activities, and annual budgets
• communicates ETV team and program activities, progress, outputs, and recommendations to EPA, Congress, agencies in the Executive Branch, customer groups, and the general public
• maintains an up-to-date ETV website containing materials relevant to the program and to each pilot
• manages overall ETV program outreach activities to ensure that customer groups are knowledgeable about the existence and use of ETV generated data
• collects data on operational parameters and program outputs to continuously evaluate the ETV program and make recommendations to management and the Congress on its present and future operation.
• reviews, approves, and assists in revision of ETV QMP
• ensures ETV QMP is implemented in the ETV program

1.2.5 ETV team responsibilities:

• establish mutually acceptable program-level strategies and protocols
• participate in development of an overarching ETV outreach strategy
• communicate pilot-specific progress, issues, difficulties, and lessons learned
• meet to discuss program objectives, seek collegial guidance, and evaluate success
• review ETV QMP

1.2.6 EPA pilot managers responsibilities:

• select and oversee verification partners
• communicate requirements for and oversee the verification partner’s quality system
• arrange for peer review of verification partner proposals and ETV verification reports
• attend and/or conduct regular meetings with stakeholders
• oversee production and approval process of ETV verification reports and ETV verification statements
• assist with ETV outreach activities for the assigned ETV pilot
• participate in ETV team activities

1.2.7 Verification partners responsibilities:

• establish, attend, and/or conduct meetings of stakeholder
• maintain communication with EPA to assure mutual understanding and conformance with EPA quality procedures and expectations and ETV policies and procedures
• manage the oversight and conduct of verification activities
• assure that quality procedures are incorporated into all aspects of each ETV pilot
• develop, conduct, and/or oversee test/QA plans in cooperation with technology vendors
• solicit technology vendor proposals or vendor products
• operate ETV activities within their documented and approved quality management plan
• prepare ETV verification reports on technology tests
• prepare a three-to-five page ETV verification statement at the completion of each technology verification
• appoint a quality manager, responsible for ensuring that the VP’s organization and suppliers/contractors to the VP organization have quality systems in compliance with this QMP, and that the VP complies with their documented quality system.

1.2.8 Stakeholders group responsibilities may include the following:

• assist in development of generic verification protocols
• assist in prioritizing the types of technologies to be verified
• review pilot-specific procedures and selected ETV verification reports emerging from the ETV pilot
• assist in the definition and conduct of outreach activities appropriate to the technology area and customer groups
• serve as information conduits to the particular constituencies that each member represents

1.2.9 ETV directors of quality assurance responsibilities:

• develop and implement the ETV quality system at the direction of the ETV coordinator and in coordination with the ETV team
• document the ETV quality system in the ETV QMP
• review, and update, if necessary, the ETV QMP annually in cooperation with the ETV coordinator
• work with ETV pilot quality managers to ensure implementation of ETV QMP
• provide current copies of the ETV QMP to the appropriate participants in the ETV program
• communicate quality issues and information to the ETV team in a timely manner
• conduct internal management system reviews (MSR)

1.2.10 EPA pilot quality managers responsibilities:

• communicate quality system requirements, quality procedures, and quality issues to the assigned EPA pilot manager and verification partner
• review and comment on verification of partner’s quality system description to verify conformance to the quality provisions of this document
• perform MSR of each pilot’s quality system to verify conformance to the quality provisions of this document perform technical systems audits (TSA's) and performance evaluations (PEs) of pilots, as appropriate
• provide assistance to pilot personnel in resolving quality assurance issues

Tables available on the ETV website  present a current listing of the pilots that are either underway or soon to be awarded. Included in the table are the EPA pilot managers, EPA pilot quality managers, verification partner managers, and verification partner quality managers. The tables contain their names, geographic locations, ORD laboratory or company affiliations, and phone numbers.

1.3 ETV customer identification and ETV customer needs/expectations/work objectives

Return to Table of Contents

The ETV coordinator, pilot managers, and partners are responsible for coordinating the identification of customers and communicating the needs of the internal and external customers to ensure that ETV work products satisfy their needs.

1.3.1 As identified in the ETV Verification Strategy, external customers (i.e., outside EPA) include, but are not limited to:

• public and private sector buyers and users of technology
• developers and vendors of technology
• the consulting engineering community who recommend technologies to buyers
• federal, state and local government permitting/regulatory agencies
• international marketers and the financial community
• Congress

In a general sense, needs and expectations of external customers include:

• ETV verification reports and ETV verification statements supported by objective and reliable data, provided in a timely manner
• a justifiable approach to selecting technologies for testing
• a practical approach in testing which provides efficient, timely, and cost-effective technology tests
• user-friendly documents (e.g., easy to read and to implement)
• technology operation consistent with statements in the ETV verification report

For each pilot, needs and expectations of external customers are defined and documented in the minutes of stakeholders meetings. The process to define these pilot-specific needs and expectations includes:

• discussions between the EPA pilot managers, verification partners and stakeholders
• development by EPA pilot managers, verification partners and stakeholders of pilot objectives for use prior to testing

NOTE: Not all pilots are structured the same. For example, the independent pilot uses an advisory committee, an expert review, and an alliance group in determining pilot-specific needs and expectations.

1.3.2  Internal customers of the ETV program are those EPA staff responsible for execution of the ETV program in accordance with the expectations of Congress and the Administration. These customers include EPA and ORD senior managers who expect conformance with management and quality policies of the Agency.

Other EPA staff, such as EPA technical experts in the regions and headquarters, will benefit incidentally from the program in the following areas:

• data of known and useful quality
• expedited use of improved environmental technologies
• ETV testing accomplished on a wide variety of technologies
• user-friendly documents (e.g., easy to read and to implement)
• development of appropriate verification testing protocols

Return to Table of Contents

EPA line management shall oversee the selection of ETV verification partners.

Each ETV pilot seeks to evaluate a wide variety of verification partnerships, using both interagency and cooperative agreements. When cooperative agreement holders conduct the verification testing, they are competitively solicited using the Request for Application (RFA) process, whereby notice of EPA's intent to issue an RFA is published, typically in the Commerce Business Daily (CBD).

1.5 Management resolution for verification partner quality constraints

Return to Table of Contents

When necessary, appropriate EPA management shall negotiate acceptable measures of quality and success when constraints of time, costs, or other problems affect the verification partner's capability to fully satisfy customer's needs and expectations.

When constraints of time, costs, or other problems significantly affect the verification partner's capability to fully satisfy the ETV's quality system needs and expectations, EPA pilot managers negotiate with the verification partner by the following procedure to establish acceptable measures of quality and success:

• If any one of the above problems occur, the verification partner notifies the EPA pilot manager.
• The EPA pilot manager negotiates and documents an acceptable agreement with the verification partner.
• If agreement cannot be reached between the EPA pilot manager and the verification partner, the negotiation discussion will be elevated to the appropriate EPA branch chief, division director, and/or ETV coordinator.
• If agreement cannot be reached between EPA management and the verification partner, appropriate actions will be performed (e.g., increased funding for the benefit of the program, non-funding of extensions, or non-support for verifications statements).
• If negotiation of issues reaches a stalemate, the Grants Administration Division (GAD) will be notified for possible legal action.

Return to Table of Contents

The laboratory directors shall provide adequate resources to the ETV directors of quality assurance, EPA pilot managers and EPA pilot quality managers to enable them to plan, implement, assess, and improve the overall ETV program and quality system effectively.

Laboratory directors take the following actions to achieve the above policy:

• provide FTE allotment of EPA pilot managers
• provide FTE allotment of QA and other support personnel at each laboratory’s geographical location
• provide sufficient travel funds for each pilot for an appropriate level of oversight and external assessments
• provide for maintenance of communication lines between ORD laboratory directors, the ETV team, and the ETV coordinator

 Return to Table of Contents

The verification partner shall stop unsafe work and work of inadequate quality, or shall delegate the authority to do so to others.

The following procedures are necessary to stop unsafe work and work of inadequate quality:

• The verification partners shall ensure compliance with all federal, state, and local health and safety policies during the performance of the pilot tests. This includes obtaining appropriate permits.
• The verification partner's quality system shall identify one or more individuals who may issue a stop work order in the event that unsafe work or work of inadequate quality is identified.
• EPA pilot managers and EPA pilot quality managers shall contact the authorized individual(s) in the event that work of inadequate quality is discovered.
• In extreme circumstances, the EPA pilot managers may ask GAD to intervene if the verification partner does not implement their approved quality management plan.

Return to Table of Contents

A quality system shall be planned, established, documented, implemented, and assessed as an integral part of an ETV management system for environmental technology verification programs defined by ETV quality policy.

Development and subsequent endorsement of this plan by the ETV coordinator and EPA line management are evidence that the ETV quality system is planned, established, documented, implemented, and assessed as an integral part of an EPA ETV management system.

Return to Table of Contents

The ETV quality system shall address applicable parts of E4 and shall include the organizational structure, policies and procedures, responsibilities, authorities, resources, and guidance documents.

The authority for developing appropriate quality systems for ETV is USEPA Order 5360.1. The requirement for assistance agreement holders is found in Federal Register, CFR Parts 30 & 33, February 15, 1996.

This plan complies with ANSI/ASQC E4-1994, Specifications and Guidance for Quality Systems for Environmental Data Collection and Environmental Technology Programs, the Agency standard applicable to assistance agreements. E4 is comparable to the International Standards Organization (ISO) 9000 standards series, as shown in the comparison table provided in Annex B-5 to E4.

The ETV quality system addresses each applicable individual “specification” provided in the published quality standard, ANSI/ASQC E4-1994, using the policies and procedures in this plan, as appropriate.

Verification partners develop quality system descriptions to be consistent with both ANSI/ASQC E4-1994 (and/or ISO 9001) and/or this document.

Return to Table of Contents

The ETV quality system shall be described in a QMP that is reviewed and approved by the ETV coordinator and EPA line management.

The ETV quality system is described in this quality and management plan.

• The ETV team develops and implements the quality system.
• The ETV coordinator, ORD laboratory directors, and appropriate ORD division directors review and approve the ETV QMP and subsequent revisions to the plan, as policy for the ETV program.
• Verification partners' quality systems (which are consistent with ANSI/ASQC E4-1994) are described in a written ETV pilot quality management plan, and are reviewed and approved by verification partner management, the EPA pilot manager, and EPA pilot quality manager. Subsequent revisions are reviewed in a similar manner.

Return to Table of Contents

The ETV quality system description shall identify in general terms those items, programs, or activities to which it applies.

This quality system description applies to the following:

• the EPA ETV program
• selection and oversight of verification partners
• review and approval of verification partners’ quality management plans
• ETV products (e.g., test/QA plans, reports, ETV verification statements)
• planning, implementation, and assessment activities supporting ETV verification activities

Return to Table of Contents

The ETV quality system shall include provisions to ensure that products or results of the environmental programs defined by the ETV program are of the type and quality needed and expected by ETV clients.

The preeminent products of the ETV program are the environmental technology verification reports and statements issued by EPA and the verification partner. Provisions to ensure that these products and other results of the ETV program are of the quality expected include:

• Products are reviewed as described in part A, section 5.0.
• MSR's and technical assessments are conducted as described in part A, section 9.0. Technical assessments may include field and laboratory audits, performance evaluation audits, and audits of data quality.

Return to Table of Contents

Following approval of the ETV QMP, management elements of the quality system shall be implemented as described.

Verification partners must operate the ETV pilots under a written and EPA-approved quality management plan that is based on E4 and/or the provisions of this plan.

• Environmental technology pilots initiated prior to approval of the ETV QMP will be requested to provide quality system elements consistent with requirements of their assistance agreements with EPA. The ETV coordinator requests that existing pilots comply with the requirements set forth in this plan within 6 months of the date of this document.
• For pilots initiated after the approval of the ETV QMP, verification partners provide evidence of compliance before verification activities are started, as required by the assistance agreement signed by the verification partner.

The EPA pilot manager is responsible for obtaining a copy of the verification partner's quality management plan, as specified in the RFA, for their own review and forwarding the document to the EPA pilot quality manager for review and approval prior to planning technology tests.

2.6 Quality controls

Return to Table of Contents

The ETV quality system description shall define when and how controls are to be applied to specific technical or technology testing efforts and shall outline how these efforts are planned, implemented, and assessed.

2.6.1 ETV program controls include:

• existing EPA policies and procedures for selection and administration of verification partner efforts in ETV
• an approved ETV QMP
• quality management, assurance, and control procedures as part of the assistance agreement

2.6.2 Pilot-specific controls include:

• test/QA plans, developed and approved prior to testing
• oversight by the EPA pilot quality managers of the implementation process and follow-up to any finding of nonconformance
• technical operations assessment
• specified quality control requirements in the assistance agreement

Pilot-specific procedures for planning, implementation, and assessment are described in the verification partner's quality system. Procedures for planning, implementing, and assessing the overall ETV quality system are detailed in part A sections 7.0, 8.0, and 9.0 and in part B.

2.7 Management system reviews (MSRs)

Return to Table of Contents

At regular intervals (at least annually) the ETV quality system shall be reviewed and its description updated, if appropriate, to reflect changes in the organization as well as changes in ETV quality policy.

The ETV directors of quality assurance perform an internal MSR of the program in accordance with the process as outlined in part A section 9.0.

3.0 PERSONNEL QUALIFICATION AND TRAINING

3.1 Personnel training and qualification procedures

Return to Table of Contents

Personnel performing work shall be trained and qualified based on appropriate requirements prior to the start of the work or activity.

3.1.1 EPA pilot managers are selected based on:

• educational background and/or a degree that is directly relevant to the pilot technology area
• work experience specific to the pilot technology area
• experience in program management
• participation in required training for project officer responsibilities on assistance agreements, as documented in training records

3.1.2 EPA pilot quality managers are selected based on:

• educational background and/or a degree relevant to the technology tests and programs
• work experience specific to QA of technology tests and programs
• experience in quality management

3.1.3 Key participants working directly for or on behalf of the verification partner in support of the pilot and/or individual test operations are selected by the verification partner and evaluated by the EPA during the RFA process. RFA evaluation criteria for key personnel will vary, but typically include a consideration of the following:

• educational background and/or a degree(s) relevant to technical areas represented in the pilot
• work experience related to the technology areas represented in the pilot
• experience in quality management

The verification partner's documented quality management plan will address training and qualification procedures for pilot personnel.

3.2 Formal qualifications and certifications

Return to Table of Contents

The need to require formal qualification or certification of personnel performing certain specialized activities shall be evaluated and implemented where necessary.

ETV program management, quality management, and pilot management requires no formal qualification or certification other than where applicable:

• EPA Project Officer Training and Assistance Training (or Work Assignment Manager training as appropriate)
• Appropriate OSHA courses

Formal qualification or certification of personnel performing specialized activities for each pilot or for specific test/QA plans is addressed on a pilot-specific or test/QA plan-specific basis. Verification partners maintain records of the qualification or certification of such personnel.

NOTE: Requirements for formal qualifications or certification may be based on applicable federal, state, or local requirements associated with a particular test. Examples of possible certifications include but are not limited to drinking water plant operators certification, professional engineering registration, and certification of industrial hygienists.

3.3 Technical management and training

Return to Table of Contents

Appropriate technical and management training, which may include classroom and on-the-job, shall be performed and documented.

EPA line management is responsible for appropriate technical and management training for staff working on the ETV program. Such training will be documented in each individual's training file.

Verification partners are responsible for personnel training and qualification procedures for each pilot or for specific test/QA plans. Verification partners maintain the training records (available for review by EPA).

The ETV team will be trained at meetings occurring at least twice a year to develop a policy, share information and lessons learned. The directors of quality assurance provide training on the requirements of the ETV QMP during the periodic workshops organized by the ETV coordinator.

3.4 Retraining

Return to Table of Contents

When job requirements change, the need for retraining to ensure continued satisfactory job proficiency shall be evaluated.

Return to Table of Contents

Evidence of personnel job proficiency shall be documented and maintained for the duration of the technology test or activity affected, or longer if required.

3.5.1 EPA pilot managers - The existing performance standards of the EPA pilot managers may already include tasks consistent with the following items. These items should be considered for specific identification in the performance standards:

• active participation in the ETV team; communicating pilot issues, lessons learned, required reports, and appropriate assistance to members of the ETV team and management
• developing RFAs and/or management of cooperative agreements/IAGs
• facilitating stakeholders group activities
• ensuring development of and contributing to generic verification protocols and test/QA plans
• providing a leadership role to ensure technologies are selected consistent with the ETV Verification Strategy
• serving as a communication link between EPA and the verification partner, in particular, providing information and documents to support the ETV website
• reviewing draft and final ETV verification reports and other pilot documents
• reporting to program management the completeness and validity of the ETV verification statement prior to report issuance
• ensuring the timely delivery of complete and consistent ETV products and services

NOTE: Evaluations are the responsibility of the appropriate supervisor and are not a record of the ETV program.

3.5.2 EPA pilot quality managers -The existing performance standards of the EPA pilot quality managers may already include tasks consistent with the following items. These items should be considered for specific identification in the performance standards:

• verification partner pilot quality management plan review
• management system reviews
• technical system audits, performance evaluation audits, and audits of data quality
• verification reports and statements reviews
• providing complete and timely pilot audit reports

NOTE: Evaluations are the responsibility of the appropriate supervisor and are not a record of the ETV program.

3.5.3 Verification partner staff - Verification partners document and maintain records (such as annual performance reviews) of personnel job proficiency for work performed directly in support of the verification partner’s ETV activities.

NOTE: Evaluations are the responsibility of the verification partner and are not a record of the ETV program.

4.0 ETV VERIFICATION PARTNER SELECTION 

4.1 Planning and control of selection process

Return to Table of Contents

Funding of assistance associated with the ETV program shall be planned and controlled to ensure that the quality of verification tests is known, documented, and meets technical requirements and acceptance criteria of the clients.

The ETV program is designed to investigate ways to facilitate the verification and use of environmental technology and exists solely for the benefit of industry and the user community. Two pilots operate under interagency agreements; however, the ETV program primarily operates by securing the cooperation of verification partners through open competition utilizing the agency’s assistance agreement program. Agency procedures for advertising, reviewing, and awarding assistance agreements are followed during the selection process. The procedures governing this process are available from GAD and are not discussed in this section. The procedures used to plan and control the selection of verification partners for ETV are listed below.

Planning to select verification partners requires:

• assessing and prioritizing environmental technology categories for use in pilot projects (i.e., defining the scope of each pilot project in terms of technology areas to be tested)
• establishing ANSI/ASQC E4-1994 as an applicable quality standard
• issuing RFA's
• selecting the appropriate verification partner based on their experience and proficiency
• Controlling the selection process to ensure the quality of verification tests includes:
• implementing controls stipulated in EPA policies and procedures for assistance agreements
• establishing specific language in each RFA requiring development and implementation of a quality system consistent with the ETV quality system, and ANSI/ASQC E4-1994.
• reviewing the applicant’s proposed quality system to verify it meets the RFA requirement and provides for quality of verification tests which will be known, documented, and meet technical requirements

Return to Table of Contents

Assistance solicitation documents shall contain information clearly describing the technical and quality requirements associated with the verification testing.

Technical and quality requirements expressed in the RFA include technical evaluation criteria for technical skills and experience of staff members, and demonstrated experience in the development of quality systems relevant to ETV. Cooperative agreements require that a verification partner, once selected, develops and submits for approval to EPA, a pilot quality management, plan consistent with E4, prior to conducting technical activities.

4.3 Quality specification/conformance

Return to Table of Contents

Assistance solicitation documents shall specify the ETV quality requirements for which the verification partner is responsible and how the verification partner’s conformance to client's requirements shall be verified.

ETV quality requirements for which the verification partner is responsible are specified in the RFA and in this ETV QMP. During verification partner selection, the applicants’ proposals and written responses to the requirements are reviewed for conformance to the RFA specifications. After a verification partner is selected, the EPA pilot quality manager reviews and the EPA pilot manager approves written quality system documents (e.g., pilot QMP) for conformance to the EPA and ETV quality policies and procedures.

4.4 Peer review of assistance agreements

Return to Table of Contents

Assistance award documents shall be reviewed for accuracy and completeness by qualified personnel prior to award.

Peer review is an integral part of EPA’s project planning, implementation, and assessment process. RFA packages are internally peer reviewed prior to their issuance. Responses to the RFA undergo a peer review process which supports the award of the assistance agreement.

4.5 Conformance of Verification Testing Efforts

Return to Table of Contents

Appropriate measures shall be established to ensure that the verification testing efforts satisfy all terms and conditions of the assistance agreement. Verification partners shall have a demonstrated capability to meet all terms and conditions.

Once a verification partner has been selected, measures to ensure continued conformance to terms and conditions in the assistance agreement are implemented as described in part A, sections 8, 9, and 10.

5.0 RECORDS

5.1 Scope

Return to Table of Contents

Procedures shall be established, controlled, and maintained for identifying, preparing, reviewing, approving, revising, collecting, indexing, filing, storing, maintaining, retrieving, distributing, and disposing of pertinent quality documents and records. Such procedures shall be applicable to all forms of documents and records, including printed and electronic media. Measures shall be taken to ensure that users understand the documents to be used. Records requiring control shall be identified.

Records to which this policy applies include:

__________________________

Information in this section applies to both electronic and printed records, as well as original records developed on behalf of the ETV program that are required to demonstrate the quality of information and data provided in ETV verification reports.

TABLE 5.1 Records Management Scheme

Return to Table of Contents

Sufficient records shall be specified, prepared, reviewed, authenticated, and maintained to reflect the achievement of the required quality for completed work and/or to fulfill any statutory requirements. Documents used to perform work shall be identified and kept current for use by personnel performing the work. Documents, including revisions, shall be reviewed by qualified personnel for conformance with technical requirements and quality system requirements and approved for release by authorized personnel.

Table 5.1 lists the pertinent quality records for ETV, the person(s) responsible for preparing and updating these records, the reviewers, those given approval authority for each record type, and the distribution plan. Where a procedure is not applicable (e.g., a document is not subject to approval), N/A is entered in Table 5.1. All reviewers and approving officials receive copies of the records they review/approve; the Distribution column in Table 5.1 lists only those individuals who receive final copies, in addition to the reviewers and approving official. For revised documents, these same review, approval, and distribution pathways are followed. Unless otherwise noted, material placed on the ETV website is available for public inspection, comment, and use.

5.3 Records Storage and Obsolete records

Return to Table of Contents

Obsolete or superseded documents shall be identified and measures shall be taken to prevent their use, including removal from the work place and from the possession of users when practical. Maintenance of records shall include provisions for retention, protection, preservation, traceability, and retrievableness. While in storage, records shall be protected from damage, loss, and deterioration. Retention times for records shall be determined based on contractual and statutory requirements, or, if none stated, as specified by the EPA coordinator and EPA line management.

Obsolete records should be clearly marked as such. These records may be retained in the workplace for historical reference, or they may be removed to archival storage. ETV will follow ORD'S Records Management Policy, Part 003 (see Appendix A), which addresses requirements for indexing, filing, maintaining, retrieving, and disposing of documents and records from all extramural financial agreements. The current minimum requirement is that all records be kept for seven years after the final payment on a cooperative agreement or interagency agreement.

6.0 COMPUTER HARDWARE AND SOFTWARE

6.1 General procedures

Return to Table of Contents

Computer software and computer hardware configurations used in the ETV program shall be installed/tested/used/maintained/controlled/documented to meet users' requirements and shall conform to this quality policy and applicable consensus standards and/or data management criteria.

At the program level, ETV does not expect to develop software. At the pilot level, if verification partners intend to develop software to support their ETV process (or an individual test/QA plan), the partner should have procedures in place as specified here. If the verification partner uses only commercial software for office operations (e.g., word processing software, spreadsheet software), it is unlikely that the partner would need specific procedures for assessing software quality. Part A, sections 6.2 through 6.6, apply only to software and software/hardware configurations developed specifically for the ETV program.

The following are the ETV program procedures which ensure that each pilot controls the quality of all computer hardware/software configurations for the program.

• The EPA pilot manager and the verification partner discuss and agree upon the computer hardware and software requirements of the pilot and/or specific test/QA plan.
• Once decisions are finalized, the verification partner supplies evidence of meeting all requirements before data collection, reduction, or validation procedures begins.
• For software developed for ETV programs, the verification partner tests all applications and configurations using a test data set or by running a shakedown test of the system to ensure all applications/configurations are operating to specifications. The verification partner must show evidence of a system to maintain, control, and document such software and hardware configurations. This includes, but is not exclusive of; resources to correct any hardware/software failure with minimal downtime to the program, tracking upgrades/revisions to software or configuration changes, documenting software names, versions, and copyright dates, and complete documentation of the code. Complete documentation of code includes the written code with comments structured in a modular form.

Return to Table of Contents

Computer software and computer hardware/software configurations covered by ETV’s quality policy includes, but is not limited to:

• operation or process control of environmental technology systems (including automated data acquisition and laboratory instrumentation)
• and data bases containing environmental data

Computer software and computer hardware/software configurations covered by this quality and management plan include all agreed upon, pilot-specific applications or configurations. These include, but are not limited to:

• evaluating and reducing environmental data
• reporting environmental data
• data bases containing environmental data

Return to Table of Contents

Computer hardware/software configurations shall be tested prior to actual use and the results shall be documented and maintained.

On a pilot level, the verification partner conducts tests of the computer hardware/software configuration using a standard set of testing conditions.

NOTE: Verification partner is required to have a system to document all testing of computer hardware/software configurations, as required by part A section 6.1. A test data set or a standard set of testing conditions should be developed on a pilot- or test/QA plan-specific basis. Maintenance testing should be easily tracked and retrievable.

6.4 Measurement and testing equipment configurations

Return to Table of Contents

Computer hardware/software configurations integral to measurement and testing equipment that are calibrated for a specific purpose do not require further testing unless:

• the scope of the software usage changes OR
• modifications are made to the hardware/software configuration

On a pilot level, verification partners perform the following procedures (as provided in the verification partner's quality system).

Whenever computer hardware/software configurations integral to measurement and testing equipment are calibrated for a specific purpose, further testing is not normally performed unless the scope of the software usage changes or modifications are made to the hardware/software configuration.

In the event either of the above mentioned changes occurs, the verification partner retest the changes as described in part A sections 6.1 and 6.3. Retesting is documented to the same extent as the original application/configuration.

6.5 Change assessments - configurations, components, and requirements

Return to Table of Contents

Changes to hardware/software configurations, components, or program requirements shall be assessed to determine the impact of the change on the technical and quality objectives of the ETV program supported.

The verification partner is responsible for assessing the changes, determining the need for testing, and reporting the assessments to the pilot manager.

6.6 ETV website roles and responsibilities

Return to Table of Contents

The ETV website shall be operated in such a way that it serves all ETV participants and customers through prompt and accurate posting of ETV information and documents.

The pilot managers, or alternate(s) designated in writing by the pilot manager, are responsible for sending the following information to the ETV webmaster:

• general fact sheets and brochures
• stakeholders lists (and updates)
• meeting announcements and summaries
• generic testing protocols (indicating draft or final)
• test/QA plans (indicating draft or final)
• CBD announcements
• ETV verification statements
• upcoming meetings/speeches/announcements

7.0 PLANNING

7.1 Systematic planning process

Return to Table of Contents
 

A systematic planning process shall be established, implemented, controlled, and documented to:

• identify the customer(s), and their needs and expectations
• identify the technical and quality goals that meet the needs and expectations of the customer
• translate the technical and quality goals into specifications that shall produce the desired result
• consider any cost and schedule constraints within which technology test activities are required to be performed
• identify acceptance criteria for the results or measures of performance by which the results shall be evaluated and customer satisfaction shall be determined

7.1.1 Systematic planning process established for ETV is conducted as follows:

• EPA establishes the number and type of ETV pilots necessary to comply with the Presidential mandate to cover all environmental technologies within three years.
• EPA lays out basic program operation parameters in two documents, the ETV Verification Strategy and ETV QMP.
• Based upon the ETV Verification Strategy, the ETV coordinator, in consultation with the Agency's Innovative Technology Council, the ETV team, and EPA line management, designs an annual budget.
• Appropriate personnel are appointed from within the ORD laboratories to fill ETV positions. Selection and information on qualification is presented in part A sections 3.1 and 3.2.
• The division and branch management provides resources and planning to support EPA staff assigned to the pilot, such as training and travel. Technical training is discussed in part A sections 3.3 and 3.4.
• At the beginning of each pilot, verification partners are selected in conformance with part A section 4.0 .  EPA’s requirements for the appropriate extramural agreement (e.g., cooperative agreement, interagency agreement) must also be met.
• After selection, the verification partner, in consultation with the EPA pilot manager, establishes a stakeholders group that contains representatives of customer groups of concern to that pilot area. [Note: in order to evaluate the efficacy of stakeholders groups, at least one pilot will not establish such a group.]
• The EPA pilot manager and the verification partner develop plans for pilot verification tests and present them to the stakeholders for review, comment, and advice.
• The EPA pilot manager, the EPA pilot quality manager, the verification partner, and the stakeholders group hold at least one joint meeting annually to:

• These four parties appoint a recorder to document the minutes of each meeting.
• Minutes of each meeting are taken and then are distributed to participants for comment. Minutes of stakeholders meetings are incorporated within the record management scheme described in part A section 5.0.

7.1.2 Implementation of the systematic planning process Planning is accomplished through frequent meetings among participants and through posting initial planning documents and stakeholders meeting minutes on the ETV website. Procedures for planning at the pilot and test level are addressed in Part B. Procedures for implementing the planning process are detailed below:

• Customer Identification - ETV customers are identified in part A section 1.
• Technical and quality goals identification - These are identified during planning meetings with senior management, conference calls with ETV participants, and meetings with EPA quality professionals and technical staff.
• Technical and quality goal specifications - The ETV coordinator works with the EPA pilot managers, EPA directors of quality assurance, and other quality professionals to translate technical and quality goals of the overall program into the ETV QMP.
• Cost and schedule constraints - These are discussed during planning meetings with senior management and considered yearly for allocation to each pilot.
• Measures of performance - The ETV coordinator develops measures of performance for the program, which are evaluated by the ETV team and verification partners throughout the course of the program. Appendix B contains the most current measures of performance.
• Customer satisfaction evaluation - The program evaluates all pilots on an annual basis to collect data for a final report on the pilot phase. This evaluation includes customer satisfaction measures as appropriate.

7.1.3 Systematic planning process controls include:

• development and implementation of written procedures (verification test protocols and test/QA plans)
• requirement of minutes of stakeholders group meetings
• review of verification partner work efforts by the EPA pilot manager and EPA pilot quality manager

7.1.4 Systematic planning process documentation includes the ETV Verification Strategy, the ETV QMP, the verification partners QMPs, and test/QA plans.

Return to Table of Contents

All planning documentation shall be reviewed and approved for implementation by authorized personnel before the specific work commences. Such documentation includes but is not limited to test/QA plans and generic verification test protocols.

Planning document review is discussed in part A section 5.2.

8.0 IMPLEMENTATION OF WORK PROCESSES 

8.1 Implementation

Return to Table of Contents

Work shall be performed according to approved planning and technical documents.

The planning for the implementation of the EPA management and quality work processes is contained in part A section 7.0. The individual ETV pilot work is performed according to planning documents written by the pilot. All technology verification work shall occur according to protocols and test/QA plans developed and agreed upon by EPA, the verification partner, and the vendor. The authors, reviewers, and approvers of these documents are specified in part A section 5.0, Table 5.1.

The approved protocols and test/QA plans shall be present on the site of testing, and the work shall be implemented in accordance with them. During the work phase, modifications to plans and procedures shall be documented, and the modifications shall be incorporated into the final protocols and test/QA plans. The authors, reviewers, and approvers of changes to these documents are the same as for the original documents and are specified in part A section 5.0, Table 5.1.

Verification partners are responsible for implementing their work processes in accordance with their quality systems. 

8.2 Procedures

Return to Table of Contents

Procedures shall be developed, documented, and implemented for appropriate routine, standardized, special, or critical operations. Operations needing procedures shall be identified. The form, content, and applicability shall be addressed, and the reviewers and approvers shall be specified.

Procedures for the overall operation of the ETV program are contained in the ETV Verification Strategy, the ETV QMP and in other appropriate EPA policies (e.g., contractual, records management). The individual ETV pilots shall identify and document those operations in their pilots requiring procedures as discussed in Part B. Procedures shall be written in a format that can be readily comprehended by the user and shall contain sufficient detail and clarity to ensure that results are achieved effectively. Appropriate operations documents, authors, reviewers, and approvers are specified in part A section 5.0, Table 5.1.

8.3 Oversight

Return to Table of Contents

Implementation of work shall be accomplished with a level of management oversight and inspection commensurate with the importance of the program and the intended use of the results, and shall include the routine measurement of performance against established technical and quality specifications.

EPA line management has responsibility for oversight of verification work processes as discussed in part A section 1.0. Verification partner oversight and responsibilities for the verification work processes are given in the individual pilot QMPs.

9.0 ASSESSMENT AND RESPONSE

9.1 Numbers and types of assessments

Return to Table of Contents

Assessments shall be planned, scheduled, and conducted to measure the effectiveness of the implemented management and quality systems. Several types of assessments are available for this purpose. Management shall determine during the planning stage the appropriate types of assessment activities. Assessments shall include an evaluation to determine and verify whether technical requirements, not just procedural compliance, are being implemented effectively.

The assessments shown in Table 9.1 and the minimum frequency are commensurate with the importance of the ETV program and the intended use of the verification results. Management assessments shall be used to measure the effectiveness of the implemented management systems and technical systems. Performance assessments shall be used to evaluate performance of the pilot technical operations. Data assessments shall assess reported data quality. Verification partners perform self-assessments in accordance with the individual pilot management plans, and EPA performs independent assessments of verification partners.

Table 9.1 Assessments
 

Return to Table of Contents

Assessments shall be performed according to written and approved procedures, based on careful planning of the scope of the assessment and the information needed. Assessment results shall be documented and reported to management. Management shall review the assessments.

Assessments shall be planned according to the scope of the assessment and the information needed. Suitable written procedures for planning and conducting audits shall be contained in the operating manuals of EPA quality teams, the operating and quality manuals of the verification partners, and EPA guidance documents. Assessments are based on interviews, on the physical examination of objective evidence, and on the examination of the documentation of past performance. Results are documented in audit reports, and reviewed by appropriate management.
 

9.3 Personnel qualifications, responsibility, and authority

Return to Table of Contents

Personnel conducting assessments shall have the appropriate technical or management skills to perform the assigned assessment. Management shall determine and document the level of competence, experience, and training necessary to ensure the capability of personnel conducting assessments. The responsibilities and authorities of personnel conducting assessments shall be clearly defined and documented, particularly in regard to authority to suspend or stop work in progress upon detection and identification of an immediate adverse condition affecting the quality of results or the health and safety of personnel.

EPA and verification partner management determines and documents the level of competence, experience, and training of their respective audit personnel during hiring and periodic performance reviews. Qualified audit personnel, as listed in Table 9.l, have access to the appropriate management personnel and documents required to perform their audit duties. They are organizationally independent of the program or pilot they are auditing. They have the responsibility and authority to:

• identify and document problems that affect quality of verification results
• propose recommendations for resolving problems that affect quality of verification work processes or results
• independently confirm implementation and effectiveness of solutions

If auditors identify a severe problem affecting verification quality, EPA pilot managers have the authority to request of the verification partner manager that work be stopped until the problem is addressed. If auditors identify a problem where the health and safety of personnel are in danger, they have the responsibility to bring it to the immediate attention of appropriate EPA management, verification partner management, and onsite testing personnel.

9.4 Response

Return to Table of Contents

Responses to adverse conclusions from the findings and recommendations of assessments shall be made in a timely manner. Conditions needing corrective action shall be identified and the appropriate response made promptly. Follow-up action shall be taken and documented to confirm the implementation and effectiveness of the response action.

When the recommendations and conclusions from the findings of assessments are adverse, response from the auditee detailing the corrective action shall be expected within 10 working days of receiving the audit report. Auditors shall follow up with appropriate documentation to confirm the implementation and effectiveness of the response.

10.0 QUALITY IMPROVEMENT

10.1 Annual review for quality improvement

Return to Table of Contents

A quality improvement process shall be established and implemented to continuously develop and improve the ETV Quality System.

The ETV coordinator and EPA directors of quality assurance review the quality and management plan annually and recommend improvements to the plan.

The EPA directors of quality assurance recommend improvements and negotiate improvements with the ETV team during the annual meeting and through the ETV website.

10.2 Detecting and correcting quality system problems

Return to Table of Contents

Procedures shall be established and implemented to prevent as well as detect and correct problems that adversely affect quality during all phases of technical and management activities.

EPA pilot managers and EPA pilot quality managers report problems in any of the areas to EPA line management and the ETV directors of quality assurance:

• adequacy of the ETV quality system
• consistency of the quality system
• implementation of the quality system
• correction of quality system procedures
• completeness of documented information
• quality of data
• quality of planning documents
• implementation of the work process

EPA line managers respond promptly to address correction of the quality problem.

10.3 Cause and effect relationship

Return to Table of Contents

When problems are found to be significant, the relationship between cause and effect and the root cause shall be determined.

The following are general procedures. Specific procedures are found in the individual verification partners written quality systems. When problems are significant, the quality manager determines and documents the relationship between cause and effect, and when possible, determines and documents the root cause of the problem. The quality manager provides this information to the appropriate project managers so corrective action can be authorized and implemented.

A significant problem is any problem requiring:

• a testing protocol change OR
• a management system change OR
• a quality system change (either internal or external to EPA, but still within the ETV program)

NOTE: The verification partner quality managers in accordance with their quality systems are continually reviewing and assessing their projects for conformance with their quality documents. At the program level, assessment reports from the individual projects are monitored and evaluated by the ETV directors of quality assurance for trends or recurring problems that are indicative of significant problems affecting the ETV program as a whole. Any such situation is immediately communicated to the ETV coordinator. The ETV coordinator shares the information and any corrective actions with the EPA pilot managers.

10.4 Root cause

Return to Table of Contents

The root cause should be determined before permanent preventative measures are planned and implemented.

To guard against implementing ineffective changes, EPA personnel ensure when possible that root causes are determined before preventative measures are planned and implemented.

10.5 Quality improvement action

Return to Table of Contents

Appropriate actions shall be planned, documented, and implemented in response to findings in a timely manner.

In the event that a significant problem is identified that requires a structural change to the ETV program, the ETV Coordinator will initiate discussions with EPA line management appropriate to correct the deficiency.

PART B

Return to Table of Contents

COLLECTION AND EVALUATION OF ENVIRONMENTAL DATA

Part A of the ETV Quality and Management Plan contains the specifications and guidelines that are applicable to common or routine quality management functions and activities necessary to support the ETV program.

Part B of the ETV Quality and Management Plan contains the specifications and guidelines that apply to test-specific environmental activities involving the generation, collection, analysis, evaluation, and reporting of test data.

1.0 PLANNING AND SCOPING

Return to Table of Contents

The work of the ETV program at the pilot level is to verify the performance of commercial-ready technologies. As discussed in part A section 4.0, the planning process begins with the Statement of Work (SOW) contained in the Request for Applications (RFA). The successful applicant becomes the verification partner for the pilot.

1.1 Systematic planning of the verification test

Return to Table of Contents

All work involving the generation, acquisition, and use of environmental data shall be planned and documented. The type and quality of environmental data needed for their intended use shall be identified and documented using a systematic planning process. The test-specific planning must involve the key users and customers of the data. EPA pilot managers should guide planning activities and ensure that participants are informed of and understand completely the requirements of each test.

The programmatic planning for verification of commercial-ready technologies is discussed in part A section 7.1.1. This section continues the discussion of systematic planning at the pilot level.

Verification partners, working with the EPA pilot managers, begin a systematic process to plan the individual pilot tests. Systematic planning may be accomplished through any demonstrated technique including the data quality objectives process (EPA QA/G-4) and the observational method. The planners perform the following actions:
 

• refine the scope of their respective technology areas
• determine interest in verification from the manufacturers of commercial-ready technologies within the defined scope of the technology areas
• convene stakeholder groups, containing representatives of verification customer groups, which advise during the planning process
• mediate and facilitate the selection of focus areas
• prepare generic verification protocols which are developed to promote uniform testing for a given type of technology
• coordinate the review and revision of the protocols (See the review and approval scheme in part A section 5.0.) keeping in mind both the customers and EPA’s objectives for verification as defined in the ETV Strategy
• solicit vendor agreements to participate in verification of their products based on the generic protocol (some iteration of the two previous points frequently occurs here as the vendors review and request revision of portions of the generic protocols)
• prepare test/QA plans for the acquisition of data to verify the performance of the vendors technologies

The protocols and test/QA plans describe the experimental approach, with clearly stated test objectives and associated quality objectives for the related measurements.

1.2 Systematic planning for verification testing

Return to Table of Contents

• Organizations that participate in the test shall participate in the planning.
• The scope and objectives of the verification testing and the desired action or result from the work shall be defined.
• The data to be collected to achieve verification shall be identified, and the QA and QC requirements to establish the quality of the data shall be defined.
• Verification tests shall undergo a design process
• Verification tests shall be documented.
• Equipment, operators, and skill levels required for the verifications shall be identified.
• Any constraints (e.g. time and budget) shall be identified.
• Conditions, which will suspend work, shall be identified.
• Assessment tools shall be determined.
• Methods and procedures for storing, retrieving, analyzing, and reporting the data shall be identified.
• Methods and procedures for minimizing, characterizing, and disposal of hazardous waste generated during the test shall be identified.

1.2.1 Planning personnel

The verification partner shall coordinate test planning among the participating organizations including EPA, the stakeholders, the vendors, and any testing organizations and laboratories participating in the test. The verification partner, with the concurrence and oversight of the EPA pilot manager, shall identify the planning roles of the various players, and shall conduct planning activities by shared communication via teleconference, video conference, and in-person meetings, as appropriate, and within the constraints of the budget.

1.2.2 Purpose, scope and objectives

The purpose of this testing is to verify the performance of commercial-ready technologies. Another objective is to develop an efficient method for testing commercial-ready technologies. Many of the pilot tests accomplish this objective by preparing generic verification protocols whereby the performance of similar technologies can be verified in the future using the same protocol. The characteristics of individual technologies and the specifics of individual tests are covered in the test/QA plan that incorporates the generic verification protocol by reference.

1.2.3 Data to be collected and design of experiment

During planning of the technology verification test, the process, environmental, laboratory, response, and QA data to be collected are identified. Also identified are testing organizations, test personnel, skill levels, methods, procedures, and equipment unique to each verification test. Planning is integrated into design as discussed in part B section 2.0.

1.2.4 Documentation and reporting

Records generated during the pilot tests are listed in part A section 5.0. Records consist of both paper and electronic records. Electronic methods for storing, retrieving, analyzing, and reporting the data are generally commercially available programs for word processing, spreadsheet, or database processing, or commercial software developed especially for data collection and processing on a specific instrument or piece of equipment. Pilots may also develop software/hardware configurations, as appropriate, in their technology verification tests. The use of computer hardware and software is discussed in part A section 6.0. Paper records such as field notebooks, bench sheets, field data sheets, custody sheets, and instrument printouts are part of the raw data test record and kept with the study records.

1.2.5 Assessments

The assessment tools and minimum frequencies of assessments for the verification tests are identified in part A section 9.0. The definitions of the assessment tools and suggested frequencies are given in part B section 4.0.

1.2.6 Constraints, suspension of work, waste minimization and disposal

Verification partners work under the constraints of time and resources communicated to them by the EPA ETV Coordinator and the EPA pilot manager. When constraints are determined by the verification partner to affect quality, the resolution of the problem proceeds as described in part A section 1.5. Circumstances under which work can be suspended are discussed in part A section 1.7. If waste is generated as part of the verification testing, the verification partner seeks to minimize the amount, and disposes of it in accordance with applicable local, state, and federal laws.

2.0 DESIGN OF TECHNOLOGY VERIFICATION TESTS 

2.1 Design process

Return to Table of Contents

The design shall incorporate those activities pertaining to verification of performance identified during the planning process, establish test specifications, and identify appropriate controls. The design shall include