On April 28, 2004, the Council requested an ISRP review of the Nez Perce Tribe-Department of Fisheries Resource Management-Watershed Division's (NPT) statistical design for monitoring effectiveness of watershed restoration projects. The submittal to the ISRP included two documents: 1) Effectiveness Monitoring and Evaluation Plan for Restoration Projects in the Nez Perce Treaty Territory and 2) Watershed Monitoring and Evaluation: A plan to assess watershed condition and recovery throughout Nez Perce Tribe Territory.
This NPT monitoring project was originally proposed during the 2001 Mountain Snake Provincial Review. The Council recommended partial funding for the new project as one of the few that NOAA Fisheries stated responds to RPA 183 pertaining to monitoring and evaluation. This funding was to allow the sponsor to develop a more detailed statistical design. As a condition for continued funding, the Council recommended the detailed design address the ISRP's questions and comments about the choice of physical habitat parameters.
The ISRP understands that the original plan as proposed was significantly revised in response to comments from the Bonneville Power Administration (BPA); consequently, the ISRP reviewed the current submittals anew as a stand-alone proposal rather than as specific responses to previous ISRP comments on the original proposal.
Summary
1. Coordination. Although the ISRP understands that these plans for action effectiveness and watershed monitoring were somewhat coordinated with the Action Agency RME Plan (NOAA 2003) and BPA project no. 35019 for Pilot Status and Trend Monitoring Program for Salmonids and their Habitat in the Wenatchee, John Day, and Upper Salmon, more coordination is needed. This further coordination should be documented in the NPT plans. In so far as possible, the same data collection protocols should be used, especially given that one of the pilot projects for the status and trend monitoring will be in the Upper Salmon River Basin.
We further support coordination with the relatively recent efforts to coordinate and standardize M&E efforts in the Pacific Northwest by the ad hoc group of fisheries scientists under the banner of the "Pacific Northwest Aquatic Monitoring Partnership." We believe that this point in time is a unique and possibly limited opportunity to accomplish better coordination of M&E activities throughout the Tribal Lands, States, Provinces, the Columbia Basin, and the Pacific Northwest. The benefits of consistent, coordinated monitoring are indisputable. The issues are related to how to best achieve the benefits.
2. Effectiveness Monitoring. The authors have made a good faith attempt to bring appropriate statistical design and analyses into Tier 3 action effectiveness monitoring as envisioned by the NOAA Fisheries "2000 BiOp and the Action Agencies" RME Plan. However, it appears that the authors are well aware of the fact that feasible study designs are not manipulative experiments (Hurlburt 1984) and will not yield cause and effect conclusions. Furthermore, these large scale treatment/control pair or BACI mensurative experiments are very difficult to maintain in relatively long-term environmental studies. The authors have appropriately hedged their proposed analyses to include the use of correlation/multiple regression modeling techniques to analyze the data from these mensurative experiments.
3. Watershed Monitoring and Evaluation Plan. The design for selection of study sites in this draft document will not meet the objectives for Tier 2 status and trend monitoring of the watershed. The authors should adopt the probabilistic site selection procedures being used in BPA project no. 35019 for Pilot Status and Trend Monitoring Program for Salmonids and their Habitat in the Wenatchee, John Day, and Upper Salmon.
The plan has made a good start to provide written data collection methods for Tier 2 status and trend monitoring. The NPT note that the Watershed Monitoring and Evaluation Plan is a work in progress, and the ISRP provides comments below to inform further development of the plan. The ISRP recently gave a favorable review to the Step Two Review of the Northeast Oregon Hatchery (NEOH) Spring Chinook Master Plan: Monitoring and Evaluation Plan (ISRP 2004-10). Although the NEOH M&E plan concentrates on M&E for the supplementation project, the data to be collected in the Imnaha, Grande Ronde, and Asotin Creek basins would appear to be directly applicable to meet many of the objectives of the action effectiveness monitoring in this document. To the extent possible, study designs and data collection protocols should be the same, so that data can be exchanged.
4. Feasibility. The plans are extensive and will be expensive to carry out. There is no suggestion of costs. As an idealized plan, the draft for effectiveness could be a good model. As a practical plan, it may not be feasible.
5. Recommendation. The ISRP recommends continued funding of the project subject to a serious attempt of the sponsor to coordinate status and trend monitoring within all of the tribal lands with procedures being developed in BPA project no. 35019 for Pilot Status and Trend Monitoring Program for Salmonids and their Habitat in the Wenatchee, John Day, and Upper Salmon.
Review Comments On Effectiveness Monitoring And Evaluation Plan For Restoration Projects In The Nez Perce Treaty Territory
1. Replica. Reviewers were puzzled by the project sponsors' use of the word replica. Much of the plan deals with statistical design and analysis issues, yet this word is used incorrectly. Throughout the plan, taking more than one sample is called a replica instead of a replicate. A replica is a false copy; a replicate is a repeated sample from the same population or universe. Maybe this is a simple mistake of assuming that replica is the plural of replicate? Reviewers did not find other similar errors.
2. Hypothesis testing. The authors give a much more appropriate approach for testing hypotheses than the ISRP has usually found in the Columbia Basin, although the approach remains based on assumed models (see below for more discussion of model based analyses). For example, they state early on "The statistical hypothesis for the paired-sample one-tailed t-test is
H0:
H1:
where 
and Δt is the expected increase in the response variable by time t. The value of Δt reflects the statistical change in the mean biological response for a given restoration action that would constitute a meaningful improvement in abundance and/or survival rates of fish. It also corresponds to the effective size of the response needed in order to have an 80% probability of detecting a significant change in the response variable." This is an appropriate method of reversing the null hypothesis, or setting up a test for bio-equivalence and is an excellent approach for hypotheses to be tested in evaluation of management actions. Other appropriate analyses include correlation/regression methods with estimation and with measures of bias and precision.
3. Multi-regression Methods. The authors appropriately state that multiple regression methods are a likely necessary backup to the idea of testing hypotheses based on treatment/control pairs in long-term studies. For example, a quote is "In cases where there is not a statistical relationship between a given restoration project and survival, multi-regression analysis and model selection techniques can be used to identify confounding factors (e.g. location effects, density-dependence, watershed land use patterns) that may have obfuscated a significant result." The ISRP cautions that maintaining treatment/control pairs in long-term studies of this scale is extremely difficult and may be infeasible. ISRP members are not aware of successful implementation of a study with this design at the scale envisioned anywhere in the world.
4. Terminology. The document should use M&E jargon that is consistent with the NOAA Fisheries BiOp, Action Agencies RME, and ISRP/ISAB reports. For example, "implementation monitoring" as defined by the ISRP is similar to the authors' "Technique Effectiveness" and "Tier II status and trend monitoring" as defined by the 2000 NOAA Fisheries BiOp is similar to the authors' "Stream Condition Assessment."
5. BACI Designs. The ISRP cautions that BACI studies are mensurative experiments (Hurlbert 1984) or "observational studies" with statistical inferences limited to conclusions on the specific study areas and time periods involved. Years as replicates are not equivalent to replicates created in manipulative experiments where treatments and controls are randomly assigned to experimental units.
The ISRP grants that using inductive logic, more "replicate" pairs of treatment and controls in a BACI design give more confidence if consistent results are seen. That is, if we see the same results of a treatment several times then we begin to believe the treatment caused the result, but there is no universe of "treatment/control" pairs that is being randomly sampled to give rise to probabilistic arguments about power, alpha, etc. Concepts of statistical theory are model based if the researchers are willing to make the assumptions required by the model.
The bottom line is that treatment/control and BACI studies are mensurative experiments with design based statistical inferences limited to conclusions on the specific study areas and time periods involved. The statistical inferences are made as if our replicate pairs are a random sample from some imaginary universe in space and as if years are a random sample from time past, present, and future.
6. Variance in Spawner Counts. Spawner counts (S in equation 9) are random variables subject to variance. Spawner counts are not fixed known values. Most often they are subject to sampling error (i.e., counts on surveyed units are used to make inferences about unsurveyed areas) and are subject to measurement error of a magnitude that should not be ignored (i.e., not all spawners are detected).
7. Means of Ratios or Ratios of Means. The mean in equation (9) is the average of ratios of random variables and is certainly not a maximum likelihood estimate for the parr survival index. In fact, we have the classic decision on whether it is better to average ratios or take the ratio of averages, the latter being the choice of the ISRP in this situation.
Minor Comments On The Effectiveness Monitoring Plan
Spawner-to-parr survival should be defined. Are these mean parr/spawner ratios, or egg to par survival?
The subscripts and notation between equations (1) and (7) could be more consistent.
In the Goal under Culvert Replacement, we assume that passage to habitat for all anadromous salmonids is involved, not just steelhead habitat.
We assume that in the Retrospective Analysis section, regression models with multiple predictor (independent) variables would be considered in modeling of survival rates.
The text has many typos.
Review Comments On The Watershed Monitoring And Evaluation: A Plan To Assess Watershed Condition And Recovery Throughout Nez Perce Tribe Territory
1. Overall Comment. The authors realize that because of limited scope or different focus existing monitoring programs are not adequate to assess watershed condition throughout the Treaty Territory. Unfortunately, neither is the plan outlined in this document. Objective 2, in Chapter 1, "Determine the quality and extent of habitat available to anadromous and resident fishes within Treaty Territory", i.e., Level two monitoring: Baseline Data Collection and Stream Condition Assessment, is simply not possible with the current direction of these four documents. Concentration on selected watersheds and monitoring the effectiveness of management actions will not take the place of needed Tier II status and trend monitoring.
2. Data Collection Protocols. The document may become a good manual for study design and data collection protocols, but it is still a very rough draft. There are figures missing and not all text has been written. What has been written needs editing badly. There are extensive and detailed methods for many procedures that should be good guidance for staff. The ISRP did not review in detail each procedure, but the ones reviewers picked out seemed complete enough for a technician to follow easily. It is not clear how this manual relates to the Effectiveness Monitoring Plan, for there are some redundancies (maybe an introduction could list the various guides that Tribe is producing). The ISRP would certainly encourage the Tribe to continue to develop this manual.
3. 100 m Reaches for Physical Habitat Monitoring. Is a randomly selected reach of length 100 m in each third of a stream sufficient for monitoring physical habitat? The authors should consider further cooperation with NOAA Fisheries' pilot status and trend monitoring in the Upper Salmon to select study sites by the EMAP procedures. Site selection with more or less the same properties but with increased capacity for expansion would result. Rules such as "If random selection results in selection of a segment that is dewatered or inaccessible, another random value will be generated for reach placement within the basin." and "Should discrete sampling obstacles (cliffs, property borders w/access denied, etc.) or tributary confluences be found within 50 m of the reach site point, the beginning and end of the selected reach shall be shifted so as to exclude obstacles without exclusion of site point." must be eliminated from the document to meet the stated objectives.
4. Use of existing permanent monitoring reaches. "Existence of permanently established monitoring reaches within the watershed shall take precedence if they are viewed as being reasonably representative of their respective basin." It is not acceptable to substitute existing study sites for randomly selected sites in a survey designed to make statistical inferences. The ISRP recommends doing both the randomly selected sites and the established monitoring reaches for a period of time.
5. Standards. The section on meeting numerical standards necessary for reporting whether stream habitat and water quality are in acceptable condition does not seem to be consistent with ISAB/RP reports[1], on need for variation in water quality parameters.
